DCS started using Legato (now EMC) Networker as a centralized, diverse OS supporting backups solution in 1999.

It replaced Budtool, a previous product purchased by Legato and incorporated into Networker, and various ufsdump/tar scripts on production services. Over the years, additional scripts and reports were written by TSG to help maintain the Networker installation. These were developed as needs arised, and have been worked on by full time programmers and student admins. The Networker listservhas been a valuable resource for the creation and sharing of these lists. These scripts and Networker are supplemented with an old isql database that contains what the backup state of systems should be (not in backups, in backups, out-of-service, etc.) and their cluster/group association in the department. Another file associates those clusters with contact information for backups notices. Response reports (triggered by Networker action) * Savegroup reports are parsed through a perl script that mails the original Networker report unchanged, and mails a summary report following it. (Attached as email-saveset.txt) * Bootstrap messages are delivered to two different hosts and stored locally, in addition to being mailed to the backupsmgr. * Tape mount requests are piped through a script (tape-mount-notify.pl) that checks to see if the tape is in the jukebox, and if not, sends email to the pagers. #! /usr/bin/perl -w use strict; # my $RECIPIENTS = "admin\@host.com, operator\@host.com, pager\@host.com"; my $notice = <STDIN>; # # Dave debugging # #open(MAILDAVE, "| /usr/lib/sendmail -t"); #print MAILDAVE "To: you\@yourdomain\n"; #print MAILDAVE "Subject: Legato tape mount -- RAW\n\n"; #print MAILDAVE "$notice\n"; #close MAILDAVE; # The original script skipped messages like this # This script is just for specific tape mounts if ($notice =~ /[Ww]aiting for [0-9]+ writable/) { exit; } $notice =~ / tape ([\S]+)/; my $tape = $1; my $injuke = 0; open (JB, "/usr/sbin/mmlocate -n $tape |") or die "Cannot run mmlocate"; while (<JB>) { # see if it matches one of my jukebox names.

if (/dcs-dltjb/ or /dcs-ltojb/) { $injuke = 1; } } close (JB); #print $notice . "\n" . "tape is $tape\n"; #print "injuke is $injuke\n"; if (! $injuke) { open (SENDMAIL, "| /usr/lib/sendmail -t"); print SENDMAIL "To: $RECIPIENTS\n"; print SENDMAIL "Subject: Networker tape mount request\n\n"; print SENDMAIL "Tape $tape needs to be mounted\n"; print SENDMAIL "\n$notice\n"; close SENDMAIL; } #print "Tape is $tape\n"; # #NetWorker media: (waiting) waiting for dlt7000 tape Aug01.022 on mynsrserver.mydomain.com tape-mount-notify.pl #! /usr/bin/perl -w use strict; # my $RECIPIENTS = "admin\@host.com, operator\@host.com, pager\@host.com"; my $notice = <STDIN>; # # Dave debugging # #open(MAILDAVE, "| /usr/lib/sendmail -t"); #print MAILDAVE "To: you\@yourdomain\n"; #print MAILDAVE "Subject: Legato tape mount -- RAW\n\n"; #print MAILDAVE "$notice\n"; #close MAILDAVE; # The original script skipped messages like this # This script is just for specific tape mounts if ($notice =~ /[Ww]aiting for [0-9]+ writable/) { exit; } $notice =~ / tape ([\S]+)/;

my $tape = $1; my $injuke = 0; open (JB, "/usr/sbin/mmlocate -n $tape |") or die "Cannot run mmlocate"; while (<JB>) { # see if it matches one of my jukebox names. if (/dcs-dltjb/ or /dcs-ltojb/) { $injuke = 1; } } close (JB); #print $notice . "\n" . "tape is $tape\n"; #print "injuke is $injuke\n"; if (! $injuke) { open (SENDMAIL, "| /usr/lib/sendmail -t"); print SENDMAIL "To: $RECIPIENTS\n"; print SENDMAIL "Subject: Networker tape mount request\n\n"; print SENDMAIL "Tape $tape needs to be mounted\n"; print SENDMAIL "\n$notice\n"; close SENDMAIL; } #print "Tape is $tape\n"; # #NetWorker media: (waiting) waiting for dlt7000 tape Aug01.022 on mynsrserver.mydomain.com # # #!/local/bin/perl -w # # Scan the saveset report mailed by networker and report the interesting # parts of it, mailing both the original report and the savegroup report # # 2 Jan 2002 - Dave Mussulman <mussulma@uiuc.edu> # 13 Feb 2003 - James Doyle <jgdoyle@uiuc.edu> ** UPDATED ** # post edits by Dave Mussulman use strict; use Net::SMTP; # Configurable options below my $smtphost = "localhost"; # address of SMTP server my $fromaddr = "backupsmgr\@yourdomain"; # sender my $fromname = "DCS Backups"; # sender name my $rcptaddr = "backupsmgr\@yourdomain"; # who gets the email

my $hostname = `/bin/hostname`; chomp($hostname); # End configurable options my $origmail = Net::SMTP->new($smtphost); my $ignore_email_headers = 0; if (defined($ARGV[0]) && $ARGV[0] eq "-fromemail") { $ignore_email_headers = 1; print "Ignoring email headers\n"; } my $bltoggle = 0; my $failline = ""; my $weekfailtoggle = 0; my $badclockwarn = 0; my $openvbwarn = 1; my $group = ""; #my $subject = ""; my $clientcount = 0; my $failcount = 0; my $messageid = ""; my $outmail_subject = ""; my $now = time; my %open = (); my %openvb = (); my %fail = (); my %quietfail = (); my %big = (); my %timesync = (); my %full = (); my %abandoned = (); my %cannotstat = (); my %backuptime = (); my %tracksuccess = (); my %tracknoconnect = (); my %trackfail = (); my $trackstate = "unknown"; $origmail->mail($fromaddr); $origmail->to($rcptaddr); $origmail->data(); $origmail->datasend("From: $fromname <$fromaddr>\n"); $origmail->datasend("To: $rcptaddr\n");

$messageid = "<backups." . time() . "." . $$;

$origmail->datasend("Message-ID: $messageid.A\@$hostname>\n"); while (<STDIN>) { if ($ignore_email_headers && /^$/) { $ignore_email_headers = 0; next; } if ($ignore_email_headers) { next; } # Display the failed line if (/^NetWorker savegroup: \((.+?)\) ([\w\d\-]+)/) { if ($1 eq "alert") { $failline = $_; } $group = $2; # # # # print "Group $group\n"; print $_; print "\n"; $subject = "[DCS-Backups] Completion report - $group"; if ($failline) { $clientcount = $failline; # $clientcount =~ /\(alert\) $group completed, (\d+) client\(s\) \((.+?) Failed\)/; $clientcount =~ /total ([\d]+) client.+?([\d]+) Failed/; # my @tmp = split(/,/,$2); # $failcount = @tmp; $failcount = $2; $origmail->datasend("Subject: [DCS-Backups] $group - Completion report - $failcount of $1 client(s) failed\n"); $outmail_subject = "[DCS-Backups] $group - Saveset check - $failcount of $1 client(s) failed\n"; } else { $origmail->datasend("Subject: [DCS-Backups] $group - Completion report - Success\n"); $outmail_subject = "[DCS-Backups] $group - Saveset check - Success\n"; } $origmail->datasend("\n"); } $origmail->datasend("$_"); # Warn if this is a weekly

if (/level=5,/) { $weekfailtoggle = 1; } # Did a non-full backup a gigabyte or more? if (/^[\s]+(.+?)[\s]+level=([\w\d]+),[\s]+([\d]+) (GB|TB) / && $2 ne "full") { $big{$1} = $_; } elsif (/^[\s]+(.+?)[\s]+level=([\w\d]+),[\s]+([\d]+) MB/ && $2 ne "full" && $3 > 1000) { $big{$1} = $_; } # Checking for full backups if (/[\s]+(.+?)[\s]+level=full,/ && ($1 !~ /SYSTEM DB:/ && $1 !~ /SYSTEM STATE:/ && $1 !~ /SYSTEM FILES:/ && $1 !~ /ASR:/ && $1 !~ /VSS ASR DISK:/ && $1 !~ /VSS OTHER:/ && $1 !~ /VSS SYSTEM BOOT:/ && $1 !~ /VSS SYSTEM SERVICES:/ && $1 !~ /VSS USER DATA:/ && $1 !~ /bootstrap/) ){ $full{$1} = $_; } # Check for abandoned saves if (/\* (.+?)[\s]+is being abandoned/) { $abandoned{$1} = $_; } # Check for files too large to backup if (/\* (.+?)[\s]+save: cannot stat /) { $cannotstat{$1} = $_; } # Check for clients that only backed up indexes and nothing else if ($bltoggle && /: index:(.+) level=/) { my $mach = $1; if ($failline =~ /$mach/) { #print "$mach is in the fail list\n"; $fail{$mach} = 1; } else { # # } $quietfail{$mach} = 1; print "Did a machine fail?\n"; print $_;

} # Check for open files if (/^\* ([\w\d\-_\.]+):.+process cannot access the file/ || /^\* ([\w\d\-_\.]+):.+Resource temporarily unavailable/ || /^\* ([\w\d\-_\.]+):.+Cannot save locked profile/) { my $mach = $1; my $line = $_; $open{$mach} = 1; $openvb{$line} = $line; } # Check for unsynchronized client clock if (/^\* ([\w\d\-_\.]+):.+unsynchronized client clock detected/) { $timesync{$1} = 1; } # Check for backups that took a long time if (/B (\d\d):(\d\d):(\d\d)/) { my $t = $1 . $2 . $3; #print "time is $t\t$_\n"; if ($t > 20000) { # more than 2 hours long $backuptime{$t} .= $_; } } if (/Unsuccessful Save Sets/) { #db("State is unsuccessful!"); $trackstate = "unsuccessful"; } if (/Successful Save Sets/) { #db("State is successful!"); $trackstate = "successful"; } if (/\* (.+?):All Connection (timed out|refused)/) { #db("$1 timed out"); $tracknoconnect{$1} = $now; } if (/\* (.+?):/ && $trackstate eq "unsuccessful") { #db("$1 had unsuccessful backups"); $trackfail{$1} = $now; }

if (/^ (.+?): (\S+) (.+)$/ && $trackstate eq "successful" && $2 !~ /^index:/ && $3 !~ /level=skip/) { #db("$1 $2 had successful backups"); $tracksuccess{$1} = $now; }

# Toggle off a blank line (this should be last) if (/^$/) { #print "It's a blank line!\n"; $bltoggle = 1; next; } else { $bltoggle = 0; } } # End of looping through the input $origmail->dataend(); # Send off the raw data now $origmail->quit(); sleep (10); # # Start report mail here # my $outmail = Net::SMTP->new($smtphost); $outmail->mail($fromaddr); $outmail->to($rcptaddr); $outmail->data(); $outmail->datasend("From: $fromname <$fromaddr>\n"); $outmail->datasend("To: $rcptaddr\n"); $outmail->datasend("Message-ID: $messageid.B\@$hostname>\n"); $outmail->datasend("In-Reply-To: $messageid.A\@$hostname>\n"); $outmail->datasend("References: $messageid.A\@$hostname>\n"); $outmail->datasend("Subject: $outmail_subject"); $outmail->datasend("\n"); #$outmail->datasend("Done.\n\n"); $outmail->datasend("$failline\n") if ($failline); if ($weekfailtoggle) { $outmail->datasend("\n*****************************************************************\n"); $outmail->datasend("*** WEEKLY BACKUP - Remember to put failures in Failed-Weekly ***\n"); $outmail->datasend("*****************************************************************\n"); }

### my @quietfail = sort keys %quietfail; if (@quietfail) { $outmail->datasend("\nFailed clients NOT in the fail list\n"); $outmail->datasend("=" x 78 . "\n"); foreach my $i (@quietfail) { $outmail->datasend(" $i\n"); } } ### my @fail = sort keys %fail; if (@fail) { $outmail->datasend("\nFailed clients\n"); $outmail->datasend("=" x 78 . "\n"); foreach my $i (@fail) { $outmail->datasend(" $i\n"); } } ### my @full = sort keys %full; if (@full) { $outmail->datasend("\nClients with full backups (Might be abnormal)\n"); $outmail->datasend("=" x 78 . "\n"); foreach my $i (@full) { $outmail->datasend("$full{$i}"); } } ### my @abandoned = sort keys %abandoned; if (@abandoned) { $outmail->datasend("\nClients with abandoned save sets:\n"); $outmail->datasend("=" x 78 . "\n"); foreach my $i (@abandoned) { $outmail->datasend("$abandoned{$i}"); } } ###

my @big = sort keys %big; if (@big) { $outmail->datasend("\nClients with large incrementals\n"); $outmail->datasend("=" x 78 . "\n"); foreach my $i (@big) { # print "$i\n"; $outmail->datasend("$big{$i}"); } } ### my @backuptime = sort keys %backuptime; if (@backuptime) { $outmail->datasend("\nClients that took a long time to backup:\n"); $outmail->datasend("=" x 78 . "\n"); foreach my $i (@backuptime) { $outmail->datasend("$backuptime{$i}"); } } ### my @cannotstat = sort keys %cannotstat; if (@cannotstat) { $outmail->datasend("\nClients with files that are too large to backup:\n"); $outmail->datasend("=" x 78 . "\n"); foreach my $i (@cannotstat) { $outmail->datasend(" $cannotstat{$i}"); } } ### my @open = sort keys %open; if (@open) { $outmail->datasend("\nClients with open file errors:\n"); $outmail->datasend("=" x 78 . "\n"); foreach my $i (@open) { $outmail->datasend(" $i\n"); } } ###

my @openvb = sort keys %openvb; if (@openvb && $openvbwarn) { $outmail->datasend("\nClients with open file errors:\n"); $outmail->datasend("=" x 78 . "\n"); foreach my $i (@openvb) { $outmail->datasend(" $openvb{$i}"); } } ### my @timesync = sort keys %timesync; if (@timesync && $badclockwarn) { $outmail->datasend("\nClients with unsynchronized clocks:\n"); $outmail->datasend("=" x 78 . "\n"); foreach my $i (@timesync) { $outmail->datasend(" $i\n"); } } $outmail->datasend("\n\n"); track_store(); $outmail->datasend("\nEnd of report.\n"); $outmail->dataend(); $outmail->quit();

sub db { my ($msg) = @_; print STDERR "DB: $msg\n"; }

sub track_store { my (@noconnect, @failall, @failpart, @success); foreach my $k (keys %trackfail) { if (defined($tracksuccess{$k})) { push(@failpart, $k); }

elsif (defined($tracknoconnect{$k})) { push(@noconnect, $k); } else { push(@failall, $k); } } foreach my $k (keys %tracksuccess) { if (!defined($trackfail{$k})) { push(@success, $k); } } foreach my $i (sort @noconnect) { $outmail->datasend("Could not connect to $i\n"); } foreach my $i (sort @failall) { $outmail->datasend("Total failure to $i\n"); } foreach my $i (sort @failpart) { $outmail->datasend("partial failure for $i\n"); } foreach my $i (sort @success) { $outmail->datasend("successful $i\n");; }

}groups-running-check #! /local/bin/perl # # Script to check if a group is running # (to be run from cron late in the day) # # mussulma - 6 Dec 2001 # 17 Oct 02 - updated # use strict; my $ADMIN = 'youremail@host1.com, youremail2@host2.com, pager@host3.com'; my $SUBJECT = 'Groups running'; my $SERVER = "YOUR_BACKUP_SERVER.domain.com"; open (NW, "/usr/bin/echo 'show name;status\nprint nsr group' | /usr/sbin/nsradmin -s $SERVER -i |") or die "Could not open nsradmin";

my $group = ""; my $status = ""; my $OUT = ""; while (<NW>) { if (/\s+name: (.*);/) { $group = $1; # print "$group\n"; } if (/\s+status: (.*);/ && $1 ne "idle") { $status = $1; #print "$status\n"; #$OUT .= "$group\t\t$status\n"; $OUT .= "$group \n"; } # print "\t\tRAW $_"; } close (NW); if ($OUT) { my $ttime = localtime(time); open (MAIL, "| mailx -s \"$SUBJECT\" $ADMIN") or die "Cannot open mailx"; # Shorted to fit in alphanumeric pager # print MAIL "$ttime, not all savegroups are idle\n"; # print MAIL "=" x 60 . "\n"; print MAIL "$OUT\n"; close (MAIL); } Is there anyway to get start & end times and saveset info that the savegroup reports generates, from some other source, like mminfo? I'm aware of most of the options to mminfo, couldn't find ones that show the above mentioned items. I was wondering if there is anyway to get the above mentioned info without parsing through the savegroup report. Also, I need to export all this to one Excel sheet. I'm working on inserting "commas" between fields. Does anyone have a better way of doing it? Answere:mminfo -q "ssid=X" -r savetime,sscreate,sscomp savetime = saveset creation time (server) sscreate = saveset start time (client) sscomp = saveset completion time (not documented, I assume client)

Also, I need to export all this to one Excel sheet. I'm working on inserting "commas" between fields. Does anyone have a better way of doing it? If using NetWorker 7.x, you can do this by adding after your mminfo query: -xc, for comma separated variable format. e.g., mminfo -q "savetime>=2 weeks ago" -xc, will produce the last 2 weeks savesets in comma separated variable list for easy import into excel. NetWorker Management Console does this, except it gives the elapsed time for the backup. However, if you know the start & the elapsed, you can calculate the end time. It can export in CSV format, so that can be imported to Excel. You can also define a report, and run the report from the command line, so you can schedule it. Is there anyway to get start & end times and saveset info that the savegroup reports generates, from some other source, like mminfo? mminfo -q "ssid=X" -r savetime,sscreate,sscomp savetime = saveset creation time (server) sscreate = saveset start time (client) sscomp = saveset completion time (not documented, I assume client) Also, I need to export all this to one Excel sheet. I'm working on inserting "commas" between fields. Does anyone have a better way of doing it? If using NetWorker 7.x, you can do this by adding after your mminfo query: -xc, for comma separated variable format. e.g., mminfo -q "savetime>=2 weeks ago" -xc, will produce the last 2 weeks savesets in comma separated variable list for easy import into excel. I am desperately trying to write a unix shell script (sh) to get information from the daemon.log file. What I am trying to capture from this file is whenever a tape becomes full it writes to this log

something like this; "media notice: dlt7000 tape abc.123 on /dev/rmt/1ubn if full" I need to capture that message and parse out the name of the tape that I need to feed to the scipt that's to be spawned off. The other issue with this daemon.log file is it doesn't write one file for each day, so I need to make sure that the message I am capturing is from the last 24 hours. Also there are 7 dlt tape drives so I need to capture this event for each tape drive. Use "crontab" to schedule everything. Step 1 Copy the existing /usr/lib/newsyslog to "/nsr/log/rotate_nsr_logs". Edit out the extra BS in this file, but leave a set of entries that pertain to /nsr/logs/daemon.log. You will see what I mean. Edit the crontab file "crontab -e" and add a new entry: "10 3 * * * /nsr/log/rotate_nsr_logs" Editing the crontab is just like using vi, so when you are done, enter ":wq" to exit. Now your /nsr/logs/daemon.log file will rotate every morning at 10 past 3am. If you don't like 10 past 3am, change it. Step 2 Create a shell script that uses "grep" to look for a string and then emails to a user. It has been a long time since I did any shell scripts, but something like this might work: vi /nsr/log/check_if_full.sh #/bin/csh Mail -s "Tape is full" email_address < grep "is full" /nsr/log/deamon.log :wq Change the permissions on the file to make it runable "chmod 755 /nsr/log/check_if_full.sh" and make a new entry in the crontab "15 3 * * * /nsr/log/check_if_full.sh". The "email_address" is any valid address in your domain - maybe yours? Of course, you want to debug the script by running it a few times. And make sure you fake an entry in the daemon.log file to "find" a full tape. Or, Buy a tape library and forget about it. * To Your Posts * Keyword Search * One-Click Access To Your Favorite Forums * Automated Signatures On Your Posts * Best Of All, It's Free! E-mail* Handle Password

Verify P'word *Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail. Tek-Tips LinkedIn Group Join our LinkedIn Group! Hook up with past and present colleagues and classmates quickly. Partner With Us! "Best Of Breed" Forums Add Stickiness To Your Site Partner Button (Download This Button Today!) Member Feedback "...Thanks a lot Mate! I can't tell you how many times your site has saved my "rear". hehe..." More... Geography Where in the world do Tek-Tips members come from? Click Here To Find Out! Tek-Tips Shirts! Get your Tek-Tips Forums SWAG here! Home > Forums > MIS/IT > Operating Systems - UNIX based > Sun: Solaris Forum nsrmm legato networker thread60-1196247 Share This Forum Search FAQs Links Jobs Whitepapers MVPs Tek-Tips Forums is Member Supported. Click Here to donate. bkreynolds48 (TechnicalUser) 24 Feb 06 10:17 I am trying to relabel a tape in a script using nsrmm legato's networker program This is the command I tried: nsrmm -m -l -f /dev/rmt/0n -R 'test2' I get a response back saying I can't use "-l" with "-R" when trying to run the command without the "-R" a prompt comes up asking for a yes/no to relabel tape. when trying without the "-l" I get the syntax incorrect error. Here are the actual error messages: =================================== root# nsrmm -m -R 'test take2' nsrmm: when used with -l, -R excludes the use of a volume name usage: nsrmm [-v | -q] [-s server] [-f device] or nsrmm -l [-v | -q] [-s server] [-f device] [-myB] [-e forever] [-c capacity] [-b pool] [-R | volume] or nsrmm {-u | -j} [-v | -q] [-s server] [-y] [-f device | volume...] or nsrmm -p [-v | -q] [-s server] [-f device] or nsrmm -m [-v | -q] [-s server] [-f device] [-r] [volume] or nsrmm {-d | -o mode} [-v | -q] [-s server] [-Py] [-S ssid[/cloneid] | -V volid | volume...] or nsrmm [-s server] -S ssid {-w browsetime | -e retentiontime} root# nsrmm -R 'test take2'

nsrmm: -R is only valid when used with -m or -l usage: nsrmm [-v | -q] [-s server] [-f device] or nsrmm -l [-v | -q] [-s server] [-f device] [-myB] [-e forever] [-c capacity] [-b pool] [-R | volume] or nsrmm {-u | -j} [-v | -q] [-s server] [-y] [-f device | volume...] or nsrmm -p [-v | -q] [-s server] [-f device] or nsrmm -m [-v | -q] [-s server] [-f device] [-r] [volume] or nsrmm {-d | -o mode} [-v | -q] [-s server] [-Py] [-S ssid[/cloneid] | -V volid | volume...] or nsrmm [-s server] -S ssid {-w browsetime | -e retentiontime} Mike042 (MIS) 24 Feb 06 12:56 We don't use Legato NetWorker on Solaris, but this is what I use in a script on Compaq/HP Tru64 v5.1a : nsrmm -m -l -y -f <tape-drive> -b "<pool-name>" <tapelabel> We are using Legato Networker 6.0.1 for Sun Solaris for backing up Unix & NT servers. Sometimes when the backups are failed, it is very tedious for us to re-run these failed backups manually. So We are looking fo some kind of scripting language / tool which is supported with Legato. Does anyone has information on that. Does anyone know about report generation tool for Legato. The only thing I know of that Legato supports is GEMS, but I've never used it, so I'm not sure of what information that supplies. I use the following process to identify failed backups: 1. When a new system is added, we run a backup, and using mminfo, create a client "profile" of all save sets that should be backed up nightly. (eg /tmp/ntsystem would contain "c:" "d:" "e:" etc...) 2. Every morning, we have a script that runs (again, uses mminfo) and checks to ensure that every save set ran successfully and alerts me to those that didn't. It seems to be the simplest way to check a large number of clients without any manual intervention. Here is in example of what line is run for every client: mminfo -c $c -q "savetime > 12 hours ago" -v -ot | grep -v cE | grep -v ci | awk '{print $10}' | awk -F\\ '{print $1}' >& /tmp/save_$c I'll go through it a bit.. The $c is a variable fed from a list that contains all clients on that backup server.. This command runs in a foreach loop. It then looks for all save sets that have run in the last 12 hours that have not ended in error, and are not still running. The awk commands take the 10th field (which is the save set) and discounts the "\" at the end of the save set name (I have had problems grep'ing for anything with backslashes at times) and dumps the contents in a file. The end result looks something like: Registry: Repairdisk: C: D: F: I then compare the contents of that file with the config file created after the initial backup. If anything in the original config file is not found to have been backed up in the last 12 hours, it reports a failure.

EMC Legato Networker Admin Tools Skip to end of metadata * Page restrictions apply * Attachments:1 * Added by David Mussulman, last edited by David Mussulman on Aug 29, 2007 (view change) Comment: Go to start of metadata DCS started using Legato (now EMC) Networker as a centralized, diverse OS supporting backups solution in 1999. It replaced Budtool, a previous product purchased by Legato and incorporated into Networker, and various ufsdump/tar scripts on production services. Over the years, additional scripts and reports were written by TSG to help maintain the Networker installation. These were developed as needs arised, and have been worked on by full time programmers and student admins. The Networker listservhas been a valuable resource for the creation and sharing of these lists. These scripts and Networker are supplemented with an old isql database that contains what the backup state of systems should be (not in backups, in backups, out-of-service, etc.) and their cluster/group association in the department. Another file associates those clusters with contact information for backups notices. Response reports (triggered by Networker action) * Savegroup reports are parsed through a perl script that mails the original Networker report unchanged, and mails a summary report following it. (Attached as email-saveset.txt) * Bootstrap messages are delivered to two different hosts and stored locally, in addition to being mailed to the backupsmgr. * Tape mount requests are piped through a script (tape-mount-notify.pl) that checks to see if the tape is in the jukebox, and if not, sends email to the pagers. Daily reports (triggered by cron/scheduler) * Media diff report * Client backup check (lists activity/inactivity of clients and backups status) * List of failed client group memberships * Report on laptops that have not backed up in the last 7 days, fulls older than 4 weeks * Client consistency checks (checks the client configs in Networker (matches OS to directive, confirms proper group memberships, client instance counts, retention dates, save sets, remote access, aliases, etc.) * groups-running-check - sends email to admins (and pager) when backup groups are running. Runs at 9am, 4pm, 10pm. * quantity-report.pl - tells me how much each client backed up in a given timeframe, totals, averages, etc. Manual reports * incview - view the contents of the last backup for a client/saveset * incview.pl - view the contents of the last backup for a client/saveset ordered by descending file

size * incview-ssid - view the contents of a backup by ssid * new-report - generates a CSV list of Networker information, merged with database information that can be viewed in Excel or used as a source for other reports * session-report - about the only place to get aggregate information about restores. The downside is you need to store the data outside of Networker, because if Networker restarts, these counters get reset. Tape operations scripts * tapeview - shows the unexpired savesets written to a past tape (if none, the tape is or can be recycled) * start-group - starts a savegroup from the command line Research group notification reports * email-reports - sends email to research groups about all systems in backups * email-failure-reports - sends email to research groups about systems failing backups Networker tools I use regularly * nsrjb - control the jukebox operations * nsrwatch - command line, ncurses view of Networker admin screen quantity-report.pl Looks like this: bash-3.00$ ./quantity-report.pl 'today' | head -16 156 GB total over today (185 hosts, ave 846,782,216) arch-backup 17,344,251,424 11.07 1,238,875,102 ave, 14 backups, 38085 files yangtze 11,275,808,832 7.20 2,255,161,767 ave, 5 backups, 474 files dcshome 7,492,194,692 4.78 1,498,438,939 ave, 5 backups, 181125 files seclab-tehran2 5,496,115,020 3.51 916,019,171 ave, 6 backups, 14799 files parrot 5,357,190,948 3.42 1,339,297,738 ave, 4 backups, 257 files bane 5,124,311,364 3.27 1,281,077,842 ave, 4 backups, 2904 files edinburgh 4,486,743,876 2.86 1,495,581,293 ave, 3 backups, 171510 files dcsfiles 3,855,078,660 2.46 428,342,074 ave, 9 backups, 52577 files dcs-video1 3,827,776,708 2.44 273,412,622 ave, 14 backups, 16254 files choi 3,814,584,240 2.44 1,271,528,081 ave, 3 backups, 218 files uoficont 3,646,335,748 2.33 520,905,107 ave, 7 backups, 192 files msl 3,402,629,840 2.17 486,089,978 ave, 7 backups, 2976 files charm 3,021,806,820 1.93 1,007,268,941 ave, 3 backups, 30156 files dcs-maillist 2,972,135,472 1.90 495,355,912 ave, 6 backups, 4967 files dcs-kickstart 2,716,551,552 1.73 388,078,794 ave, 7 backups, 1228 files I usually kick it off in a cron job that sends email once a day. It runs this script: #! /bin/sh echo echo "Today" echo /usr/dcs/backups/devel/dailyuse/report.pl 'today' | head -16

echo echo "Last week" echo /usr/dcs/backups/devel/dailyuse/report.pl 'last week' | head -16 The script itself #! /usr/bin/perl # # Generate a report of backed up quantities by system over a given # timeframe. Nice to see how much is really backing up in a day. # # July 11, 2007 - mussulma@uiuc.edu # use Statistics::Descriptive; $SERVER = "your_servername"; $MMINFO = "/usr/sbin/mminfo"; if ($ARGV[0] eq '') { die "Needs time range as argument (mminfo -t arg)\n"; } $MMINFO_ARGS = "-s $SERVER -xc, -t '$ARGV[0]' -r 'client,name,level,volume,savetime,totalsize(15),nfiles,ssid'"; open (MM, "$MMINFO $MMINFO_ARGS |") or die; while (<MM>) { $line = $_; chomp($line); my ($client, $name, $level, $volume, $savetime, $totalsize, $nfiles, $ssid) = split (/,/, $line); next if ($client eq "client" and $name eq "name"); # header line # next unless ($client eq "dcs-media1"); # print "$client $totalsize --- $line\n"; # next if ($client eq "dcs-iisweb1"); # # SSIDs that cross tapes show up twice. Keep a cache of what we've # seen so we only count it once. # next if (defined($SEENSSID{$ssid})); $SEENSSID{$ssid} = 1; $HOSTS{$client} += $totalsize; if (!defined($HOSTSTAT{$client})) { $HOSTSTAT{$client} = Statistics::Descriptive::Full->new(); }

$HOSTSTAT{$client}->add_data($totalsize); # $DRIVES{"$client_$name"} += $totalsize; $overall += $totalsize; $NFILES{$client} += $nfiles; } close (MM); $totalGsize = int($overall / 1000 / 1000 / 1000); #$totalGsize = int($overall); #$HOSTS{"Total of $HOSTS hosts"} = $overall; $totalhosts = scalar keys(%HOSTS); $avehosts = int($overall / $totalhosts); print commify($totalGsize) . " GB total over $ARGV[0] ($totalhosts hosts, ave " . commify($avehosts) . ")\n";

foreach $i (sort { $HOSTS{$b} <=> $HOSTS{$a} } keys %HOSTS) { $foo = $HOSTSTAT{$i}; # print "$i\t" . commify($foo->sum()) . "\t" . ($foo->sum() / $overall) . "\n"; printf "%-25s %20s %5.2f %20s ave, %2i backups, %i files\n", $i, commify($foo->sum()), ($foo->sum() / $overall * 100), commify(int($foo->mean()+1)), $foo->count(), $NFILES{$i};

# "\t" . $foo->count() . "--" . commify($foo->trimmed_mean) . "\n"; # $Gsize = int($HOSTS{$i} / 1000 / 1000); # # print "$i " . commify($Gsize) . "\n"; }

sub commify { # commify a number. Perl Cookbook, 2.17, p. 64 my $text = reverse $_[0]; $text =~ s/(\d\d\d)(?=\d)(?!\d*\.)/$1,/g; return scalar reverse $text; } Labels parameters Labels

Media diff report My media diff report keeps a daily snapshot of the media database and compares today's with yesterday's (and sends changes.) It needs some setup work before it can be run. 1. Create a directory for the working files for this script. I use /usr/dcs/backups/mmreport 2. Setup a cronjob to run this daily #! /bin/sh date=`date +%d` cd /usr/dcs/backups/mmreport /usr/sbin/mminfo -m -s dcs-backups1 > new.txt /usr/sbin/mmlocate -s dcs-backups1 >> new.txt diff orig.txt new.txt > diff.txt result=$? if [ $result -eq 0 ]; then rm new.txt rm diff.txt echo "$date report is the same" >> log.txt exit; fi echo "$date new info! mailing mussulma" >> log.txt echo "New mm info:" > mail.txt echo "Diff is:" >> mail.txt cat diff.txt >> mail.txt echo >> mail.txt cat new.txt >> mail.txt cat mail.txt | mailx -s "[DCS-backups] media diff report" YOUR_EMAIL@YOURDOMAIN.COM mv orig.txt $date-orig.txt mv new.txt orig.txt rm diff.txt Labels parameters Labels Enter labels to add to this page: Please wait Looking for a label? Just start typing.

incview
Run it as 'incview clientName' to see a listing of what backed up last for the client. Can nail down 'incview clientName partitionName' but sometimes the quoting gets weird. ie: incview Dave "C:\" #! /bin/sh

if [ ! -z "$2" ]; then SLICE="-N $2" else SLICE="" fi STIME=`mminfo -s YOURSERVER -r nsavetime -v $SLICE -c $1 -t "yesterday" -ot | tail -1` nsrinfo -s YOURSERVER -V -t $STIME $1 Output looks like: bash-3.00$ incview endeavour /var scanning client `endeavour' for savetime 1162976731(Wed Nov 8 03:05:31 2006) from the backup namespace on server YOURSERVER /var/run/utmp, size=17836, off=0, app=backup(1) /var/run/, size=1116, off=17836, app=backup(1) /var/lock/, size=336, off=18952, app=backup(1) /var/lib/php/session/sess_2a8421e28688b76e9d71e6f2257702a4, size=532, off=19288, app=backup(1) /var/lib/php/session/sess_06231f4c099f7a27ea38d063e49ae710, size=532, off=19820, app=backup(1) /var/lib/php/session/sess_bd787427df2c1b13cb1586cd2ae3c3f5, size=632, off=20352, app=backup(1) /var/lib/php/session/sess_db16d5895773b631866e55cec06ff1a9, size=632, off=20984, app=backup(1) ... /var/spool/mqueue/, size=252, off=809912, app=backup(1) /var/spool/clientmqueue/, size=256, off=810164, app=backup(1) /var/spool/, size=548, off=810420, app=backup(1) /var/, size=628, off=810968, app=backup(1) /, size=768, off=811596, app=backup(1) 73 objects found Labels parameters Labels Enter labels to add to this page: Please wait Looking for a label? Just start typing.

incview.pl
This one is similar to incview, but sorts the output by descending file size. Handy for seeing what made a particular backup so large (.ISOs, ripped DVDs, etc.) #! /local/bin/perl if ($ARGV[1] eq "") { $SLICE = ""; } else { $SLICE = "-N $ARGV[1]"; } print "Getting savetime information for $ARGV[0] $ARGV[1] ... "; open(MM, "mminfo -s YOURSERVER -r nsavetime -v $SLICE -c $ARGV[0] -t yesterday -ot |");

while(<MM>) { $STIME = $_; chomp($STIME); } print STDERR "OK\n"; print "Getting index information for that save time (this could take a while) ... \n"; open (NSR, "nsrinfo -s YOURSERVER -V -t $STIME $ARGV[0] |"); while(<NSR>) { $line = $_; if ($line =~ /^(.+), size=(\d+),/) { $file = $1; $size = $2; #$DISPLAY{$size} .= "$file\n"; push(@{$DISPLAY{$size}}, $file); } #print STDERR "$file\t\t\t$size\n"; } my $runningtotal = 0; foreach $i (sort {$b <=> $a} keys %DISPLAY) { my $val = $i; $val = CommaFormatted($val); if ($ARGV[2] ne "" and $ARGV[2] > $i) { print "ENDING EARLY\n"; exit; } foreach my $f (@{$DISPLAY{$i}}) { $runningtotal += $i; $printrt = CommaFormatted($runningtotal); print "$val\t($printrt total)\t$f\n"; } } sub CommaFormatted { my $delimiter = ','; # replace comma if desired my($n,$d) = split /\./,shift,2; my @a = (); while($n =~ /\d\d\d\d/) { $n =~ s/(\d\d\d)$//; unshift @a,$1; } unshift @a,$n; $n = join $delimiter,@a; $n = "$n\.$d" if $d =~ /\d/; return $n; }

Looks like: bash-3.00$ incview.pl endeavour /var Getting savetime information for endeavour /var ... OK Getting index information for that save time (this could take a while) ... 269,368 (269,368 total) /var/log/sa/sar07 264,268 (533,636 total) /var/log/sa/sa07 46,636 (580,272 total) /var/log/wtmp 46,136 (626,408 total) /var/log/maillog 39,868 (666,276 total) /var/log/cron ... Labels parameters Labels Enter labels to add to this page: Please wait Looking for a label? Just start typing.

incview-ssid
My other incview and incview.pl scripts look at just the most recent backup. Sometimes it's helpful to look at older backups. I use the nwadmin GUI or mminfo to get an SSID, and then give the SSID to this program. This one's in perl, but could be in shell ... It just shows the inconsistencies in how these were written. #! /usr/bin/perl $STIME=`/usr/sbin/mminfo -s YOURSERVER -xc, -r client,name,nsavetime -q ssid=$ARGV[0] | tail -1`; chomp($STIME); print $STIME . "\n"; ($client, $name, $ntime) = split(/,/, $STIME); print `/usr/sbin/nsrinfo -s YOURSERVER -V -t $ntime $client`; Looks something like this: bash-3.00$ mminfo -q 'client=endeavour,name=/var' -s YOURSERVER -t '-1 day' -r 'ssid,savetime' ssid date 1951455101 11/07/06 3964771803 11/08/06 bash-3.00$ incview-ssid 3964771803 endeavour,/var,1162976731 scanning client `endeavour' for savetime 1162976731(Wed Nov 8 03:05:31 2006) from the backup namespace on server YOURSERVER /var/run/utmp, size=17836, off=0, app=backup(1) /var/run/, size=1116, off=17836, app=backup(1) /var/lock/, size=336, off=18952, app=backup(1) /var/lib/php/session/sess_2a8421e28688b76e9d71e6f2257702a4, size=532, off=19288, app=backup(1) /var/lib/php/session/sess_06231f4c099f7a27ea38d063e49ae710, size=532, off=19820, app=backup(1) /var/lib/php/session/sess_bd787427df2c1b13cb1586cd2ae3c3f5, size=632, off=20352, app=backup(1) ... Labels parameters Labels Enter labels to add to this page: Please wait Looking for a label? Just start typing.

session-report
#! /bin/sh nsradmin -s YOURSERVER -i - <<EOF show recover totals;save totals;totals since print type:nsr EOF yields results like: save totals: "5870 session(s), 10 TB total"; recover totals: 0 session(s); totals since: "Sat Nov 4 22:47:21 2006"; Run this daily and save it to a file to get an idea of how many sessions/sizes you've experienced over a given timeframe for backups and restores.