Lab Backups

From Biowiki
Jump to: navigation, search

Lab backups

---

Backups of user data

All users are responsible for their own backups. There is only one thing I will guarantee: if you put stuff on the NFS (or on lorien:/home/, which is the same thing), it will get backed up every night.

So, I recommend backing up your stuff daily to the NFS. rsync is good for this, described here: Backups Using Rsync

Then, the NFS RAID will get backed up nightly at 9:30PM, as described here: TSMBackupSystem

Backups of server data

Overall process

All servers use a 2-step process outlined below.

Yes, it would be more efficient, robust, and glorious to use rsync for this. However, because none of the servers have rsync installed, I went with the above solution, which was:

  1. deemed faster to implement at the moment, and
  2. intended to be temporary.

Of course, it turned out to be slower to implement, and has now become permanent. Someone should take a crack at this with rsync.

Step 1: Backup is made on the server

There is one nightly cronjob for each user whose data is being backed up. Each cronjob runs a wrapper script, which in turn runs a set of backup scripts.

The wrapper script:

  1. triggers the backup scripts, logging their output
  2. e-mails the logs to the sys admin (the e-mail subject is tagged server-side logs)

Each backup script:

  1. cleans up zipped tarball from the previous iteration
  2. tars and zips (using bzip2) the data for backup, with lots of verbose output
  3. places the zipped tarball in a private dir

Why have a wrapper around the backup scripts? Because it was a quick and dirty fix of certain annoyances. In retrospect, you are better off putting the contents of the wrapper script, separated by semicolons, into the cronfile itself.

Step 2: Backup is copied to the NFS RAID

Early in the morning, after all the servers presumably prepared their zipped tarballs of data to back up, the NFS RAID machine logs into each server and downloads them.

This is done by a cronjob under user avu running the script /home/avu/cron/backup-wrapper.bash. This is another wrapper script, which triggers an individual script for each server, logging their output and then e-mailing the logs to the sys admin (the e-mail subject is tagged RAID-side logs). The individual scripts use scp to retrieve the zipped tarball.

The zipped tarballs of backed-up data are written to /home/users/avu/backups/. We store the last 3 days' worth of backups. The extra backups are deleted by the individual backup scrips.

Note that this means that the backups themselves are backed up from the NFS RAID to the TSM system (see TSMBackupSystem).

How to restore the backups

ssh lorien
su -
cd /home/users/avu/backups/

There you will see a directory for each server being backed up. Just get the zipped tarball you want (they are labelled with the date of the backup) and copy it elsewhere to unpack it, e.g.:

cd buffy/
cp 2007-01-10_cvs_backup.tar.bz2 /tmp/
cd /tmp/
bunzip2 2007-01-10_cvs_backup.tar.bz2
tar xvf 2007-01-10_cvs_backup.tar

Make sure to untar as root, because this way the ownership of the backed-up files is preserved. Otherwise, the ownership will be changed to that of the user untarring the file! The file timestamps will be preserved no matter which way you use.

How to change the e-mail address where the logs are sent

Change the value of the $CONTACT variable in the beginning of each wrapper script on the NFS RAID and on the backed-up servers. You can also add a -c cc-addr part to that variable if you want to CC others, e.g.:

CONTACT="sys.admin@someplace.edu -c other.person@someplace.edu"

Server-specific info

anya

cronjob info:

su - avu
crontab -l
  00 01 * * * /home/avu/cron/backup-wrapper.bash

backup scripts:

  • /home/avu/cron/backup-biowiki.bash
    • backs up:
      • /var/www/twiki/
      • /home/yam/public_html/
      • /home/avu/cron/
      • /home/avu/public_html/

logs are kept in:

  • /home/avu/cron/logs/

buffy

cronjob info:

su - yam
crontab -l
  00 01 * * * /home/yam/cron/backup-wrapper.sh

su - avu
crontab -l
  30 01 * * * /home/avu/cron/backup-wrapper.bash

backup scripts:

  • /home/yam/cron/backup-YAM-CVS.tcsh
    • backs up:
      • /home/yam/YAM-CVS/
      • /home/yam/cron/
  • /home/avu/cron/backup-cvs.bash
    • backs up:
      • /usr/local/cvs/
      • /home/avu/cron/
  • /home/avu/cron/backup-svn.bash
    • backs up:
      • /usr/local/svn/

logs are kept in:

  • /home/yam/cron/logs/
  • /home/avu/cron/logs/

wesley

cronjob info:

su - yam
crontab -l
  00 01 * * * /home/yam/cron/backup-wrapper.sh

backup scripts:

  • /home/yam/cron/backup-mail.tcsh
    • backs up:
      • /home/yam/mail/
      • /home/yam/cron/

logs are kept in:

  • /home/yam/cron/logs/

---

-- Created by: Andrew Uzilov on 10 Jan 2007