rsync-based backup

I suffered an external USB failure last year;  didn’t lose much but I had to work hard to recover the data as my backup scheme was compromised.

I resolved to correct the issue.  Here are the pieces of my current, tested, and functional solution:

  1. store as much data as possible on external usb drives instead of local machines.  Makes upgrading and adding/changing workstations much easier, and simplifies backup logic.
  2. use NAS devices to mount drives when possible.  Inexpensive NAS like the Cirago NUS1000 will do fine;  it will mount 4 USB drives and has a neat web interface.  Theoretically it will also act as a print server but haven’t got that working yet.
  3. each USB drive gets a SAFE subdirectory.  Any items in this subdir will be mirrored to another drive.   Corollary:  any items outside this directory are not mirrored and are considered disposable or temporary.  This sounds annoying, but I’m getting constant feedback from the directory structure about what is destined for backup and what is not.  In practice it’s not annoying at all.
  4. each USB drive also gets a MIRRORS subdirectory, wherein SAFE subdirs from other drives are mirrored in a subdir named after the drive that’s being mirrored.
  5. small, critical files are encrypted and mirrored into an offline backup – dropbox is what most people use but using Ubuntu One for now.
  6. a script (mirror.sh) calls rsync to check the status of all the SAFE files and their counterparts in the MIRRORS directories.  Tested working in actual use.
  7. if you need to wipe/replace/reformat/repartition a drive, run the mirror script one last time (making no further changes), wipe the drive, then copy over the MIRRORS copy of the files back to SAFE.  Tested working in actual use.

A simplified version looks like this:

DriveA – {misc unimportant stuff}, SAFE/subdirs, MIRRORS/DriveB/subdirs

DriveB – {misc unimportant stuff}, SAFE/subdirs, MIRRORS/DriveC/subdirs

DriveC – {misc unimportant stuff}, SAFE/subdirs, MIRRORS/DriveA/subdirs

Here’s my last actual run.  rsync checked 79,532 files and updated (diff-style, not complete replacement) about 25:

time mirror.sh 
{output redacted for clarity}
real 3m59.010s
user    0m2.020s
sys     0m4.248s

4mins to sync up ~80k files across four drives (3 usb and one internal drive).  This solution is meeting my needs nicely.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s