Using a smaller drive to mirror a larger server to an offsite backup?

Question

Say I have a 10TB array that I wish to backup to another 10TB array that is off site. I have a 500GB portable drive that I can use to move the data around. The data is static. How can I copy it over a chunk at a time on Linux? I realize this means around 20 trips.

Answer 1

There are several options that will depend upon your filesystem, the size and the nature of the data.

1. Use dump/restore. These are rock solid and will allow you to back up the file in chunks. While I can't give you the exact command you'll need, read through the dump and restore man pages and look online for tutorials which should give you a good idea of how to proceed.

2. If the array data is genuinely static and won't change from day to day, then use dd which will take a byte for byte copy and supports offsets. simply provide a skip=offset when reading from the original array's device and a seek=offset when writing to the copy. For example:

   origin$ dd if=/dev/array_device of=/dev/protable_device ibs=512 skip=3 count=1
   # travel home...
   destination$ dd if=/dev/portable_device of=/dev/array_device obs=512 seek=3 count=1
   

   Obviously, you'll want to copy more than 512 bytes every trip, but you get the idea. Once again, the manpage is fairly good and will be useful. It's recommended that you umount your array while doing this so that nothing new get written in. (alternatively, mount as read-only.)

   You can also pipe dd into bzip2 to minimise space taken:

   dd if=/dev/some_device | bzip2 -c >file.gz
   <file.gz bzip2 -d -c | dd of=/dev/some_device
   

   I'd recommend that you write to a file and use checksums to verify integrity.

3. Copy directories at a time while keeping a check-list of directories transferred so far. Having transferred several TBs in this manner myself, I think that this is surprisingly manageable.