Could you suggest feature-rich disk-based personal backup program for linux (and I've seen a few)?

Question

I'm using linux on desktop since about 1996. Every time I changed my desktop or notebook, I've taken my linux with me via "dump | ssh | restore".

Along the way, I went through half a dozen HDD failures. I've been lucky - most of the time I was able to salvage everything to the new replacement HDD, and only once I've lost my /usr partition.

I want to have a backup just in case next time I would not be so lucky. I'm perfectly happy in having always-connected backup on the separate computer in more or less the same physical location. I'm reluctant to have a separate external HDD for backups, because I would forget to plug it in now and then.

I've tried to use home-grown tar/rsync based scripts, storebackup, bacula, duplicity and boxbackup and several others I already forgot, but all of them have their flaws (more on them later)

I've been thinking, and here are my requirements:

1. Backup should be able to operate via slow (802.11g) network connection

2. Backup should be disk-based (no CDs/DVDs, no tape)

3. Backup should be incremental

4. Removal of older backups should be easy (not a "just make a new full backup and remove older increments")

5. Backup should support hardlinks and other special linux files (I have a lot of mercurial and darcs repos with loads of hardlinks between them)

6. Changes detection algorithm should be robust and reliable

7. Restore should bring back the files and dirs that look like the files that were backed up (including atime/mtime)

8. Large files with small changes should be included partially in the increments (think of virtual machine disk images - when some sectors have changed, you don't want incremental backup copying all 20Gb of it)

Now, why am I not happy with existing solutions? Just a few examples:

1. rsync-based solutions either don't handle hardlinks or use enormous amount of memory to handle them via "rsync -H"

2. rsync-based solutions look at mtime+size to determine if the file was changed. This fails when you are using mtime-preserving file-modifying software

3. Classical incremental backups, like bacula, cannot remove older versions of backups without doing new full backups => spaces goes to waste.

4. duplicity makes a clever use of rsync-like algorithm to transfer only changed portions of file, but it keeps the full copy of the oldest version + rsync diffs on top, which makes removal problematic. Boxbackup does it another way - full copy of the most recent version + diffs to previous ones, so older versions could be removed with ease. However, both of them does not support hardlinks. Duh

5. block- or fs-level tools like dump or dar do not play well with limited network bandwidth, and making incremental backups takes ages.

6. Lots of backup tools pay no attention to directories on restore, and directories end up with mtime=today.

I hate to think that I have to sit and write Yet Another Backup Tool :)

I'm sure that I've missed something. Please, give me a hint!

Answer 1

Well, I went through the alternatives once again, and settled for rdiff-backup for now. Last time I checked it (many years ago) it was in the very primitive state, but right now it seems to be full-grown, with good wiki, support in 3rd-party tools etc.

To recap, how my requirements are met:

1. Backup should be able to operate via slow (802.11g) network connection - While initial full copy takes a lot of bandwidth, it only happens once, and after that bandwidth consumption is pretty modest. After initial set up, ~150 GB of disks are synced in less than an hour over 802.11g wifi.

2. Backup should be disk-based (no CDs/DVDs, no tape) - Check

3. Backup should be incremental - Check

4. Removal of older backups should be easy (not a "just make a new full backup and remove older increments") - Check. Rdiff-backup stores reverse rdiff-based patches (from most recent versions to older ones), so you just remove the oldest diffs and you are done (rather, you launch rdiff-backup with some switches and it does this for you).

5. Backup should support hardlinks and other special linux files (I have a lot of mercurial and darcs repos with loads of hardlinks between them) - Check. Hardlinks are supported, and I don't need anything else.

6. Changes detection algorithm should be robust and reliable - To be honest, I haven't checked how rdiff-backup plays with mtime, but quick search on the wiki leads me to believe that this requirement is not met. Oh well, let's chalk it up to "nothing is perfect"

7. Restore should bring back the files and dirs that look like the files that were backed up (including atime/mtime) - Check. Mtimes are restored.

8. Large files with small changes should be included partially in the increments (think of virtual machine disk images - when some sectors have changed, you don't want incremental backup copying all 20Gb of it) - Check. librsync takes care of that.

So, for now, I am happy :)

Answer 2

I've heard good things about BackInTime. It might be worth a look but I don't know how it handles hard links. There is a discussion on the site saying it works but size isn't reported properly.

Answer 3

For my laptop I use SpiderOak, which is a service but has its own backup tool that allows you to prune backed-up files stored there.

For my server I use duplicity and S3. I must admit that while some space goes to waste (well, some data is duplicated) with standard 'classical incremental backups' that bacula and duplicity do... disk is cheap. I do a full backup once per quarter and nightlies after that. I keep two full quarters of backups plus the current quarter-in progress, so from 6-9 months of backups at any given time. Yes, this means that if a file didn't change I have three copies of it, but I'm afraid I've got better ways to spend the $.50 worth of time that's costing me.

Added: re: duplicity and hard links: A discussion on the duplicity mailing list says, in part:

This doesn't directly answer your question, but: we use duplicity to
back up a FreeBSD system. Before doing to back up, we find all the hard
links on the file system. We exclude the duplicate hard links using
--exclude-file and keep a list of them in the backup. After a restore,
we use the list to restore the hard links. It's a bit of a hack, but it
seems to work really well for us.

So there's another option for you.