I'm using linux on desktop since about 1996. Every time I changed my desktop or notebook, I've taken my linux with me via "dump | ssh | restore".
Along the way, I went through half a dozen HDD failures. I've been lucky - most of the time I was able to salvage everything to the new replacement HDD, and only once I've lost my /usr partition.
I want to have a backup just in case next time I would not be so lucky. I'm perfectly happy in having always-connected backup on the separate computer in more or less the same physical location. I'm reluctant to have a separate external HDD for backups, because I would forget to plug it in now and then.
I've tried to use home-grown tar/rsync based scripts, storebackup, bacula, duplicity and boxbackup and several others I already forgot, but all of them have their flaws (more on them later)
I've been thinking, and here are my requirements:
Backup should be able to operate via slow (802.11g) network connection
Backup should be disk-based (no CDs/DVDs, no tape)
Backup should be incremental
Removal of older backups should be easy (not a "just make a new full backup and remove older increments")
Backup should support hardlinks and other special linux files (I have a lot of mercurial and darcs repos with loads of hardlinks between them)
Changes detection algorithm should be robust and reliable
Restore should bring back the files and dirs that look like the files that were backed up (including atime/mtime)
Large files with small changes should be included partially in the increments (think of virtual machine disk images - when some sectors have changed, you don't want incremental backup copying all 20Gb of it)
Now, why am I not happy with existing solutions? Just a few examples:
rsync-based solutions either don't handle hardlinks or use enormous amount of memory to handle them via "rsync -H"
rsync-based solutions look at mtime+size to determine if the file was changed. This fails when you are using mtime-preserving file-modifying software
Classical incremental backups, like bacula, cannot remove older versions of backups without doing new full backups => spaces goes to waste.
duplicity makes a clever use of rsync-like algorithm to transfer only changed portions of file, but it keeps the full copy of the oldest version + rsync diffs on top, which makes removal problematic. Boxbackup does it another way - full copy of the most recent version + diffs to previous ones, so older versions could be removed with ease. However, both of them does not support hardlinks. Duh
block- or fs-level tools like dump or dar do not play well with limited network bandwidth, and making incremental backups takes ages.
Lots of backup tools pay no attention to directories on restore, and directories end up with mtime=today.
I hate to think that I have to sit and write Yet Another Backup Tool :)
I'm sure that I've missed something. Please, give me a hint!