Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm trying to come up with some backup system which would allow me to transfer the minimal amount of data every time a new backup is made.

Currently, I'm doing an incremental backup via duplicity over to an smb mount. I'm limited with that to around 1MB/s at the moment (wifi/ndiswrapper being silly, but still - I'm not going to get more than 5x speedup by fixing it).

Unfortunately the removed, modified, etc. data makes it hard to keep the incremental backups forever. I can't easily flatten them to a full backup and taking a full copy of 40+GB is taking a bit long now.

There are a couple of things I'd like to achieve with the new system and a couple that I don't care about much:

  • I'd like to have a couple of checkpoints, but probably not more than 2 or 3
  • I don't care about encryption since the the storage is located on local network
  • Compression would be good, but I guess I can just use filesystem/device compression
  • I want to be able to completely get rid of all but the latest backup without forcing me to do a full copy again

This seems to be doable with rsync + loop device compression + fs snapshotting... but unfortunately my device doesn't really support snapshots. It's just got the standard debian's list of supported filesystems and I don't really want to use btrfs for backups.

Are there any comparable solutions? How else can I approach it?

Edit: I think that the snapshot part can be actually done with lvm read-only snapshots. That would also give another layer of protection from badly-behaving apps.

share|improve this question
add comment

1 Answer

rsnapshot does incremental backups through the use of hardlinks and rsync - if the file is the same as the last time the backup was run, then it just hardlinks it. So you if you changed nothing and ran rsnapshot every hour for three hours you would have three folders:

hourly.0
hourly.1
hourly.2

Each would appear to contain a "full" backup, but only one copy of each file would reside on the disk. This means, if you delete hourly.2, hourly.1 and hourly.0 would still contain all files. You can set your backup frequency and number of snapshots.

This would achieve your first goal, however the destination file system must support hardlinks for this to work.

An alternative to this woud be to use rsync + LVM on the backup device, and take snapshots periodically. LVM snapshots are "copy-on-write" volumes, where they remain identical to the original by re-using the same blocks as the snapshotted volume, unless blocks change, whereupon the the snapshotted volume just stores the blocks that are different. It can therefore be sized to match the rate of change of the volume being snapshotted between backups.

Compression would be good, but if you don't want to use btrfs, then yeah some sort of file system compression via a loop or fuse would be needed.

share|improve this answer
    
I've seen rsnapshot, but I'm a bit worried about it really. I'm quite sure there's something they could improve since 2008, but there doesn't seem to be any new version since then. –  viraptor Apr 12 '12 at 1:47
    
I added an lvm suggestion. Note that rsnapshot is pretty widely used. –  Paul Apr 12 '12 at 2:03
1  
rsnapshot rocks. Maybe that is why there have been no updates, it does the job very well. –  stefgosselin Apr 12 '12 at 3:56
    
LVM snapshots can't really be used that way. You can't really have more than one snapshot ( performance degrades exponentially ). LVM snapshots are meant for temporary use so you can get a consistent image to backup, then throw out the snapshot, not for history archival. –  psusi Apr 12 '12 at 14:26
    
@psusi Yeah, I was considering the small number of snapshots and that the backup system wouldn't be used for real-time writing of files. It isn't ideal I know. –  Paul Apr 12 '12 at 23:09
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.