I used this command:

rsync -avP /source1/imgs /backup/  

to copy a directory onto another directory on the same server. It's about 18 TB of data. 23 million files. For building files only it took more than 3 days.

Now that I have a backup of my production directory, my question is: How can I update the backup directory with the changes happened on the production directory?


I such large filesystems, file based backup is not efficient, esp if that file system is active and changes are made often, as during the time it takes backup the data, it changed a lot of times, so it will be hard for you to determine which file copied when.

You might want to consider a real time back up, like using a distributed filesystem or a distributed block.

I personally use drbd, And I've set up lvm on top of it and using snapshots.

You can also setup a storage based on ZFS (I liked Nexenta), ZFS have a great snapshots features and there are few options to setup a live replication, if think one of them called luster, never tried that.


rsync to it again. rsync will do its best to figure out what has changed and only synchronize that. You can pass -n to perform a dry run so that you can verify.

  • @lgnacio: my problem is also building more than 23millions files taking lots of memory and time.. your answer mean i have to add -n to myt rsync command? – AlBouazizi May 6 '11 at 11:24

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.