I take a nightly backup from my Webhosting service as .tar.gz. The Data changed is very little each night (<10MB).
I would like to archive these files and deduplicate (solid compression) files across multiple .tar.gz archives without extracting each individual backup file.
Simplified example:
2013-02-24.tar.gz
contains a.pdf
2013-02-25.tar.gz
contains a.pdf
and b.pdf
I want to compress and archive 2013-02-24.tar.gz
and 2013-02-25.tar.gz
and store a.pdf
only one time.
The only way I got the deduplication working was to unpack both .tar.gz and pack them again (using solid compression). But this is one additional step and it will probably erase filesystem information due to different OS's. Simply tar/gz both .tar.gz files won't work, the final size of this archive will be the sum of both original .tar.gz files.
Is there a way use deduplication across the contents of multiple .tar.gz files?
2013-02-24.tar.gz
and2013-02-25.tar.gz
if you only want one copy ofa.pdf
? What is the point of having different archives for different days unless you want to be able to recreate the exact state of 2013-02-24? Please tell us what exactly you are trying to do, if you just want a safe backup where files deleted on the server are not deleted in the backup, there are easier ways to do this. Please update your question stating your end goal. Anyway, whatever your goal,rsync
is your friend.