Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I take a nightly backup from my Webhosting service as .tar.gz. The Data changed is very little each night (<10MB).

I would like to archive these files and deduplicate (solid compression) files across multiple .tar.gz archives without extracting each individual backup file.

Simplified example:

2013-02-24.tar.gz contains a.pdf 2013-02-25.tar.gz contains a.pdf and b.pdf

I want to compress and archive 2013-02-24.tar.gz and 2013-02-25.tar.gz and store a.pdf only one time.

The only way I got the deduplication working was to unpack both .tar.gz and pack them again (using solid compression). But this is one additional step and it will probably erase filesystem information due to different OS's. Simply tar/gz both .tar.gz files won't work, the final size of this archive will be the sum of both original .tar.gz files.

Is there a way use deduplication across the contents of multiple .tar.gz files?

share|improve this question
    
Why is this "deduplication" important? Why do you need both 2013-02-24.tar.gz and 2013-02-25.tar.gz if you only want one copy of a.pdf? What is the point of having different archives for different days unless you want to be able to recreate the exact state of 2013-02-24? Please tell us what exactly you are trying to do, if you just want a safe backup where files deleted on the server are not deleted in the backup, there are easier ways to do this. Please update your question stating your end goal. Anyway, whatever your goal, rsync is your friend. –  terdon Feb 26 '13 at 23:07
    
The backup files are given through the Prarllels Confixx. I want to archive whitout touching the single .tar.gz archives to keep the archive's integrity (to be able to perform a restore again on Cofixx). And instead of storing the whole tar.gz files, I would prefer to deduplicate the files inside of the archive. The solution I look for is like a incremential backup or a snapshot-similar funtion. –  rimshot Feb 28 '13 at 22:38
add comment

1 Answer 1

I don't think you can do what you are trying to do this "easily"

If the .tgz files are similar enough, you may be able to use xdelta, and backup the first file in full, and then an xdelta with the diff between the first and second file.

An alternative partial solution would be to use rzip (or lrzip ??) which is designed to find compression across very big files - while not ideal it will probably be significantly smaller then what you are using at the moment, at the expense of CPU and memory.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.