Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I take a nightly backup from my Webhosting service as .tar.gz. The Data changed is very little each night (<10MB).

I would like to archive these files and deduplicate (solid compression) files across multiple .tar.gz archives without extracting each individual backup file.

Simplified example:

2013-02-24.tar.gz contains a.pdf 2013-02-25.tar.gz contains a.pdf and b.pdf

I want to compress and archive 2013-02-24.tar.gz and 2013-02-25.tar.gz and store a.pdf only one time.

The only way I got the deduplication working was to unpack both .tar.gz and pack them again (using solid compression). But this is one additional step and it will probably erase filesystem information due to different OS's. Simply tar/gz both .tar.gz files won't work, the final size of this archive will be the sum of both original .tar.gz files.

Is there a way use deduplication across the contents of multiple .tar.gz files?

share|improve this question
Why is this "deduplication" important? Why do you need both 2013-02-24.tar.gz and 2013-02-25.tar.gz if you only want one copy of a.pdf? What is the point of having different archives for different days unless you want to be able to recreate the exact state of 2013-02-24? Please tell us what exactly you are trying to do, if you just want a safe backup where files deleted on the server are not deleted in the backup, there are easier ways to do this. Please update your question stating your end goal. Anyway, whatever your goal, rsync is your friend. – terdon Feb 26 '13 at 23:07
The backup files are given through the Prarllels Confixx. I want to archive whitout touching the single .tar.gz archives to keep the archive's integrity (to be able to perform a restore again on Cofixx). And instead of storing the whole tar.gz files, I would prefer to deduplicate the files inside of the archive. The solution I look for is like a incremential backup or a snapshot-similar funtion. – rimshot Feb 28 '13 at 22:38

I don't think you can do what you are trying to do this "easily"

If the .tgz files are similar enough, you may be able to use xdelta, and backup the first file in full, and then an xdelta with the diff between the first and second file.

An alternative partial solution would be to use rzip (or lrzip ??) which is designed to find compression across very big files - while not ideal it will probably be significantly smaller then what you are using at the moment, at the expense of CPU and memory.

share|improve this answer

I use squashfs for similar purpose:

mksquashfs dir1 dir2 dir3 backup_file.fs -keep-as-directory
share|improve this answer
Does this deduplicate as requested by OP? – music2myear Sep 18 '14 at 16:41
The howto says "File duplicates are detected and removed" - – iPaulo Nov 17 '15 at 21:01

I'm late to the party, but the requirements seem to essentially boil down to this:

  1. Deduplicate tarball file contents
  2. Must be able to restore original gzip files

At first, this seems impossible. Deduplication generally depends on being able to work on the uncompressed files. But then I realized that if you can rely on gunzip/gzip producing the exact same file (better test it), then you can use zbackup, which is actually designed to deduplicate tarballs. So:

  1. gunzip the tarballs you get
  2. gzip the tarballs again to verify that they are exact matches (should be if the gzip implementation is the same, but compression level could vary for example, so you may need to find that out)
  3. Feed the tarballs to zbackup
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.