3

I take a nightly backup from my Webhosting service as .tar.gz. The Data changed is very little each night (<10MB).

I would like to archive these files and deduplicate (solid compression) files across multiple .tar.gz archives without extracting each individual backup file.

Simplified example:

2013-02-24.tar.gz contains a.pdf 2013-02-25.tar.gz contains a.pdf and b.pdf

I want to compress and archive 2013-02-24.tar.gz and 2013-02-25.tar.gz and store a.pdf only one time.

The only way I got the deduplication working was to unpack both .tar.gz and pack them again (using solid compression). But this is one additional step and it will probably erase filesystem information due to different OS's. Simply tar/gz both .tar.gz files won't work, the final size of this archive will be the sum of both original .tar.gz files.

Is there a way use deduplication across the contents of multiple .tar.gz files?

2
  • 1
    Why is this "deduplication" important? Why do you need both 2013-02-24.tar.gz and 2013-02-25.tar.gz if you only want one copy of a.pdf? What is the point of having different archives for different days unless you want to be able to recreate the exact state of 2013-02-24? Please tell us what exactly you are trying to do, if you just want a safe backup where files deleted on the server are not deleted in the backup, there are easier ways to do this. Please update your question stating your end goal. Anyway, whatever your goal, rsync is your friend.
    – terdon
    Feb 26, 2013 at 23:07
  • The backup files are given through the Prarllels Confixx. I want to archive whitout touching the single .tar.gz archives to keep the archive's integrity (to be able to perform a restore again on Cofixx). And instead of storing the whole tar.gz files, I would prefer to deduplicate the files inside of the archive. The solution I look for is like a incremential backup or a snapshot-similar funtion.
    – rimshot
    Feb 28, 2013 at 22:38

4 Answers 4

1

I'm late to the party, but the requirements seem to essentially boil down to this:

  1. Deduplicate tarball file contents
  2. Must be able to restore original gzip files

At first, this seems impossible. Deduplication generally depends on being able to work on the uncompressed files. But then I realized that if you can rely on gunzip/gzip producing the exact same file (better test it), then you can use zbackup, which is actually designed to deduplicate tarballs. So:

  1. gunzip the tarballs you get
  2. gzip the tarballs again to verify that they are exact matches (should be if the gzip implementation is the same, but compression level could vary for example, so you may need to find that out)
  3. Feed the tarballs to zbackup
0

I don't think you can do what you are trying to do this "easily"

If the .tgz files are similar enough, you may be able to use xdelta, and backup the first file in full, and then an xdelta with the diff between the first and second file.

An alternative partial solution would be to use rzip (or lrzip ??) which is designed to find compression across very big files - while not ideal it will probably be significantly smaller then what you are using at the moment, at the expense of CPU and memory.

0

I use squashfs for similar purpose:

mksquashfs dir1 dir2 dir3 backup_file.fs -keep-as-directory
2
0

There is an excellent solution which does what you asks, it deduplicates blocks inside multiple TAR files. This does not only eliminate files which have not been changed, but it also stores only changed parts of similar files.

It is however a client/server solution where the server is a payed offering. But due to the deduplication it is very affordable plus it offers encryption and stores the files safely for you.

The client uses the normal tar code so the command is a snap to use.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .