Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

When syncing large files over a slow link, it is often useful to use delta compression in order to reduce the bandwidth used. It is also useful to compress files as they take up much less space.

However, many compression algorithms have the unwanted side affect of changing large portions of the compressed output when only a small change has been made in the source.

So, what are some compression algorithms/utilities which create similar compressed blobs from similar source files?

share|improve this question

3 Answers 3

up vote 1 down vote accepted

You may want to consider using the --rsyncable argument to gzip. This slightly reduces the compression because it resets the compression algorithm at the beginning of each file in your tarball.

Note, though, that you can also use the -z option with rsync to compress the data transmitted. If you are comfortable with uncompressed files on the source and destination, this may be sufficient for your needs.

We use this to sync up a compressed MySQL database backup that's around 20 GB. Using gzip --rsyncable, we can often transfer only a fraction of the compressed file via rsync if we have the prior day's snapshot already. I have not tried transferring the uncompressed tar file and using rsync's -z option so I cannot comment on its relative efficiency.

share|improve this answer
2  
--rsyncable appears to be a Debian-only patch. –  grawity Aug 26 '13 at 18:03
    
Good solution, this patch is widely applied and quite effective. But as mentioned, it isn't upstreamed. It is present in debian, Fedora and probably other distros. –  Kevin Cox Aug 26 '13 at 19:17

One solution is to manually compress a file block-by-block. From simple test with the following script files are only slightly (~0.1%) larger with 1MiB blocks. It works for algorithms which support concatenation (gzip and bzip2 to name two). It is also a little slower, but trivial to parallelize.

#!/bin/bash

alg="${2:-gzip}"

size=$(stat -c %s "$1")
cur=0
block=$((1024*1024)) # 1M blocks.

while [ $cur -lt $size ]; do
    #echo $alg $size $block $cur >&2
    tail -c +$cur "$1" | head -c $block | $alg
    cur=$((cur+block))
done

This way changes can only affect the changed block.

share|improve this answer
    
@Hennes, not the right place, but about your edit. What system are you using, I have never had a problem and always put the space there because I think it looks prettier. –  Kevin Cox Aug 27 '13 at 1:35
    
I tried on FreeBSD, but failed to correct to /usr/local/bin/bash (no bash in /bin/ on many systems). –  Hennes Aug 27 '13 at 15:15
    
But with the space it works? That is strange. I will read up when I get a chance, always something to learn :D –  Kevin Cox Aug 27 '13 at 18:40

You may want to look in to using rdiff. It is similar to rsync however it does not require a two way connection with the server.

The way you would use it is

  1. On the source computer: rdiff signature file.ext file.ext.sig This creates a signature file containing the hashes for the blocks future diffs will use. keep the .sig file around on the source machine.
  2. Compress the file as normal and send to destination
  3. Decompress the file at the destination
  4. Time passes
  5. On the source computer: rdiff delta file.ext.sig file.ext file.ext.delta this creates a delta file that only contains the changes where the hashes did not match from the signature file.
    • You will likely do another rdiff signature file.ext file.ext.sig to update the signature file.
  6. Send the delta file to the destination, the delta file is already compressed
  7. On the destination: rdiff patch file.ext file.ext.delta file.ext this will update the destination copy of the file to match the source copy. Depending on the build of rdiff the 2nd argument and 4th argument may need to be different file names.
share|improve this answer
    
rdiff is essentially the rsync dela computation broken out of the rsync protocol, very useful but it doesn't answer the question. The other half of the answer is to calculate the diff uncompressed, this makes sense but in my case requires decompressing and every time I want to calculate a diff. The destination would also have to decompress the existing file and recompress it after patching. If compressing just for transport (and using rsync) I would recommend the -z option to let rsync do the compression transparently. –  Kevin Cox Aug 26 '13 at 19:28
    
You tagged rsync but never really brought it up in your question. I assumed there was some reason you could not use it so I was trying to re-create it's functionality via rdiff. I had to do this exact situation where we had client data that was uploaded to us on a write only FTP server. We had them upload the original file then just the diffs when we needed a data refresh. –  Scott Chamberlain Aug 26 '13 at 19:30
    
Sorry for the confusion. I tagged it but didn't mention it because the question is more general than just rsync. I will be using rsync, but also plan on doing something in the future that will use xdelta or rdiff. –  Kevin Cox Aug 26 '13 at 19:42

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.