35

I must not be the only one - I'm rsyncing .tar.gz files and notice that every time the full file gets rsynced rather than the differences. Reading into it it seems back in 1999 someone created an algorithm that fixed the issue http://svana.org/kleptog/rgzip.html (only 5% of data needed transferred)

Has this gone anywhere since, how do I create rsync friendly .tar.gz files?

Improve this question

5 Answers 5

Reset to default
39

my gzip (on ubuntu and fedora) has the --rsyncable option. So create the tarballs using:

tar -c whatever/ | gzip --rsyncable > file.tar.gz
Improve this answer
16

BeezNest has a pretty good explanation of the rsyncable option for gzip. In the author's test, this option added about 1% to the file size, but made it possible for rsync to transfer the updates to a gzipped file with over 1,300 times speedup.

For the gory details, see this discussion (specifically, section 4.4.2), which they cite. The gist of it is:

The modification is quite simple:

  1. A fast rolling signature is computed for a small window around the current point in the uncompressed file;
  2. stream compression progresses as usual;
  3. when the rolling signature equals a pre-determined value the compression tables are reset and a token is emitted indicating the start of a new compression region.
Improve this answer
11

I like this one because I wanted to tar.gz it, not just .gz

GZIP='--rsyncable' tar cvzf bobsbackup.tar.gz /home/bob
Improve this answer
2
  • 1
    You can do this with the pipeline too tar cv /home/bob | gzip --rsyncable -c >bobsbackup.tar.gz but it's really just a matter of which you happen to prefer
    – BytePorter
    Commented Nov 5, 2020 at 23:01
  • 2
    Ooops I'm a doofus and didn't read the accepted answer fully before posting this. So I'll add that I like the pipeline version because I like to use it with pv frequently to get a progress bar tar c /home/bob | pv | gzip --rsyncable -c >bobsbackup.tar.gz which will show your progress through the uncompressed files
    – BytePorter
    Commented Nov 5, 2020 at 23:07
6

http://smackerelofopinion.blogspot.com/2009/07/rsyncable-gzip.html

gzip --rsyncable  # need two dashes for long options
Improve this answer
3

I know that Ubuntu Linux applies a patch (gzip file) to gzip sources to allow for a --rsyncable flag. You can download that patch and use it yourself, or see if your distribution includes the patch.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .