4

I am using (trying) gzip to compress an SQL file in an rsyncable way in order to transfer the backup with minimal delay.

However it appears that this is not working, since the speedup is always 1.00.

The archive is created by dumping the database to a .sql file and then issuing gzip -f3 --rsyncable file.sql.

Next the remote machine does an rsync against the last backup with the following flags:

rsync -avhhiP --inplace

Why might my speedup be 1.00? Should I not be recreating the archive each time and instead updating it perhaps? I have seen no mention of this method from online guidance about the usage of the --rsyncable flag.

I am using:

# gzip -V
gzip 1.5
Improve this question
5
  • 1
    If --rsyncable is not listed in the gzip --help, the command doesn't support that option. I also have gzip 1.5 (Gentoo) and it doesn't have that option. The presence of --rsyncable option depends on whether the distribution has applied the patch that adds the option. Which distribution do you have?
    – Dan D.
    Oct 2, 2013 at 0:25
  • If you want the option, you will need to apply the patch and build gzip.
    – Dan D.
    Oct 2, 2013 at 0:26
  • Ah, I thought this should have been rectified since version 1.4. I checked both the --help output and the manpage, one mentions it but not the other although I'm not sure which specifically. This is on Debian wheezy.
    – deed02392
    Oct 2, 2013 at 8:43
  • 2
    OK, just checked and --rsyncable IS in the help output, but it's not in the manpage.
    – deed02392
    Oct 6, 2013 at 21:30
  • Have you tried without --inplace ? I don't know how rsync could not retransfer the whole file (or at least everything starting from the first modified block) if you ask it to update in place.
    – Benoît
    Jan 2, 2014 at 10:20

2 Answers 2

Reset to default
8

The question is already somewhat old, but maybe my answer still helps one or the other:

Debian Wheezy has the aforementioned bug in gzip which renders the --rsyncable-flag non-functional.

See the corresponding Debian bug.

You can use pigz as a replacement, which is a parallelizing gzip-replacement which uses multiple CPUs, can compress slightly more efficiently if you manually specify larger block sizes and officially provides an --rsyncable implementation which is supposed to be better than the one provided by the gzip patch.

Additionally, rsyncs --inplace parameter reduces the efficiency of the delta-transmission algorithm - to quote the manpage:

The efficiency of rsync's delta-transfer algorithm may be reduced if some data in the destination file is overwritten before it can be copied to a position later in the file. This does not apply if you use --backup, since rsync is smart enough to use the backup file as the basis file for the transfer.

Improve this answer
2
  • 1
    Note: that Debian bug was apparently fixed when gzip 1.6-3 was uploaded, Dec 2013. This means it should work correctly in Jessie and later.
    – thomasrutter
    Nov 13, 2017 at 0:24
  • Yes, as I wrote, Wheezy is affected. The patch works in all other versions of Debian that include it. Still pigz claims to provide a somewhat more effective --rsyncable flag.
    – Gunter Ohrner
    Nov 14, 2017 at 1:41
2

I spent a long time trying to work out why --rsyncable wasn't working for me on Ubuntu 12.04 and gzip 1.4 before stumbling upon this: https://bugs.launchpad.net/ubuntu/+source/gzip/+bug/1021975

It appears there's a regression in the 1.4 ubuntu package which means that the rsyncable patch wasn't properly included in the release, despite it being listed in the gzip --help. I'm not sure whether it affects other distros or versions of Ubuntu.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.