Is it more efficient to rsync uncompressed text data or the same data gzipped?

Question

Rsync only copies over the changed parts of a file. So if I have a large data dump, for example a SQL database in text format, and I want to copy it to my local machine using as little bandwidth as possible, what should I do?

I can leave it as uncompressed text and use rsync with the -z switch, or I can gzip it on the server and rsync the gzip file.

My suspicion is that a minor change in the source SQL text file could result in a larger shuffling around of data in the compressed gzip version, so rsync would have to do more work as more of the file has changed.

Also, since the -z switch is performing compression anyway, this could be the most efficient. However, I'm not sure.

Answer 1

I went through a formal process of checking this out a couple of years ago. I needed to copy MS-SQL backup dumps from (30!) regional offices to HQ via broadband-based VPNs and I tried all the permutations of compression and non-compression.

In my case, uncompressed files transferred the quickest using rsync with its own compression. The file sizes were typically 4-9GB of dumped data and some of them would sync up in a few minutes.

If the data dumps were compressed, rsync pretty much transferred the entire files every time and this took hours.

It's worth doing some tests with your data, but I suspect you'll find it's best to leave the files as they are for rsync to sort out.

Answer 2

Your guess is most probably right, it is better to only compress the changed parts.