4

Rsync only copies over the changed parts of a file. So if I have a large data dump, for example a SQL database in text format, and I want to copy it to my local machine using as little bandwidth as possible, what should I do?

I can leave it as uncompressed text and use rsync with the -z switch, or I can gzip it on the server and rsync the gzip file.

My suspicion is that a minor change in the source SQL text file could result in a larger shuffling around of data in the compressed gzip version, so rsync would have to do more work as more of the file has changed.

Also, since the -z switch is performing compression anyway, this could be the most efficient. However, I'm not sure.

1
  • I'd be interested to see a comparison using gzip --rsyncable Jul 27, 2017 at 16:03

2 Answers 2

5

I went through a formal process of checking this out a couple of years ago. I needed to copy MS-SQL backup dumps from (30!) regional offices to HQ via broadband-based VPNs and I tried all the permutations of compression and non-compression.

In my case, uncompressed files transferred the quickest using rsync with its own compression. The file sizes were typically 4-9GB of dumped data and some of them would sync up in a few minutes.

If the data dumps were compressed, rsync pretty much transferred the entire files every time and this took hours.

It's worth doing some tests with your data, but I suspect you'll find it's best to leave the files as they are for rsync to sort out.

0
1

Your guess is most probably right, it is better to only compress the changed parts.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .