Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Rsync only copies over the changed parts of a file. So if I have a large data dump, for example a SQL database in text format, and I want to copy it to my local machine using as little bandwidth as possible, what should I do?

I can leave it as uncompressed text and use rsync with the -z switch, or I can gzip it on the server and rsync the gzip file.

My suspicion is that a minor change in the source SQL text file could result in a larger shuffling around of data in the compressed gzip version, so rsync would have to do more work as more of the file has changed.

Also, since the -z switch is performing compression anyway, this could be the most efficient. However, I'm not sure.

share|improve this question
up vote 3 down vote accepted

I went through a formal process of checking this out a couple of years ago. I needed to copy MS-SQL backup dumps from (30!) regional offices to HQ via broadband-based VPNs and I tried all the permutations of compression and non-compression.

In my case, uncompressed files transferred the quickest using rsync with its own compression. The file sizes were typically 4-9GB of dumped data and some of them would sync up in a few minutes.

If the data dumps were compressed, rsync pretty much transferred the entire files every time and this took hours.

It's worth doing some tests with your data, but I suspect you'll find it's best to leave the files as they are for rsync to sort out.

share|improve this answer
    
Thanks, accepting answer as you've done the experiments. – Mark Theunissen Sep 22 '11 at 8:06

Your guess is most probably right, it is better to only compress the changed parts.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .