Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Rsync only copies over the changed parts of a file. So if I have a large data dump, for example a SQL database in text format, and I want to copy it to my local machine using as little bandwidth as possible, what should I do?

I can leave it as uncompressed text and use rsync with the -z switch, or I can gzip it on the server and rsync the gzip file.

My suspicion is that a minor change in the source SQL text file could result in a larger shuffling around of data in the compressed gzip version, so rsync would have to do more work as more of the file has changed.

Also, since the -z switch is performing compression anyway, this could be the most efficient. However, I'm not sure.

share|improve this question

2 Answers 2

up vote 3 down vote accepted

I went through a formal process of checking this out a couple of years ago. I needed to copy MS-SQL backup dumps from (30!) regional offices to HQ via broadband-based VPNs and I tried all the permutations of compression and non-compression.

In my case, uncompressed files transferred the quickest using rsync with its own compression. The file sizes were typically 4-9GB of dumped data and some of them would sync up in a few minutes.

If the data dumps were compressed, rsync pretty much transferred the entire files every time and this took hours.

It's worth doing some tests with your data, but I suspect you'll find it's best to leave the files as they are for rsync to sort out.

share|improve this answer
    
Thanks, accepting answer as you've done the experiments. –  Mark Theunissen Sep 22 '11 at 8:06

Your guess is most probably right, it is better to only compress the changed parts.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.