Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have to keep a single, relatively small file up to date over a slow and unstable dial up modem connection.

The file is an XML that could start with 15K and, two hours later, end up with 500K.

It's unidirectional replication. My current solution keeps downloading the whole file once every minute. So far it works fine with broadband connections, but now I have to use it in a location where only dial up is available. The illusion of "real time data" gets lost because the download is slow.

I thought that rsync could save me from developing a propietary (and defficient) "delta-only" protocol. I wonder if rsync would provide real benefits in this particular case.

So, the question is: Is the rsync protocol a good fit for a single file from 15K to 500K?

share|improve this question
Is this unidirectional or bidirectional replication of data? – Nick Aug 14 '12 at 21:55
It is unidirectional replication. – Sebastián Grignoli Aug 14 '12 at 22:12

Not in this case. You're better off using something like diffxml to send only the difference between files as an update, and applying the patch on the destination machine.

share|improve this answer
Could you explain why not? A disadvantage of the diffxml solution would be that I would have to code a solution around it, and keep a copy of every outdated version of the XML in the source server to calculate the diff against. – Sebastián Grignoli Aug 14 '12 at 21:55
A change at the beginning of the XML document would require more work to transmit using rsync than a change at the end of the document, whereas it would be about the same amount using diffxml regardless of where the change was made. – Ignacio Vazquez-Abrams Aug 14 '12 at 21:57
diffxml looks like a more analytical tool, not aimed at bandwith savings. – Sebastián Grignoli Aug 14 '12 at 22:18
Also, I just tried rsync locally and it does not seem to reduce bandwith usage at all with this file sizes. I'm starting to think of using Git for this job... – Sebastián Grignoli Aug 14 '12 at 22:20
@SebastiánGrignoli I was just about to write an answer suggesting putting it in a repo and syncing it that way. Multiple updates get merged and chunked. +1 for you. – Nick Aug 15 '12 at 1:29
up vote 0 down vote accepted

Well, I did it with rsync, and I have to say that it worked fine.

It transfers so little data that I was able to shorten the update frequency to 20 seconds under a narrow dial up connection (one third than the previous frequency we used with broadband!).

Updating an 800kb XML now takes 5kb of download + 800 bytes upload!

Checking versions when there are no changes to update takes less 500 bytes download + 100 bytes upload! (I transfer more bytes typing in a regular chat session!)

Way better than other diff solutions, and I didn't have to program anything around it. (previously we thought of adding a push mecanism to fire up rsync, but we don't really need it yet. 20 seconds are acceptable and the polling is very lightweight.)

Way to go, rsync!

PS: I didn't try Git, and it would probably have been a good solution either.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .