Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a 40G music collection that I need to sync to a server in another country.

At the moment I am rsyncing it across, however I have that same 40G collection on two machines with two internet connections.

Ideally, I'd like to start rsync from both souces to make it go twice as fast. The correct solution would be to create a torrent and then use both boxes as seeds, but the upstream end has torrent traffic blocked (and I'd rather not hack around that, the provider is doing me a favour).

Any ideas for how to accomplish this? At the moment I'm considering just dividing the remaining directories in half and specifying them all on the commandline, but that seems ugly, and I'll probably have to do this more than once.

share|improve this question
up vote 2 down vote accepted

Rsync likes to copy files in the same order. If you wanted multiple rsyncs to work in parallel the easiest approach might be to manually partition the job between two.

For example, on machine A:

rsync -avz foo/[A-Ga-g]* remote:foo/

And then, on machine B:

rsync -avz foo/[H-Zh-z]* remote:foo/

You'd want the file sets to be of comparable size. At some point there's a diminishing return based on the available network and disk bandwidth at the target system.

share|improve this answer
That's more or less exactly what I did, thanks. – richo Dec 23 '11 at 3:54
While there is theoretically a bottleneck at the other end, I'm uploading from two domestic connections to a VPS with a full duplex 100M link, so I'd need way more than 2 hosts to saturate it. – richo Dec 23 '11 at 3:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .