Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a 40G music collection that I need to sync to a server in another country.

At the moment I am rsyncing it across, however I have that same 40G collection on two machines with two internet connections.

Ideally, I'd like to start rsync from both souces to make it go twice as fast. The correct solution would be to create a torrent and then use both boxes as seeds, but the upstream end has torrent traffic blocked (and I'd rather not hack around that, the provider is doing me a favour).

Any ideas for how to accomplish this? At the moment I'm considering just dividing the remaining directories in half and specifying them all on the commandline, but that seems ugly, and I'll probably have to do this more than once.

share|improve this question

1 Answer 1

up vote 1 down vote accepted

Rsync likes to copy files in the same order. If you wanted multiple rsyncs to work in parallel the easiest approach might be to manually partition the job between two.

For example, on machine A:

rsync -avz foo/[A-Ga-g]* remote:foo/

And then, on machine B:

rsync -avz foo/[H-Zh-z]* remote:foo/

You'd want the file sets to be of comparable size. At some point there's a diminishing return based on the available network and disk bandwidth at the target system.

share|improve this answer
    
That's more or less exactly what I did, thanks. –  richo Dec 23 '11 at 3:54
    
While there is theoretically a bottleneck at the other end, I'm uploading from two domestic connections to a VPS with a full duplex 100M link, so I'd need way more than 2 hosts to saturate it. –  richo Dec 23 '11 at 3:55

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.