0

Let’s say that I have data on the order of terabytes, in files on the order of megabytes, to move away from a server that’s over 500 ms away.

Due to the nature of TCP, the command below works but only at a fraction of the bandwidth available, be it my 4 Mbit/s home ADSL connection or a fat gigabit line.

rsync -avP --remove-source-files source.host:path/to/source/ path/to/dest

I am using --remove-source-files because I may need to use multiple hosts and destination directories, these directories may not always contain everything that has successfully been received before, and the source directory is quiescent.

Will it be safe and effective to run multiple instances of the command above concurrently?

2 Answers 2

3

No, the actual behaviour is unpredictable, but chances are that the multiple instances will try to copy the same file at the same time and bandwidth will be wasted, and bad things may happen.

However, cunning use of --include and --exclude might be handy here:

rsync -avP \
    --include '*/' --include '[a-g]*' --exclude '*' \
    source.host:path/to/source/ path/to/dest/

This command will only send files that begin with the letters a to g. You can set up parallel transfers for the other portions of the alphabet.

Finally, once all the transfers are complete, you should run your original rsync command again (note that I left the --remove-source-files off my command) to ensure that any oddness that did happen gets smoothed out, and any files your original filters missed (dot-files, perhaps?) are transferred also.

Incidentally, always put the final slash on the source and destination directories (e.g. path/to/dest/) or rsync might not do what you expect!


However, rsync is not the most efficient way to replicate files the first time, especially when the latency is high (it's intended for the subsequent update passes, mostly).

You would be far better to use tar to stream the data into ssh:

ssh source.host tar -C path/to/source/ cfj - . | tar -C path/to/dest/ xfj -

This will package up and compress the data into a continuous stream, pipe it through your ssh tunnel, back into tar on your local end, and expand back into files all in one command, and without the tar file ever touching disk.

The down-side is that it's not easily resumeable if the connection gets dropped.

Tar also has an --exclude option (but not --include), so you can paralellize the streams in a similar fashion, if necessary. Again, you should probably finish with an rsync to verify the transfer.


Aside: The "nature of TCP" is not the problem here. TCP uses a sliding window scheme that should saturate the link at any normal latency, and there are knobs to twiddle if it doesn't.

The nature of rsync, however, is that it must do some chatting back and forth about each file before it transfers it, and the latency will hurt here (although the implementation uses pipelining to minimize this).

If you're not saturating your link, then the first thing to try is to take rsync out of the equation (scp of a suitably large file will do). If that still doesn't do it, then check your CPU usage: compression and encryption might be the bottleneck. If transferring plain data via Netcat (or old fashioned FTP) can't do it, then it might be you need to tune TCP. Also, check for packet loss with ping, as that'll really screw up TCP. Finally, it might just be that the slowest link in the chain isn't yours.

0

Short answer: No.

If you want to run multiple instances of rsync, you should have each instance process it's own batch of files.

I'm not certain of what will happen but my cumulative experience is telling me that I'd not trust the result.

You should be able to gain efficiency by running multiple instances as long as you are not saturating the slowest parts of the route.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .