No, the actual behaviour is unpredictable, but chances are that the multiple instances will try to copy the same file at the same time and bandwidth will be wasted, and bad things may happen.
However, cunning use of --include
and --exclude
might be handy here:
rsync -avP \
--include '*/' --include '[a-g]*' --exclude '*' \
source.host:path/to/source/ path/to/dest/
This command will only send files that begin with the letters a
to g
. You can set up parallel transfers for the other portions of the alphabet.
Finally, once all the transfers are complete, you should run your original rsync command again (note that I left the --remove-source-files
off my command) to ensure that any oddness that did happen gets smoothed out, and any files your original filters missed (dot-files, perhaps?) are transferred also.
Incidentally, always put the final slash on the source and destination directories (e.g. path/to/dest/
) or rsync
might not do what you expect!
However, rsync is not the most efficient way to replicate files the first time, especially when the latency is high (it's intended for the subsequent update passes, mostly).
You would be far better to use tar
to stream the data into ssh
:
ssh source.host tar -C path/to/source/ cfj - . | tar -C path/to/dest/ xfj -
This will package up and compress the data into a continuous stream, pipe it through your ssh
tunnel, back into tar
on your local end, and expand back into files all in one command, and without the tar file ever touching disk.
The down-side is that it's not easily resumeable if the connection gets dropped.
Tar also has an --exclude
option (but not --include
), so you can paralellize the streams in a similar fashion, if necessary. Again, you should probably finish with an rsync to verify the transfer.
Aside: The "nature of TCP" is not the problem here. TCP uses a sliding window scheme that should saturate the link at any normal latency, and there are knobs to twiddle if it doesn't.
The nature of rsync, however, is that it must do some chatting back and forth about each file before it transfers it, and the latency will hurt here (although the implementation uses pipelining to minimize this).
If you're not saturating your link, then the first thing to try is to take rsync out of the equation (scp
of a suitably large file will do). If that still doesn't do it, then check your CPU usage: compression and encryption might be the bottleneck. If transferring plain data via Netcat (or old fashioned FTP) can't do it, then it might be you need to tune TCP. Also, check for packet loss with ping
, as that'll really screw up TCP. Finally, it might just be that the slowest link in the chain isn't yours.