parallelizing rsync

Question

I just moved and discovered after some trial and error that somewhere between my home and my remote server, there is some throttling going on...but the throttling is not very intelligent. It only throttles individual connections. So if I copy one 1 GB file, it will proceed merrily at 150 kBps. But if I initialize 10 copies, each of them will go at 150 kBps (i.e. I get much higher aggregate bandwidth over multiple connections).

I use rsync fairly often to synchronize some large datasets from work to home (fortunately in the form of many files). Is there a way to tell rsync to download using multiple connections? Theoretically it should be possible since as far as I can tell, rsync first does a pass to determine the necessary changes and then performs the actual transmission. Bonus points if there's a magic way of telling rsync to slice up individual files into N pieces and then splice them back together. I believe CuteFTP is actually smart enough to pull that off.

Answer 1

Yes. Such a feature exists.

There is a utility called pssh that provides the described functionality.

This package provides parallel versions of the openssh tools. Included in the distribution:

• Parallel ssh (pssh)
• Parallel scp (pscp)
• Parallel rsync (prsync)
• Parallel nuke (pnuke)
• Parallel slurp (pslurp)

I'm not sure how easy it is to set up, but it might just do the trick!

Answer 2

I just had a similar problem having to move several TB from one NAS to a different NAS with no backup/restore capability that would allow me to just feed 1 set to the other.

So I wrote this script to run 1 rsync for each directory it encounters. It depends on being able to list the source directories (be careful to escape ARG 3) but I think you could set that stage with a non-recursive rsync that just copied files and directories to the appropriate level.

It also determines how many rsync's to run based on the number of processors but you might want to tweak that.

The other possible option that comes to mind is: run an rsync in --list-only mode.

That would give you all of the files that need to be updated Then run 1 rsync for each file in your list if you used xargs to manage the number of rsyncs you had going this could be very elegant. Actually probably a more elegant solution than my little script here...

#! /bin/bash
SRC_DIR=$1
DEST_DIR=$2
LIST=$3
CPU_CNT=`cat /proc/cpuinfo|grep processor |wc -l`
#  pseudo random heuristic
let JOB_CNT=CPU_CNT*4
[ -z "$LIST" ] && LIST="-tPavW --exclude .snapshot --exclude hourly.?"
echo "rsyncing From=$SRC_DIR To=$DEST_DIR DIR_LIST=$LIST"
mkdir -p /{OLD,NEW}_NAS/home
[ -z "$RSYNC_OPTS" ] && RSYNC_OPTS="-tPavW --delete-during --exclude .snapshot --exclude hourly.?"
cd $SRC_DIR
echo $LIST|xargs -n1 echo|xargs -n1 -P $JOB_CNT -I% rsync ${RSYNC_OPTS} ${SRC_DIR}/%/ ${DEST_DIR}/%/

Answer 3

No. No such feature exists. You could split the synch into multiple calls to rsync if you really wanted to.

I'd suggest you find whatever it is that's doing this rate-limiting and have a serious talk with whoever maintains/manages it.