Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I just moved and discovered after some trial and error that somewhere between my home and my remote server, there is some throttling going on...but the throttling is not very intelligent. It only throttles individual connections. So if I copy one 1 GB file, it will proceed merrily at 150 kBps. But if I initialize 10 copies, each of them will go at 150 kBps (i.e. I get much higher aggregate bandwidth over multiple connections).

I use rsync fairly often to synchronize some large datasets from work to home (fortunately in the form of many files). Is there a way to tell rsync to download using multiple connections? Theoretically it should be possible since as far as I can tell, rsync first does a pass to determine the necessary changes and then performs the actual transmission. Bonus points if there's a magic way of telling rsync to slice up individual files into N pieces and then splice them back together. I believe CuteFTP is actually smart enough to pull that off.

share|improve this question

I just had a similar problem having to move several TB from one NAS to a different NAS with no backup/restore capability that would allow me to just feed 1 set to the other.

So I wrote this script to run 1 rsync for each directory it encounters. It depends on being able to list the source directories (be careful to escape ARG 3) but I think you could set that stage with a non-recursive rsync that just copied files and directories to the appropriate level.

It also determines how many rsync's to run based on the number of processors but you might want to tweak that.

The other possible option that comes to mind is: run an rsync in --list-only mode.

That would give you all of the files that need to be updated Then run 1 rsync for each file in your list if you used xargs to manage the number of rsyncs you had going this could be very elegant. Actually probably a more elegant solution than my little script here...

#! /bin/bash
CPU_CNT=`cat /proc/cpuinfo|grep processor |wc -l`
#  pseudo random heuristic
[ -z "$LIST" ] && LIST="-tPavW --exclude .snapshot --exclude hourly.?"
echo "rsyncing From=$SRC_DIR To=$DEST_DIR DIR_LIST=$LIST"
mkdir -p /{OLD,NEW}_NAS/home
[ -z "$RSYNC_OPTS" ] && RSYNC_OPTS="-tPavW --delete-during --exclude .snapshot --exclude hourly.?"
echo $LIST|xargs -n1 echo|xargs -n1 -P $JOB_CNT -I% rsync ${RSYNC_OPTS} ${SRC_DIR}/%/ ${DEST_DIR}/%/
share|improve this answer
This works -- you can make a lot of improvements to how it works, but the concept of using xargs to parallelize your application is pretty novel. – MattPark Dec 17 '13 at 19:32

GNU Parallel has a solution

I have moved 15 TB through 1 Gbps and it can saturate the 1 Gbps link.

The following will start one rsync per big file in src-dir to dest-dir on the server fooserver:

cd src-dir; find . -type f -size +100000 | \
parallel -v ssh fooserver mkdir -p /dest-dir/{//}\; \
  rsync -s -Havessh {} fooserver:/dest-dir/{}

The dirs created may end up with wrong permissions and smaller files are not being transferred. To fix those run rsync a final time:

rsync -Havessh src-dir/ fooserver:/dest-dir/
share|improve this answer
Would you mind pasting the "EXAMPLE: Parallelizing rsync" section into your answer. Just in case the link breaks in the future. – picobit May 15 at 17:58

Yes. Such a feature exists.

There is a utility called pssh that provides the described functionality.

This package provides parallel versions of the openssh tools. Included in the distribution:

  • Parallel ssh (pssh)
  • Parallel scp (pscp)
  • Parallel rsync (prsync)
  • Parallel nuke (pnuke)
  • Parallel slurp (pslurp)

I'm not sure how easy it is to set up, but it might just do the trick!

share|improve this answer
The pssh utilities are used to spread commands across multiple servers, not do the same command a bunch of times on one server. In particular, prsync only supports sending a file on your local machine out to multiple external machines. It does not support downloading a remote file with multiple connections. – Derek Dahmer Jul 24 '12 at 23:29

I cannot comment, so I have added a new answer, with a little bit better code than the previous (nice & smart) code.

Check the rsync line, because it contains an optional ionice tweak.

start_time=$(date +%s.%N)
# Transfer files in parallel using rsync (simple script)
# MAXCONN: maximum number "rsync" processes running at the same time:
# Source and destination base paths. (not need to end with "/")
RSYNC_OPTS="-ah --partial"
# Main loop:
for FULLDIR in $SRC_BASE/*; do
    NUMRSYNC=`ps -Ao comm | grep '^'rsync'$' | wc -l `
    while [ $NUMRSYNC -ge $MAXCONN ]; do
        NUMRSYNC=`ps -Ao comm | grep '^'rsync'$' | wc -l `
        sleep 1
    DIR=`basename $FULLDIR`
    echo "Start: " $DIR
    ionice -c2 -n5 rsync $RSYNC_OPTS $SRC_BASE/${DIR}/ $DST_BASE/${DIR}/ &
    # rsync $RSYNC_OPTS $SRC_BASE/${DIR}/ $DST_BASE/${DIR}/ &
    sleep 5

execution_time=$(echo "$(date +%s.%N) - $start" | bc)
printf "Done. Execution time: %.6f seconds\n" $execution_time
share|improve this answer

No. No such feature exists. You could split the synch into multiple calls to rsync if you really wanted to.

I'd suggest you find whatever it is that's doing this rate-limiting and have a serious talk with whoever maintains/manages it.

share|improve this answer
Frequently those restrictions are from some ISP like Comcast. Good luck having any sort of reasonable conversation with them. – James Moore Sep 16 '12 at 21:46

Looks like someone has written this utility for you. It breaks the transfer into parallel chunks. This is a better implementation than the "parallel big file" version listed under GNU Parallel:

Also, lftp can parallelize file transfers via ftp, ftps, http, https, hftp, fish, sftp. A lot of times, there are some advantages to using lftp, because managing permissions, restricted access, etc for rsync can be challenging.

share|improve this answer

I wanted to transfer several directories (with many files) at the same time, so I created this small script:

# Transfer files in parallel using rsync (simple script)
# MAXCONN: maximum number "rsync" processes running at the same time:
# Source and destination base paths. (not need to end with "/")
RSYNC_OPTS="--stats -ilrtpog"
# Main loop:
for FULLDIR in $SRC_BASE/*/; do
    NUMRSYNC=`ps -Ao comm | grep '^'rsync'$' | wc -l `
    while [ $NUMRSYNC -ge $MAXCONN ]; do
        NUMRSYNC=`ps -Ao comm | grep '^'rsync'$' | wc -l `
        sleep 10
    DIR=`basename $FULLDIR`
    rsync $RSYNC_OPTS $SRC_BASE/${DIR}/ $DST_BASE/${DIR}/ & 
    sleep 1 
echo "Done."

I did this script quite fast, so please revise it and test it before using in a production environment.

share|improve this answer

I created the following script to upload many folders with pictures in parallel. You run it with first the sync target and then all the folder names to copy.



if [ "$dest" = "" ]; then
    echo "USAGE: $0 TARGET:/foo/bar <dir1> [dir2] [dir3]"
    exit 1

RCol='\x1B[0m' # Text Reset

for i in "$@"; do
    prefix=`printf "$BYel%50s:$RCol" "$i"`
    echo "$prefix * Starting $i"
    echo "$prefix -> syncing '$i/' to '$dest/$i/'"
    (rsync -rv "$i/" "$dest/$i/") 2>&1 | sed "s/^/$prefix /g" &
    sleep 0.5

echo "* Waiting for all to complete"

It prefixes the folder name in yellow to all rsync console output to make it look pretty.

share|improve this answer

Aria2 is a good client program to download data using many connections from many mirrors. It does not support SFTP. So, I've installed FTP server - vsftpd. My 3g connection works on full power with 5 connections to FTP server.

share|improve this answer
Would you care to expand on that to make your answer useful? – Tog Sep 4 '13 at 8:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .