Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I just moved and discovered after some trial and error that somewhere between my home and my remote server, there is some throttling going on...but the throttling is not very intelligent. It only throttles individual connections. So if I copy one 1 GB file, it will proceed merrily at 150 kBps. But if I initialize 10 copies, each of them will go at 150 kBps (i.e. I get much higher aggregate bandwidth over multiple connections).

I use rsync fairly often to synchronize some large datasets from work to home (fortunately in the form of many files). Is there a way to tell rsync to download using multiple connections? Theoretically it should be possible since as far as I can tell, rsync first does a pass to determine the necessary changes and then performs the actual transmission. Bonus points if there's a magic way of telling rsync to slice up individual files into N pieces and then splice them back together. I believe CuteFTP is actually smart enough to pull that off.

share|improve this question

7 Answers 7

I just had a similar problem having to move several TB from one NAS to a different NAS with no backup/restore capability that would allow me to just feed 1 set to the other.

So I wrote this script to run 1 rsync for each directory it encounters. It depends on being able to list the source directories (be careful to escape ARG 3) but I think you could set that stage with a non-recursive rsync that just copied files and directories to the appropriate level.

It also determines how many rsync's to run based on the number of processors but you might want to tweak that.

The other possible option that comes to mind is: run an rsync in --list-only mode.

That would give you all of the files that need to be updated Then run 1 rsync for each file in your list if you used xargs to manage the number of rsyncs you had going this could be very elegant. Actually probably a more elegant solution than my little script here...

#! /bin/bash
SRC_DIR=$1
DEST_DIR=$2
LIST=$3
CPU_CNT=`cat /proc/cpuinfo|grep processor |wc -l`
#  pseudo random heuristic
let JOB_CNT=CPU_CNT*4
[ -z "$LIST" ] && LIST="-tPavW --exclude .snapshot --exclude hourly.?"
echo "rsyncing From=$SRC_DIR To=$DEST_DIR DIR_LIST=$LIST"
mkdir -p /{OLD,NEW}_NAS/home
[ -z "$RSYNC_OPTS" ] && RSYNC_OPTS="-tPavW --delete-during --exclude .snapshot --exclude hourly.?"
cd $SRC_DIR
echo $LIST|xargs -n1 echo|xargs -n1 -P $JOB_CNT -I% rsync ${RSYNC_OPTS} ${SRC_DIR}/%/ ${DEST_DIR}/%/
share|improve this answer
    
This works -- you can make a lot of improvements to how it works, but the concept of using xargs to parallelize your application is pretty novel. –  khaki54 Dec 17 '13 at 19:32

GNU Parallel has a solution.  I have moved 15 TB through 1 Gbps and it can saturate the 1 Gbps link.

share|improve this answer

Yes. Such a feature exists.

There is a utility called pssh that provides the described functionality.

This package provides parallel versions of the openssh tools. Included in the distribution:

  • Parallel ssh (pssh)
  • Parallel scp (pscp)
  • Parallel rsync (prsync)
  • Parallel nuke (pnuke)
  • Parallel slurp (pslurp)

I'm not sure how easy it is to set up, but it might just do the trick!

share|improve this answer
12  
The pssh utilities are used to spread commands across multiple servers, not do the same command a bunch of times on one server. In particular, prsync only supports sending a file on your local machine out to multiple external machines. It does not support downloading a remote file with multiple connections. –  Derek Dahmer Jul 24 '12 at 23:29

No. No such feature exists. You could split the synch into multiple calls to rsync if you really wanted to.

I'd suggest you find whatever it is that's doing this rate-limiting and have a serious talk with whoever maintains/manages it.

share|improve this answer
2  
Frequently those restrictions are from some ISP like Comcast. Good luck having any sort of reasonable conversation with them. –  James Moore Sep 16 '12 at 21:46

Looks like someone has written this utility for you. It breaks the transfer into parallel chunks. This is a better implementation than the "parallel big file" version listed under GNU Parallel:

https://gist.github.com/rcoup/5358786

Also, lftp can parallelize file transfers via ftp, ftps, http, https, hftp, fish, sftp. A lot of times, there are some advantages to using lftp, because managing permissions, restricted access, etc for rsync can be challenging.

share|improve this answer

I wanted to transfer several directories (with many files) at the same time, so I created this small script:

#!/bin/bash
# Transfer files in parallel using rsync (simple script)
# MAXCONN: maximum number "rsync" processes running at the same time:
MAXCONN=10
# Source and destination base paths. (not need to end with "/")
SRC_BASE=/home/sites
DST_BASE=user@example.com:/var/www
RSYNC_OPTS="--stats -ilrtpog"
# Main loop:
for FULLDIR in $SRC_BASE/*/; do
    NUMRSYNC=`ps -Ao comm | grep '^'rsync'$' | wc -l `
    while [ $NUMRSYNC -ge $MAXCONN ]; do
        NUMRSYNC=`ps -Ao comm | grep '^'rsync'$' | wc -l `
        sleep 10
    done
    DIR=`basename $FULLDIR`
    rsync $RSYNC_OPTS $SRC_BASE/${DIR}/ $DST_BASE/${DIR}/ & 
    sleep 1 
done
echo "Done."

I did this script quite fast, so please revise it and test it before using in a production environment.

share|improve this answer

Aria2 is a good client program to download data using many connections from many mirrors. It does not support SFTP. So, I've installed FTP server - vsftpd. My 3g connection works on full power with 5 connections to FTP server.

share|improve this answer
1  
Would you care to expand on that to make your answer useful? –  Tog Sep 4 '13 at 8:13

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.