37

I just moved and discovered after some trial and error that somewhere between my home and my remote server, there is some throttling going on...but the throttling is not very intelligent. It only throttles individual connections. So if I copy one 1 GB file, it will proceed merrily at 150 kBps. But if I initialize 10 copies, each of them will go at 150 kBps (i.e. I get much higher aggregate bandwidth over multiple connections).

I use rsync fairly often to synchronize some large datasets from work to home (fortunately in the form of many files). Is there a way to tell rsync to download using multiple connections? Theoretically it should be possible since as far as I can tell, rsync first does a pass to determine the necessary changes and then performs the actual transmission. Bonus points if there's a magic way of telling rsync to slice up individual files into N pieces and then splice them back together. I believe CuteFTP is actually smart enough to pull that off.

9 Answers 9

15

I just had a similar problem having to move several TB from one NAS to a different NAS with no backup/restore capability that would allow me to just feed 1 set to the other.

So I wrote this script to use xargs go run several rsyncs for each directory it encounters. It depends on being able to list the source directories (be careful to escape ARG 3) but I think you could set that stage with a non-recursive rsync that just copied files and directories to the appropriate level.

It also determines how many rsync's to run based on the number of processors but you might want to tweak that.

#!/bin/bash
start_time=$(date +%s.%N)
# Transfer files in parallel using rsync (simple script)
# MAXCONN: maximum number "rsync" processes running at the same time:
MAXCONN=6
# Source and destination base paths. (not need to end with "/")
SRC_BASE=/home/user/public_html/images
[email protected]:/home/user/public_html/images
RSYNC_OPTS="-ah --partial"
# Main loop:
for FULLDIR in $SRC_BASE/*; do
    NUMRSYNC=`ps -Ao comm | grep '^'rsync'$' | wc -l `
    while [ $NUMRSYNC -ge $MAXCONN ]; do
        NUMRSYNC=`ps -Ao comm | grep '^'rsync'$' | wc -l `
        sleep 1
    done
    DIR=`basename $FULLDIR`
    echo "Start: " $DIR
    ionice -c2 -n5 rsync $RSYNC_OPTS $SRC_BASE/${DIR}/ $DST_BASE/${DIR}/ &
    # rsync $RSYNC_OPTS $SRC_BASE/${DIR}/ $DST_BASE/${DIR}/ &
    sleep 5
done

execution_time=$(echo "$(date +%s.%N) - $start" | bc)
printf "Done. Execution time: %.6f seconds\n" $execution_time
2
  • 2
    This works -- you can make a lot of improvements to how it works, but the concept of using xargs to parallelize your application is pretty novel.
    – MattPark
    Dec 17, 2013 at 19:32
  • I included updates from the comments below superuser.com/a/1044631/109778
    – kkron
    Oct 16, 2023 at 16:54
7

GNU Parallel has a solution

I have moved 15 TB through 1 Gbps and it can saturate the 1 Gbps link.

The following will start one rsync per big file in src-dir to dest-dir on the server fooserver:

cd src-dir; find . -type f -size +100000 | \
parallel -v ssh fooserver mkdir -p /dest-dir/{//}\; \
  rsync -s -Havessh {} fooserver:/dest-dir/{}

The dirs created may end up with wrong permissions and smaller files are not being transferred. To fix those run rsync a final time:

rsync -Havessh src-dir/ fooserver:/dest-dir/
1
  • 1
    Would you mind pasting the "EXAMPLE: Parallelizing rsync" section into your answer. Just in case the link breaks in the future.
    – picobit
    May 15, 2016 at 17:58
3

Yes. Such a feature exists.

There is a utility called pssh that provides the described functionality.

This package provides parallel versions of the openssh tools. Included in the distribution:

  • Parallel ssh (pssh)
  • Parallel scp (pscp)
  • Parallel rsync (prsync)
  • Parallel nuke (pnuke)
  • Parallel slurp (pslurp)

I'm not sure how easy it is to set up, but it might just do the trick!

2
  • 27
    The pssh utilities are used to spread commands across multiple servers, not do the same command a bunch of times on one server. In particular, prsync only supports sending a file on your local machine out to multiple external machines. It does not support downloading a remote file with multiple connections. Jul 24, 2012 at 23:29
  • 1
    Given @DerekDahmer's comment, the poster of this answer might want to withdraw it?
    – mc0e
    Mar 6, 2018 at 16:26
3

I cannot comment, so I have added a new answer, with a little bit better code than the previous (nice & smart) code.

Check the rsync line, because it contains an optional ionice tweak.

#!/bin/bash
start_time=$(date +%s.%N)
# Transfer files in parallel using rsync (simple script)
# MAXCONN: maximum number "rsync" processes running at the same time:
MAXCONN=6
# Source and destination base paths. (not need to end with "/")
SRC_BASE=/home/user/public_html/images
[email protected]:/home/user/public_html/images
RSYNC_OPTS="-ah --partial"
# Main loop:
for FULLDIR in $SRC_BASE/*; do
    NUMRSYNC=`ps -Ao comm | grep '^'rsync'$' | wc -l `
    while [ $NUMRSYNC -ge $MAXCONN ]; do
        NUMRSYNC=`ps -Ao comm | grep '^'rsync'$' | wc -l `
        sleep 1
    done
    DIR=`basename $FULLDIR`
    echo "Start: " $DIR
    ionice -c2 -n5 rsync $RSYNC_OPTS $SRC_BASE/${DIR}/ $DST_BASE/${DIR}/ &
    # rsync $RSYNC_OPTS $SRC_BASE/${DIR}/ $DST_BASE/${DIR}/ &
    sleep 5
done

execution_time=$(echo "$(date +%s.%N) - $start" | bc)
printf "Done. Execution time: %.6f seconds\n" $execution_time
2

Looks like someone has written this utility for you. It breaks the transfer into parallel chunks. This is a better implementation than the "parallel big file" version listed under GNU Parallel:

https://gist.github.com/rcoup/5358786

Also, lftp can parallelize file transfers via ftp, ftps, http, https, hftp, fish, sftp. A lot of times, there are some advantages to using lftp, because managing permissions, restricted access, etc for rsync can be challenging.

1
  • While this works, it can cause a massive amount of disk fragmentation quickly, as you're not simply using multiple connections to download the same file.
    – bparker
    Mar 19, 2018 at 19:27
1

No. No such feature exists. You could split the synch into multiple calls to rsync if you really wanted to.

I'd suggest you find whatever it is that's doing this rate-limiting and have a serious talk with whoever maintains/manages it.

1
  • 4
    Frequently those restrictions are from some ISP like Comcast. Good luck having any sort of reasonable conversation with them. Sep 16, 2012 at 21:46
1

I wanted to transfer several directories (with many files) at the same time, so I created this small script:

#!/bin/bash
# Transfer files in parallel using rsync (simple script)
# MAXCONN: maximum number "rsync" processes running at the same time:
MAXCONN=10
# Source and destination base paths. (not need to end with "/")
SRC_BASE=/home/sites
[email protected]:/var/www
RSYNC_OPTS="--stats -ilrtpog"
# Main loop:
for FULLDIR in $SRC_BASE/*/; do
    NUMRSYNC=`ps -Ao comm | grep '^'rsync'$' | wc -l `
    while [ $NUMRSYNC -ge $MAXCONN ]; do
        NUMRSYNC=`ps -Ao comm | grep '^'rsync'$' | wc -l `
        sleep 10
    done
    DIR=`basename $FULLDIR`
    rsync $RSYNC_OPTS $SRC_BASE/${DIR}/ $DST_BASE/${DIR}/ & 
    sleep 1 
done
echo "Done."

I did this script quite fast, so please revise it and test it before using in a production environment.

0

I created the following script to upload many folders with pictures in parallel. You run it with first the sync target and then all the folder names to copy.

#!/bin/sh

dest="$1"
shift

if [ "$dest" = "" ]; then
    echo "USAGE: $0 TARGET:/foo/bar <dir1> [dir2] [dir3]"
    exit 1
fi

RCol='\x1B[0m' # Text Reset
BYel='\x1B[1;33m';

for i in "$@"; do
    prefix=`printf "$BYel%50s:$RCol" "$i"`
    echo "$prefix * Starting $i"
    echo "$prefix -> syncing '$i/' to '$dest/$i/'"
    (rsync -rv "$i/" "$dest/$i/") 2>&1 | sed "s/^/$prefix /g" &
    sleep 0.5
done

echo "* Waiting for all to complete"
wait

It prefixes the folder name in yellow to all rsync console output to make it look pretty.

-1

Aria2 is a good client program to download data using many connections from many mirrors. It does not support SFTP. So, I've installed FTP server - vsftpd. My 3g connection works on full power with 5 connections to FTP server.

1
  • 1
    Would you care to expand on that to make your answer useful?
    – Tog
    Sep 4, 2013 at 8:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .