2

I always got the impression that rsync should be at least as fast as scp. But here I have a problem. When I use rsync -avr to copy data from the remote machine, I get something like 11 mb/s. If I use scp on the same file, I get 28 mb/s.

How come? It is a directiory with many files, but the largest one is a single 1 GB file. I mean, something is wrong with the connection, it should be 1 GBit and I only get ~300 mbit, but still, its weird that rsync is so much slower here...

Any suggestions? Thx.

  • 1
    Don't take this the wrong way, but are you sure the actual time it takes to transfer the data set is actually longer with rsync? The reason I ask is I created a set of 1000 files, each 1 MB large, and compared the results of transferring them with "rsync -avr" and "scp -pr" respectively. Just looking at the console output, scp reports transfer speeds for each individual file that are significantly faster than the average speed of rsync, but actually clocking each transfer (prepending the command with "time") reveals that rsync finished in a considerably shorter time. – anlag Feb 19 '18 at 10:02
  • I got this from the --progress output, and when it is transfering a large file it sticks around 11 mb/s. – Pythoneer Feb 19 '18 at 10:20
  • My experience is that copying files across a Gigabit connection indeed allows transfers at near 100MBytes/sec with rsync. 10-11Mbytes/second, was what I got until I figured out that something in the way was only a 100Mbit interface (a "500Mbit" PCL adapter with two Ethernet ports). Some file transfer protocols can be sensitive to ping times. – xenoid Feb 19 '18 at 10:53
  • I mounted a directory from the remote machine on the target machine and did an rsync from the mounted directory to the local directory. This now gives me around 80 mb/s ... I still do not understand it ... – Pythoneer Feb 19 '18 at 13:30
0

One difference between scp and rsync is that rsync verifies transfered files whereas scp doesn't.

From the rsync manpage:

Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking a whole-file checksum that is generated as the file is transferred, but that automatic after-the-transfer verification has nothing to do with this option’s before-the-transfer "Does this file need to be updated?" check.

So this could be one source for a performance difference.

| improve this answer | |
  • I did try --no-checksum, but it had the same result. – Pythoneer Feb 19 '18 at 10:00
0

.. when it is transfering a large file it sticks around 11 mb/s.

without knowning, you could've caused a partial transfer when comparing transfer rates, making the CPU a bottleneck.

Rsync does a delta-transfer by default if it sees a partial file on the receiver (leftover from the scp test?). The "rolling checksum" method of comparing what needs to be transfered will take its toll in CPU usage on the Sender side.

I noticed a 4-fold decrease in transfer rate on a large partial binary file with a weakish CPU as Sender (Intel Atom or smaller VM cloud instances). On bigger machines the transfer rate stayed up after the initial partial file scan when continuing. The difference in throughput to plain netcat then should be the encryption via ssh.

Explicitly telling rsync to not to use partial transfers (-W|--whole-file) will put transfer rates in the same ballpark again. If needed, --append is another way to resume an interrupted file transfer without causing a delta-transfer. get -a in sftp can resume a download too.

See Rsyncs architecture description for further explanation.

| improve this answer | |
0

You forgot about your HDD speed. SATA 2 or SATA 3? Max write speed for SATA 2 would be approx 300mb/s and for SATA 3 600mb/s, so let's say, the real speed you can get is 35-150mb/s. Only NVME drive would be able to handle 1gbps network speed, so you could get 600-800mb/s. :-)

P.S. Use rsync -a --progress /home...Zip process is killing your CPU, this way transfer will go up to max of your HDD write speed.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.