Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have been using a plain 'ole rsync command to move large amounts of data off of some hard drives on a LAN to another computer for analysis. Usually just a one-time transfer. The format of rsync that I have been using is:

rsync -ave ssh foo@bar:/source dest

I need to improve the speed of my transfers as much as possible. Someone recently mentioned setting up rsh-server and copying the files through RSH. Not having an encryption overhead can be very helpful. I suppose I have these options:

  • cp, scp, etc.
  • rsync via rsh
  • rsync via ssh, arcfour encryption for reducing speed
  • rsyncd, no encryption, some benefits by not having to build the file list

I'm curious as to which I can expect the best performance from. I know, every case is different. Are there any better alternatives?

Some notes:

Ease of install is great. Setting up an RSH server is a bit of a pain. Rsyncd setup is reasonable. Security is not a big concern.

All are 7200 rpm drives, gigabit network with speedy CPU (at least dual core i7 2.0ghz).

share|improve this question
    
Let's think out of the box. The speed limitation is mostly due to network. If you rsync to external disk using USB3, or e-SATA, it should be faster. You can just bring the disk to the other computer and perform the analysis there. Easier to setup as well. –  Sharuzzaman Ahmat Raslan Jan 11 '13 at 8:16

3 Answers 3

If you need to improve speed of transfer , use command with -z options , its for compression.

rsync -avze ssh foo@bar:/source dest

The compression is done using zlib libraries,so the transferred number of bytes will be less.

share|improve this answer

At work, we use rsyncd to backup servers on the same LAN. I think it gives maximum transfer speed. Almost no protocol overhead.

However, same as rsync over SSH, it will take time to build the file list. Building the file list is very fast, unless for very huge file trees.

rsyncd setup is very easy and powerful. See man rsyncd.conf

share|improve this answer

you write 'Security is not a big concern' and 'Not having an encryption overhead can be very helpful'.

You then should use rsync without encrypting overhead. Just use a double colon (::) separator after a host specification. See man rsync for details. For 'a one-time transfer' a temporary wildcard setup on the server side may be helpful as outlined here: File copy utility or file manager with each byte report while copying?

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.