Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

How can I transfer a large file quickly, when I don't really require a secure connection?

Often I would like to copy a large file from local machine to remote machine. The file does not already exist on remote machine. I usually use scp or rsync (which i think might use ssh underneath anyway). It can be kind of slow and I suspect it's not due to network limitations, but because it's trying to copy securely. Using blowfish cipher doesn't improve things much, and I couldn't get null cipher to work at all.

edit: for the record, copying a ~75MB file took about 17 minutes, and rsync reported 78032.72 bytes/sec. If my calculations are correct and the network is 10/100 (100,000,000 bits/sec) then isn't that only about 0.6% of the possible network speed?!

share|improve this question
    
This ServerFault Question may be of interest to you. Try FTP maybe? –  digitxp Aug 30 '11 at 3:53
2  
I think your issue is not with the encryption - I regularly get much better speeds than that using scp even over encrypted wi-fi (802.11b or g). (for what it's worth, using scp, I generally see something close to 15-16 Mbit/s under optimal conditions, several times that on a wired connection). –  user55325 Aug 30 '11 at 5:49
1  
Is the whole path from the local machine to the remote machine 100Mb/s? A network path cannot transfer data between two points faster than it can across the slowest link in the path. If you actually have a 100Mb/s path between them then are they close enough that you could just put the file on a USB stick and walk it to its destination? –  mas Aug 30 '11 at 6:29
    
I'm sure the path isn't 100Mb/s but i doubt it's as bad as 0.6% of that either. I could use a USB stick, but that would mean.. getting the keys to the server room from the authorized person, taking down a keyboard, taking the lift to L27.. hmm and my seat is more comfortable ! :) –  wim Aug 30 '11 at 6:32
2  
@digitxp Using the file transfer protocol to transfer a file? Who would want to do that? You must be joking! –  Daniel Beck Aug 30 '11 at 19:50

3 Answers 3

up vote 1 down vote accepted

You could try mirrordir - more here. Mirrordir is faster than rsync because it has no delta checks. However, even on their man page they claim it is dangerous command. Also, I believe is not supported anymore. Your pure network bandwidth test is with dd and netcat, as demonstrated here

share|improve this answer
    
thanks - could you show me how to use dd and netcat to check bandwidth test? i imagine it could be a huge mistake if i accidentally use dd incorrectly.. :) –  wim Aug 30 '11 at 5:18
1  
Link for dd & netcat added. –  grs Aug 30 '11 at 5:37

Use bzip2 to compress the file and see if it can reduce the size to be transferred. Compression should be very effective for text files, but will not do much on JPGs or videos.

By the way, a uploading bandwidth of around 512 Kbps is very possible, if you are using a home internet connection.

share|improve this answer
    
You may also want to try 7zip or Rar as they have special file filters tailored to get the best compression for that particular type of file. –  jftuga Aug 30 '11 at 20:09

Check this thread from serverfault.com and my reply:

If you want speed you could use netcat and tar. It will be faster than ssh, rsync, or scp on a local network where encryption is not a concern. Google "netcat tar".

DestinationServer

nc -l -p 7878 | tar -C /target/dir -xzf -

SourceServer

tar -cz /source/dir | nc DestinationServer 7878

This obviously requires that netcat is actually installed. Google "netcat tar" for more info.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.