2

How can I transfer a large file quickly, when I don't really require a secure connection?

Often I would like to copy a large file from local machine to remote machine. The file does not already exist on remote machine. I usually use scp or rsync (which i think might use ssh underneath anyway). It can be kind of slow and I suspect it's not due to network limitations, but because it's trying to copy securely. Using blowfish cipher doesn't improve things much, and I couldn't get null cipher to work at all.

edit: for the record, copying a ~75MB file took about 17 minutes, and rsync reported 78032.72 bytes/sec. If my calculations are correct and the network is 10/100 (100,000,000 bits/sec) then isn't that only about 0.6% of the possible network speed?!

8
  • This ServerFault Question may be of interest to you. Try FTP maybe?
    – digitxp
    Aug 30, 2011 at 3:53
  • 2
    I think your issue is not with the encryption - I regularly get much better speeds than that using scp even over encrypted wi-fi (802.11b or g). (for what it's worth, using scp, I generally see something close to 15-16 Mbit/s under optimal conditions, several times that on a wired connection).
    – user55325
    Aug 30, 2011 at 5:49
  • 1
    Is the whole path from the local machine to the remote machine 100Mb/s? A network path cannot transfer data between two points faster than it can across the slowest link in the path. If you actually have a 100Mb/s path between them then are they close enough that you could just put the file on a USB stick and walk it to its destination?
    – mas
    Aug 30, 2011 at 6:29
  • 2
    @digitxp Using the file transfer protocol to transfer a file? Who would want to do that? You must be joking!
    – Daniel Beck
    Aug 30, 2011 at 19:50
  • 1
    See also unix.stackexchange.com/questions/48399/…
    – rogerdpack
    Sep 24, 2018 at 19:00

3 Answers 3

1

You could try mirrordir - more here. Mirrordir is faster than rsync because it has no delta checks. However, even on their man page they claim it is dangerous command. Also, I believe is not supported anymore. Your pure network bandwidth test is with dd and netcat, as demonstrated here

2
  • thanks - could you show me how to use dd and netcat to check bandwidth test? i imagine it could be a huge mistake if i accidentally use dd incorrectly.. :)
    – wim
    Aug 30, 2011 at 5:18
  • 1
    Link for dd & netcat added.
    – grs
    Aug 30, 2011 at 5:37
2

Check this thread from serverfault.com and my reply:

If you want speed you could use netcat and tar. It will be faster than ssh, rsync, or scp on a local network where encryption is not a concern. Google "netcat tar".

DestinationServer

nc -l -p 7878 | tar -C /target/dir -xzf -

SourceServer

tar -cz /source/dir | nc DestinationServer 7878

This obviously requires that netcat is actually installed. Google "netcat tar" for more info.

1

Use bzip2 to compress the file and see if it can reduce the size to be transferred. Compression should be very effective for text files, but will not do much on JPGs or videos.

By the way, a uploading bandwidth of around 512 Kbps is very possible, if you are using a home internet connection.

1
  • You may also want to try 7zip or Rar as they have special file filters tailored to get the best compression for that particular type of file.
    – jftuga
    Aug 30, 2011 at 20:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.