1

I want to transmit a 1TB file over the internet. I have control over both machines (sender unix, receiver linux). I can open FTP, OpenVPN, NFS, ...

So far resuming for some reason is not stable on FTP, so I cut the file in 1GB pieces and transmit file per file, then md5 on the other side. It's very annoying.

I have between 3 and 6 mbytes/sec to the other site. Which is a pretty decent speed.

I want it to be safe and I want resuming if it fails. I tried NFS - no resuming.

Is there a cleaner, safer way?

Improve this question
5
  • Should we assume a Linux OS?
    – Brian Adkins
    May 14, 2014 at 23:06
  • yes (*nix), I will update the question
    – mist
    May 14, 2014 at 23:15
  • If you can afford it (costs and availability), you could sent the disc and copy it locally. This doesn't cover the explicit question though.
    – Ton Plomp
    May 15, 2014 at 0:08
  • rsync would do I think
    – Ashtray
    May 15, 2014 at 5:39
  • I tested rsync - it resumes after restart (or network issues) and copies successfully (md5 checked).
    – mist
    May 15, 2014 at 18:00

2 Answers 2

Reset to default
2

If you need restart, rsync is probably the way to go. One of the fastest ways over a network, I've found is with "Samba" -- but to get excellent performance it usually needs to be tuned for the environment. But what I mean by "fast" is on a 1Gbit ethernet, 125MB/s writes and 119MB/s reads. The 125MB/s writes are as fast as you are going to get unless your payload is able to be compressed -- i.e. a 1TB text file would likely compress to 1/10th the size.

Note... problem w/rsync, "direct" is that it generally uses some other protocol (rsh/ssh) to do do the transfer -- if you use rsync over 'ssh' you will incur an encryption cost, which on a good machine adds a ceiling of 140-160MB/s. That will usually drive up latency on a network connection and slow down overall transfer be 50% or more.

So the fastest way depends on what type of network you have in place between the two machines. Slow-ish Internet speeds (i.e. .5-10MB/s), or a local area network (w/1Gb or perhaps even 10Gb).

If transfer speed is most important and more so than overall speed, i'd "prep" the file for transfer by running it through a good compressor (like xz or 7z) -- BUT that will take a large chunk of time by itself (so overall time is likely to be larger). But if actual transfer speed is more important, then compression becomes a good way to cut down on that.

It really depends on where your priorities are and how much time you want to spend optimizing the transfer speed. Overall, though, I think sirlancelot gave the right answer, just that there can be many mitigating factors depending on your priorities.

Improve this answer
6
  • 1
    Try 500MB/s with AES-128 on a CPU that supports AES-NI - any halfway decent modern CPU will. Also, 125MB/s is literally impossible with a 1Gbit/s line - there will be overhead, at the transport and network layers even without considering the application protocol. I'm not certain I trust your numbers.
    – Bob
    May 15, 2014 at 2:00
  • 1
    Without very significant compression (which is only achievable on extremely low entropy data), you're never going to see 125 MB/s in any direction on GbE. Protocol overhead at the IP, TCP/UDP, and application layers will account for more than a trivial amount of the throughput, especially over the public Internet. On a point to point link (e.g. a patch cable between two systems' NICs) you may be able to get closer using jumbo IP frames and a very low chattiness protocol, but even if you could get it to burst to 124 MB/s, it would never sustain that consistently.
    – allquixotic
    May 15, 2014 at 2:06
  • And even with compression, you're still going to be sending less than 125 MB/s of compressed data per second, no matter how much actual (decompressed) data is being sent. Obviously if you 7-zip a file full of a quadrillion zero bytes, you can achieve transfer speeds in the terabytes per second (of decompressed data) by sending the couple-KB 7z image over even a WAN network. But that's disingenuous to the point.
    – allquixotic
    May 15, 2014 at 2:08
  • Sorry, but it was measured multiple times. Other people on the samba list have had similar experiences. You have to realize I'm using 9k packets at the IP level, and generally around 32MB of buffer space/socket as default. On the windows side you allocate packet buffers. @ 4k of packet buffers x 9000 bytes=>nearly 18MB on the windows side for buffering. The overhead is low -- the writes are faster because multiple packets can be outstanding w/out waiting for ACKS from the receiver. @ Full duplex switched networking, there is no packet loss or retransmission. (previous comment incomplet)
    – Astara
    May 15, 2014 at 5:02
  • As for encryption speed, I've tried various ssh algorithms, but I don't think I've tried any hardware assisted algorithms. If you want 500MB/s -- you have to be using hardware with specialized instructions. As for the compression angle --- I very clearly stated it depended on what the poster's priorities were. Was it minimum transmission time, or minimal time overall, I didn't get that they were going for fastest MB/s transfer rate, which is why I agreed that rsync would be good as it can be restarted...
    – Astara
    May 15, 2014 at 5:08
5

I think rsync allows resuming previous transfers.

$ rsync -aP src/ dest:dir/

man rsync

Improve this answer
3
  • is it stable enough?
    – mist
    May 14, 2014 at 23:28
  • 1
    Yes, rsync has been around since late '90s and it's available on most Linux distros or available as a package to install.
    – matpie
    May 14, 2014 at 23:50
  • resuming is problematic: If I transfer 1tb and the transfer stops at 500gb, and I then resume, rsync starts rehashing locally and remotely, which takes hours. Can I cache the hashing?
    – mist
    Dec 7, 2014 at 16:14

Not the answer you're looking for? Browse other questions tagged .