How to send huge files from one server to another server

Question

I need to transfer a huge file (more than 70GB) from one server in Canada to another server in Africa.

I tried FTP, but always get disconnected somewhere. And I guess the network is not stable for the African server, some of the file got uploaded, but when I open it, it is corrupted.

So I really need a tool to transfer the huge file:

1. Utilize the bandwidth, so I can upload it ASAP.
2. If network not stable, the tool can detect it, drop the damaged data and upload it again.
3. I don't want to upload to other place (like most of the file sharing solution, upload to their server, and get one share link)
4. I prefer a tool which can be installed on both of the servers.

The OS is Windows Server 2008 R2 for both the servers. I cannot use the 3rd party transfer channel.

Answer 1

BitTorrent might be a good solution for you.

Create the torrent on the origin server, transfer the torrent file to the destination, and then use one of the clients as the bittorrent tracker. Bittorrent breaks up the file into manageable pieces and makes sure they're all transferred without error.

There's obviously some overhead in creating the torrentfile and there's no swarm to enhance your download speed, but if you're dealing with a crappy link, it may work very well for you.

Answer 2

FTP is awful -- it was born in the mid-1980s and, frankly, it should've died there.

I'd probably start with scp (Secure Copy), which should be part of the openssh or openssh-client package on your favorite Linux distro (including Cygwin), or available as part of the PuTTY package if you're running Windows without Cygwin. You'll need to configure an ssh server on the destination host, but that's pretty straightforward assuming you've got root/Administrator access (if you don't, things get harder); once you've got the ssh server running, and can access it from the source host, it's just a matter of

 user@source $ scp /path/to/file user@destination:/path/to/receiving/directory

This should satisfy your point 1 pretty well, since scp has reasonably low overhead; it should certainly satisfy point 2, as it will definitely detect a failing connection and can be (probably) configured or (certainly) scripted to retry as many times as necessary; it covers point 3 easily, as no intermediate host or service is required; and it covers point 4 nicely as well, since you can install an ssh server on both hosts and then transfer the file in whichever direction you prefer. You also get encryption for free, which may or may not be of use to you.

The OpenSSH manual is probably a good place to start, and I'll be glad to offer further assistance if you do end up going this route -- I have some experience in using scp/ssh for these sorts of transfers (although not from Canada to Africa or vice versa, and not for a single file topping 70GB in size, I admit!)

Hope this helps!

Answer 3

I think it would be a good idea to split the file into several small ones, transmit them, and then put them together again on the remote server.

An example (on Linux) of how to split and concatenate is here: http://www.techiecorner.com/107/how-to-split-large-file-into-several-smaller-files-linux/

You should also have something like an ssh connection to the remote server so that you can concatenate them there.

Helpful would also be a tool like "md5sum" to check if the transmitted files are unchanged by comparing the hashes.

You could either write a small shell script to automate some stuff, so that you can painlessly transmit a lot of small files, or you make it manually and try it with few but bigger chunks...

Answer 4

Depending on your upload and download speeds (usually, upload is the problem), your best way is to write it on a HD and send it via Fedex, DHL or similars for someone to copy it to the server if possible.

For example, if your upload speed is 1 Mbps, you will upload 1M / 8 = 128 KB/s. So, not counting any "problems" like overhead due to encryption (using scp for example) or connection not 100% full rate, you file will need 70G/128K = more than 500.000 s or 160h (more than 6 days). If your connection is not very stable, it will take (possibly a lot) more time.

Answer 5

I'd consider putting the file onto a set of DVDs or BD-ROMs or 2.5" HDD and airmail them.

If your upload bandwidth is 1 Mbit/s, 70 GB could take 6 days to transfer over the Internet.

Answer 6

put the file on the ftp site as you tried, but from Africa use

wget -c ftp.server.com/filename

-c will resume interrupted download

Answer 7

If you're using Mac OS X or Linux on both sides (?), then rsync might be your best bet.

Check out the manual pages here.

Answer 8

Mail.ru offers 100Gb of space. You could transfer your file that way. Of course, you have to be able to speak Russian, or maybe use a translating tool.

Answer 9

I know this is an old question, and someone's already suggested rsync, but the following concrete example worked well for me with moving huge files between servers:

rsync -r -v --progress --partial -e ssh user@remote-system:/path/to/remote/file /path/to/local/destination/

Whereby, you invoke this command on the destination side, such that rsync pulls the huge file from the remote server to a local directory. This example could be modified to snyc entire directories instead of just a single file.

Answer 10

I had the same situation. the destination server was remote but in the same country. so I used Team Viewer - File Transfer. It worked perfectly on windows server.