Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I need to transfer a huge file (more than 70GB) from one server in Canada to another server in Africa.

I tried FTP, but always get disconnected somewhere. And I guess the network is not stable for the African server, some of the file got uploaded, but when I open it, it is corrupted.

So I really need a tool to transfer the huge file:

  1. Utilize the bandwidth, so I can upload it ASAP.
  2. If network not stable, the tool can detect it, drop the damaged data and upload it again.
  3. I don't want to upload to other place (like most of the file sharing solution, upload to their server, and get one share link)
  4. I prefer a tool which can be installed on both of the servers.

The OS is Windows Server 2008 R2 for both the servers. I cannot use the 3rd party transfer channel.

share|improve this question
3  
I like rsync for this purpose. Give it the --partial flag and it will save any progress if it's disconnected. –  David Schwartz Oct 1 '12 at 23:12
1  
What OS on both sides? –  grs Oct 2 '12 at 0:13
    
The suggested exact duplicate is not, as it doesn't address the questioner's restriction against using a third-party service to facilitate the file transfer. –  herrtodd Oct 2 '12 at 18:25
    
@grs, windows server 2008 R2 –  Robby Shaw Oct 2 '12 at 22:53
    
After 5 days you want to send another 70gb. Are these 100% different from the initial 70gb? How much data changes in these 5 days? If the delta is relatively small it may be wise to airmail the first 70gb and then come up with a scheme to just transmit the deltas. –  akira Oct 3 '12 at 4:24
show 4 more comments

7 Answers

up vote 7 down vote accepted

Bittorent might be a good solution for you.

Create the torrent on the origin server, transfer the torrent file to the destination, and then use one of the clients as the bittorrent tracker. Bittorrent breaks up the file into manageable pieces and makes sure they're all transferred without error.

There's obviously some overhead in creating the torrentfile and there's no swarm to enhance your download speed, but if you're dealing with a crappy link, it may work very well for you.

Info on using BitTorrent's built in tracker.

share|improve this answer
    
That might be a good solution because the torrent client would automatically detect corrupt pieces and dropped connections and just keep on going until it’s done. One side would have to set up a tracker, but µTorrent has a built-in mini-tracker, so that’s not a problem. You can create the torrent file (which is tiny), set it to private, tweak the network settings (you don’t want to limit it unnecessarily) and let it run to completion. –  Synetech Oct 2 '12 at 0:51
    
That's looks great. But is there any security risk? –  Robby Shaw Oct 2 '12 at 23:00
    
You guys save my life. It works and everything just great. However, it took some times for me to set up the torrent file. I have to change the advance setting first (enable tracker), then change the tracker address to my local address and port. –  Robby Shaw Oct 3 '12 at 6:06
    
@Robby Shaw - Just to satisfy my personnal curiosity and knowledge, how much time did it took to transfer the 70GB this way? –  laurent Nov 7 '12 at 16:55
add comment

If you're using Mac OS X or Linux on both sides (?), then rsync might be your best bet.

Check out the manual pages here.

share|improve this answer
    
Sorry, forgot to mention it is Windows OS. Thanks anyway –  Robby Shaw Oct 2 '12 at 23:04
add comment

put the file on the ftp site as you tried, but from Africa use

wget -c ftp.server.com/filename

-c will resume interrupted download

share|improve this answer
add comment

Depending on your upload and download speeds (usually, upload is the problem), your best way is to write it on a HD and send it via Fedex, DHL or similars for someone to copy it to the server if possible.

For example, if your upload speed is 1 Mbps, you will upload 1M / 8 = 128 KB/s. So, not counting any "problems" like overhead due to encryption (using scp for example) or connection not 100% full rate, you file will need 70G/128K = more than 500.000 s or 160h (more than 6 days). If your connection is not very stable, it will take (possibly a lot) more time.

share|improve this answer
    
Because we will transfer the files regularly (every 5~10 days), so I don't think sending a HD is doable. My upload speed is quite fast, but it seems the other site got packet lost –  Robby Shaw Oct 2 '12 at 22:56
add comment

I'd consider putting the file onto a set of DVDs or BD-ROMs or 2.5" HDD and airmail them.

If your upload bandwidth is 1 Mbit/s, 70 GB could take 6 days to transfer over the Internet.

share|improve this answer
    
Because we will transfer the files regularly (every 5~10 days), so I don't think sending a HD is doable –  Robby Shaw Oct 2 '12 at 23:00
add comment

I think it would be a good idea to split the file into several small ones, transmit them, and then put them together again on the remote server.

An example (on Linux) of how to split and concatenate is here: http://www.techiecorner.com/107/how-to-split-large-file-into-several-smaller-files-linux/

You should also have something like an ssh connection to the remote server so that you can concatenate them there.

Helpful would also be a tool like "md5sum" to check if the transmitted files are unchanged by comparing the hashes.

You could either write a small shell script to automate some stuff, so that you can painlessly transmit a lot of small files, or you make it manually and try it with few but bigger chunks...

share|improve this answer
    
Yup, if no good tool for this. I will probably use your method. But since the file is more than 70GB, i need to split it to a lot of files and check the MD5 sum for all of them. It sounds too much work for me :) –  Robby Shaw Oct 2 '12 at 23:03
add comment

FTP is awful -- it was born in the mid-1980s and, frankly, it should've died there.

I'd probably start with scp (Secure Copy), which should be part of the openssh or openssh-client package on your favorite Linux distro (including Cygwin), or available as part of the PuTTY package if you're running Windows without Cygwin. You'll need to configure an ssh server on the destination host, but that's pretty straightforward assuming you've got root/Administrator access (if you don't, things get harder); once you've got the ssh server running, and can access it from the source host, it's just a matter of

 user@source $ scp /path/to/file user@destination:/path/to/receiving/directory

This should satisfy your point 1 pretty well, since scp has reasonably low overhead; it should certainly satisfy point 2, as it will definitely detect a failing connection and can be (probably) configured or (certainly) scripted to retry as many times as necessary; it covers point 3 easily, as no intermediate host or service is required; and it covers point 4 nicely as well, since you can install an ssh server on both hosts and then transfer the file in whichever direction you prefer. You also get encryption for free, which may or may not be of use to you.

The OpenSSH manual is probably a good place to start, and I'll be glad to offer further assistance if you do end up going this route -- I have some experience in using scp/ssh for these sorts of transfers (although not from Canada to Africa or vice versa, and not for a single file topping 70GB in size, I admit!)

Hope this helps!

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.