Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a ~35 GB file on a remote Linux Ubuntu server. Locally, I am running Windows XP, so I am connecting to the remote Linux server using SSH (specifically, I am using a Windows program called SSH Secure Shell Client version 3.3.2).

Although my broadband internet connection is quite good, my download of the large file often fails with a Connection Lost error message. I am not sure, but I think that it fails because perhaps my internet connection goes out for a second or two every several hours. Since the file is so large, downloading it may take 4.5 to 5 hours, and perhaps the internet connection goes out for a second or two during that long time. I think this because I have successfully downloaded files of this size using the same internet connection and the same SSH software on the same computer. In other words, sometimes I get lucky and the download finishes before the internet connection drops for a second.

Is there any way that I can download the file in an intelligent way -- whereby the operating system or software "knows" where it left off and can resume from the last point if a break in the internet connection occurs?

Perhaps it is possible to download the file in sections? Although I do not know if I can conveniently split my file into multiple files -- I think this would be very difficult, since the file is binary and is not human-readable.

As it is now, if the entire ~35 GB file download doesn't finish before the break in the connection, then I have to start the download over and overwrite the ~5-20 GB chunk that was downloaded locally so far.

Do you have any advice? Thanks.

share|improve this question
I believe rsync over ssh can do this. – Keith Jul 1 '12 at 21:25
@Keith Thanks. What is rsync? Does it run on Windows? – Andrew Jul 1 '12 at 21:27
@Andrew: Rsync is a program for incremental file transfers (syncing, backup, etc), and you can find a list of some Windows implementations here or by googling. – darvids0n Jul 2 '12 at 2:41

My 'proper' solution would be to find and fix what causes the problem, but these things might work as workarounds:

  1. use split ( man split -- split a file into pieces). It seems to be installed on most unix systems.
  2. Split the file via dd (dd if=inputfilename of=file_part0 skip=0 bs=500MB). (repeat with skip=1 and a different file name.)
  3. Use a program which can resume downloading. FTP would work (but eww. Plain text passwords, separate control and data connection which do not play nice with most firewalls)
  4. Cheat and move the file to a web directory. Most brosers supports resuming a download,
share|improve this answer

The bittorrent protocol would handle this task well though it may not be the easiest to set up.

share|improve this answer

You could split the file into smaller chunks and then reassemble them at the other end. Alternatively, if you have an FTP or HTTP daemon running, you ought to be able to run the download through the daemon, both of which usually support download resumption.

share|improve this answer
You typed a bit faster, but we seem to agree on solutions. :) – Hennes Jul 1 '12 at 21:18
up vote 1 down vote accepted

I used rsync --partial on Cygwin on Windows. It works great.

share|improve this answer

I think it is smart (and simple) to use the good and old WinSCP

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .