I have about 500K files (~250GB) I need to download off my server. I have attempted to download all of them several times but for some reason, my Win7 computer keeps restarting over night. This could be because of 'Windows auto-update', but that is not the issue now.

The problem is that each time I re-initiate the download, the FTP program (I have tried Filezilla and WinSCP) has to skip the files that have already been downloaded. This indexing process can take several hours in itself!

Is there a more efficient technique or program to download or resume downloading so many files?

  • Have you try to pack them somehow (zip, tar)? IMHO this will take in sum less time than filetransfer of so much files – Romeo Ninov Apr 17 '15 at 16:04
  • I tried that first but it was still taking so long that the overnight shutdowns would still break the download and then I had to start over from scratch the next morning. – Jeff Apr 17 '15 at 16:18
  • 2
    Maybe I should be focusing more on the shutdown issue instead of the FTP issue. – Jeff Apr 17 '15 at 16:19
  • Had the same issue. Resolved it by moving them using mget, with only first letter of the filename specified (then a wildcard), so that it took 26 passes to do the job. Of course, that only works if the files are not changing during the time it took to move them all. You can also pipe the file lists (ls) from each machine to files and do a text comparison afterwards to be sure you've got them all. – DrMoishe Pippik Apr 17 '15 at 20:35

I use lftp which can be run from Cygwin running on windows.

The merits of which have been highlighted before: unix stack exchange

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.