0

I have about 500K files (~250GB) I need to download off my server. I have attempted to download all of them several times but for some reason, my Win7 computer keeps restarting over night. This could be because of 'Windows auto-update', but that is not the issue now.

The problem is that each time I re-initiate the download, the FTP program (I have tried Filezilla and WinSCP) has to skip the files that have already been downloaded. This indexing process can take several hours in itself!

Is there a more efficient technique or program to download or resume downloading so many files?

4
  • Have you try to pack them somehow (zip, tar)? IMHO this will take in sum less time than filetransfer of so much files Apr 17, 2015 at 16:04
  • I tried that first but it was still taking so long that the overnight shutdowns would still break the download and then I had to start over from scratch the next morning.
    – Jeff
    Apr 17, 2015 at 16:18
  • 2
    Maybe I should be focusing more on the shutdown issue instead of the FTP issue.
    – Jeff
    Apr 17, 2015 at 16:19
  • Had the same issue. Resolved it by moving them using mget, with only first letter of the filename specified (then a wildcard), so that it took 26 passes to do the job. Of course, that only works if the files are not changing during the time it took to move them all. You can also pipe the file lists (ls) from each machine to files and do a text comparison afterwards to be sure you've got them all. Apr 17, 2015 at 20:35

1 Answer 1

0

I use lftp which can be run from Cygwin running on windows.

The merits of which have been highlighted before: unix stack exchange

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .