Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I need to download 10,000 zip files from a client's FTP that has about 40,000 items on it. I have contemplated doing:

$ cat > files.txt
file1
file2
file3
file4

with

$ wget -i files.txt

Is there a better solution for such a large amount in case I get timed out?

share|improve this question

3 Answers 3

up vote 2 down vote accepted

The solution is good and quite solid: with the proper options, wget will retry and download any file whose transfer got interrupted.

You can also script most command-line FTP clients, however (e.g. BSD ftp client). But wget is better in that you can configure it to not download files you already have, which makes it very convenient for syncs.

wget supports (on Linux at least) also rate limiting, and it is quite easy to distribute the file list between several files in order to download in parallel, or you can use GNU parallel.

share|improve this answer

wget is good and competent and will probably work fine in this case as mentioned by Iserni's answer, if you dig into the manual for available options. I'll just state some alternatives.

I use LFTP for transfers and syncing over both FTP and SFTP. It has an internal queuing system that works well for my use case, supports mirroring, reverse mirroring, FXP, all regular FTP functions and more (even Bittorrent nowadays).

In this case it seems simple enough to just use a script and wget, but I wanted to mention a program that simplified FTP transfers for me greatly.

I've also used NcFTP that has a very nice batch system, but the deal breaker in LFTP's favor for me was that it supported both FTP and SFTP.

Another alternative is rsync which also supports FTP, and perhaps you are already used to this. It also has quite advanced options ready as per filtering and resuming broken downloads.

An advantage with a "real" FTP client as compared to batch wget use is that a single connection to the server can be reused, which might boost performance. I'm not sure there is such an alternative for wget, but maybe there is. If this is a true one-off operation, you could probably use more or less anything.

share|improve this answer
    
+1 for lftp. I wasn't aware of such an utility, I think I'll check it out. Thanks! –  lserni Apr 16 '13 at 19:40

lftp is pretty good at this. Note the continue and expandnd wildcards OPTS.

lftp :~> help mirror
Usage: mirror [OPTS] [remote [local]]
Mirror specified remote directory to local directory

-c, --continue continue a mirror job if possible -e, --delete delete files not present at remote site --delete-first delete old files before transferring new ones -s, --allow-suid set suid/sgid bits according to remote site --allow-chown try to set owner and group on files --ignore-time ignore time when deciding whether to download -n, --only-newer download only newer files (-c won't work) -r, --no-recursion don't go to subdirectories -p, --no-perms don't set file permissions --no-umask don't apply umask to file modes -R, --reverse reverse mirror (put files) -L, --dereference download symbolic links as files -N, --newer-than=SPEC download only files newer than specified time -P, --parallel[=N] download N files in parallel -i RX, --include RX include matching files -x RX, --exclude RX exclude matching files RX is extended regular expression -v, --verbose[=N] verbose operation --log=FILE write lftp commands being executed to FILE --script=FILE write lftp commands to FILE, but don't execute them --just-print, --dry-run same as --script=-

When using -R, the first directory is local and the second is remote. If the second directory is omitted, basename of first directory is used. If both directories are omitted, current local and remote directories are used.

lftp :~> help mget
Usage: mget [OPTS] 
Gets selected files with expanded wildcards
 -c  continue, reget
 -d  create directories the same as in file names and get the
     files into them instead of current directory
 -E  delete remote files after successful transfer
 -a  use ascii mode (binary is the default)
 -O  specifies base directory or URL where files should be placed

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.