Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm on a mac and have a list of files I would like to download from an ftp server. The connection is a bit buggy so I want it to retry and resume if connection is dropped.

I know I can do this with wget, but unfortunately Mac OS X doesn't come with wget. I could install it, but to do that (unless I have missed something) I need to install XCode and MacPorts first, which I would like to avoid.

Curl is available though it seems, but I don't know how that works or how to use it really. If I have a list of files in a text file (one full path per line, like ftp://user:pass@server/dir/file1) how can I use curl to download all those files? And can I get curl to never give up? Like, retry infinitely and resume downloads where it left off and such?

share|improve this question
add comment

2 Answers

up vote 1 down vote accepted

After googling and man reading and such for a while I have figured out a solution that worked for me:

curl ftp://server/dir/file[01-30].ext --user user:pass -O --retry 999 --retry-max-time 0 -C -
  • [01-30] will make it download 30 files named file01.ext, file02.ext and so on
  • --user user:pass should be obvious
  • -O to output to files with original name
  • --retry 999 to retry a lot
  • --retry-max-time 0 to prevent it from timing out on retrying
  • -C - to make it continue where it dropped of (if you run the command again). The dash afterwards tells it to figure out where to resume from

If someone knows how to get the filenames from a file instead, please let me know.


In the man page it says the following:

--url <URL>
Specify a URL to fetch. This option is mostly handy when you want to specify URL(s) in a config file.

Seems like that could be something, but don't quite understand how it would be used...

share|improve this answer
add comment

You can use the -K option to curl to specify a config file. In that case, you use the syntax:

optionname=<value>

If you have a file containing a list of URL's, you can use curl like this:

sed 's/\(.*\)/-O\nurl=\1/g' <i>url_list.txt</i> | curl -K -

which transforms a list of urls, like:

http://host1.com/foo.html
http://host2.com/bar.html

into a format like this:

-O
url=http://host1.com/foo.html
-O
url=http://host2.com/bar.html

Curl reads that and interprets each of those like options.

HTH,

Adam

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.