Persistent retrying resuming downloads with curl

Question

I'm on a mac and have a list of files I would like to download from an ftp server. The connection is a bit buggy so I want it to retry and resume if connection is dropped.

I know I can do this with wget, but unfortunately Mac OS X doesn't come with wget. I could install it, but to do that (unless I have missed something) I need to install XCode and MacPorts first, which I would like to avoid.

Curl is available though it seems, but I don't know how that works or how to use it really. If I have a list of files in a text file (one full path per line, like ftp://user:pass@server/dir/file1) how can I use curl to download all those files? And can I get curl to never give up? Like, retry infinitely and resume downloads where it left off and such?

Answer 1

After googling and man reading and such for a while I have figured out a solution that worked for me:

curl ftp://server/dir/file[01-30].ext --user user:pass -O --retry 999 --retry-max-time 0 -C -

• [01-30] will make it download 30 files named file01.ext, file02.ext and so on
• --user user:pass should be obvious
• -O to output to files with original name
• --retry 999 to retry 999 times
• --retry-max-time 0 to prevent it from timing out the retrys. The default behavior if you don't specify a fixed --retry-delay is to sleep first one second between retries, then doubling that, until it reaches 10 min. between retries
• -C - to make it continue where it dropped of (if you run the command again). The dash afterwards tells it to figure out where to resume from

If someone knows how to get the filenames from a file instead, please let me know.

---

In the man page it says the following:

--url <URL>
Specify a URL to fetch. This option is mostly handy when you want to specify URL(s) in a config file.

Seems like that could be something, but don't quite understand how it would be used...

Answer 2

You can use the -K option to curl to specify a config file. In that case, you use the syntax:

optionname=<value>

If you have a file containing a list of URL's, you can use curl like this:

sed 's/\(.*\)/-O\nurl=\1/g' url_list.txt | curl -K -

which transforms a list of urls, like:

http://host1.com/foo.html
http://host2.com/bar.html

into a format like this:

-O
url=http://host1.com/foo.html
-O
url=http://host2.com/bar.html

Curl reads that and interprets each of those like options.

HTH,

Adam