1

I usually download files on parallel with curl and urls in a file like this:

cat links.txt | parallel --will-cite curl --connect-timeout 5 --max-time 10 --retry-max-time 40 --retry 5 --retry-delay 0 -s -f -O -C -

Where links.txt has one url per line.

Now I need to assign a custom filename to each file and I can't figure out how to add this second input to curl trough parallel. Among other things, I tried to add -o filename to each line of links.txt but it didn't work.

Regards,

4
  • Show us how you would do it if you had to do 3 "by hand" (i.e. without using GNU Parallel). (And consider running parallel --bibtex once).
    – Ole Tange
    Nov 15, 2017 at 7:05
  • copy-paste and change url an filename of this: curl --connect-timeout 5 --max-time 10 --retry-max-time 40 --retry 5 --retry-delay 0 -s -f -C - -o filename1 url1 Nov 15, 2017 at 13:24
  • Do you really want all urls saved into filename1?
    – Ole Tange
    Nov 16, 2017 at 8:45
  • nop. copy-paste means curl ... filename1 url1, curl ... filename2 url2 , etc Nov 16, 2017 at 15:04

1 Answer 1

1

According to man parallel, you can use some placeholders to aid you. For example, you could rewrite your code to:

parallel curl "${CURL_ARGS[@]}" -o '{#}'.curl_output '{}' :::: links.txt

where ${CURL_ARGS[@]} are all your arguments to curl and links.txt has one URL per line. This command you fetch the URLs in links.txt and will store the result in a file named after the number of the job than run curl (e.g., 10.curl_output). You would have to march job number with URLs.

Another approach would be to call curl with -o '{}'._curl_output. In this case, you would have to deal with special characters in URLs (/, for example).

Besides those, you could also split the columns on a links.txt which contains an URL and an output file per line. This would let you do

parallel --colsep " " curl "${CURL_ARGS[@]}" -o '{1}' '{2}' :::: links.txt

The --colsep will split the lines based on the delimiter provide as its argument (" " in this case).

1
  • Thanks! Your third option was exactly what I was looking for. Nevertheless, the other options can be useful in other situations... Nov 16, 2017 at 0:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.