Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am looking to download multiple files from a server using cURL from the Terminal on a Mac. I know that using the {} operator, I can load one file after another but I am looking to download all (10) of them using multiple connections at the same time. Is there a way to do this (note: Not using PHP which seem to have a bunch of answers here)?

share|improve this question

migrated from stackoverflow.com May 4 '11 at 12:09

This question came from our site for professional and enthusiast programmers.

    
Flagged for relocation to superuser site. –  ewindisch May 3 '11 at 21:16

1 Answer 1

up vote 2 down vote accepted

You can do this from your shell:

cat <<URLS | xargs -P5 -n1 curl -O
http://www.example.com/
http://www.example.net/
ftp://ftp.example.org/
URLS

This will download one file per curl process, with a maximum of 5 concurrent curl workers/processes.

Note that with xargs, it will separate the input by both spaces and new-lines. Since your URLs shouldn't contain spaces (or should have them encoded), this shouldn't be too significant.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.