Load multiple files simulatenously with cURL

Question

I am looking to download multiple files from a server using cURL from the Terminal on a Mac. I know that using the {} operator, I can load one file after another but I am looking to download all (10) of them using multiple connections at the same time. Is there a way to do this (note: Not using PHP which seem to have a bunch of answers here)?

Answer 1

You can do this from your shell:

cat <<URLS | xargs -P5 -n1 curl -O
http://www.example.com/
http://www.example.net/
ftp://ftp.example.org/
URLS

This will download one file per curl process, with a maximum of 5 concurrent curl workers/processes.

Note that with xargs, it will separate the input by both spaces and new-lines. Since your URLs shouldn't contain spaces (or should have them encoded), this shouldn't be too significant.