Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have 100,000 URLs of small files to download. Would like to use 10 threads and pipelining is a must. I concatenate the result to one file. Current approach is:

cat URLS | xargs -P5 -- curl >> OUTPUT

Is there a better option that will show progress of the whole operation? Must work from the command line.

share|improve this question
    
"Would like to use 10 threads and pipelining is a must. I concatenate the result to one file." So the order doesn't matter? – Bobby Aug 16 '13 at 13:21
1  
Use GNU parallel, it will even keep the order of the output. If you tag your question accordingly, you might be lucky and the author might chime in ;-) – Adrian Frühwirth Aug 16 '13 at 14:37
    
Order is not an issue. Tagged for gnu-parallel good idea. Is it possible to use parallel and still get the pipelining in curl? – Full Decent Aug 16 '13 at 15:45
    
Don't you get the files intermingled when you do that? Unless your webserver is single-threaded, I don't see how you would avoid having two processes writing simultaneously to your output file. – rici Aug 16 '13 at 16:30
    
Mangling, jumbling are all not a problem for me. – Full Decent Aug 16 '13 at 20:18
up vote 3 down vote accepted
cat URLS | parallel -k -P10 curl >> OUTPUT

or if progress is more important:

cat URLS | parallel -k -P10 --eta curl >> OUTPUT

or:

cat URLS | parallel -k -P10 --progress curl >> OUTPUT

The 10 seconds installation will try do to a full installation; if that fails, a personal installation; if that fails, a minimal installation.

wget -O - pi.dk/3 | sh

Watch the intro video for a quick introduction: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

share|improve this answer
    
I had tried this installer wget -O - pi.dk/3 | sh but seem to have gotten some lame excuse for parallel that really does nothing: parallel [OPTIONS] command -- arguments / for each argument, run command with argument, in parallel – Full Decent Aug 18 '13 at 14:55
1  
Ah, I had to uninstall moreutils first. apt-get remove moreutils – Full Decent Aug 18 '13 at 15:01
    
Working great, thank you – Full Decent Aug 18 '13 at 16:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .