Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have 100,000 URLs of small files to download. Would like to use 10 threads and pipelining is a must. I concatenate the result to one file. Current approach is:

cat URLS | xargs -P5 -- curl >> OUTPUT

Is there a better option that will show progress of the whole operation? Must work from the command line.

share|improve this question
    
"Would like to use 10 threads and pipelining is a must. I concatenate the result to one file." So the order doesn't matter? –  Bobby Aug 16 '13 at 13:21
1  
Use GNU parallel, it will even keep the order of the output. If you tag your question accordingly, you might be lucky and the author might chime in ;-) –  Adrian Frühwirth Aug 16 '13 at 14:37
    
Order is not an issue. Tagged for gnu-parallel good idea. Is it possible to use parallel and still get the pipelining in curl? –  Full Decent Aug 16 '13 at 15:45
    
Don't you get the files intermingled when you do that? Unless your webserver is single-threaded, I don't see how you would avoid having two processes writing simultaneously to your output file. –  rici Aug 16 '13 at 16:30
    
Mangling, jumbling are all not a problem for me. –  Full Decent Aug 16 '13 at 20:18

1 Answer 1

up vote 3 down vote accepted
cat URLS | parallel -k -P10 curl >> OUTPUT

or if progress is more important:

cat URLS | parallel -k -P10 --eta curl >> OUTPUT

or:

cat URLS | parallel -k -P10 --progress curl >> OUTPUT

The 10 seconds installation will try do to a full installation; if that fails, a personal installation; if that fails, a minimal installation.

wget -O - pi.dk/3 | sh

Watch the intro video for a quick introduction: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

share|improve this answer
    
I had tried this installer wget -O - pi.dk/3 | sh but seem to have gotten some lame excuse for parallel that really does nothing: parallel [OPTIONS] command -- arguments / for each argument, run command with argument, in parallel –  Full Decent Aug 18 '13 at 14:55
1  
Ah, I had to uninstall moreutils first. apt-get remove moreutils –  Full Decent Aug 18 '13 at 15:01
    
Working great, thank you –  Full Decent Aug 18 '13 at 16:03

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.