Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am using cURL to download many files and concatenate them to STDOUT. About 100,000 small files. I would like to see progress against the 100,000. Is this possible with curl or by using curl into xargs?

Interested in only standard command line solutions.

share|improve this question

migrated from stackoverflow.com Dec 16 '12 at 19:13

This question came from our site for professional and enthusiast programmers.

    
it is possible to display something like: "downloading [200/100000]". but as you said, you want to print the file contents to stdout. if you got status/progress information, they should be printed to stdout as well. it will mess your stdout up. what do you exactly want? –  Kent Dec 14 '12 at 14:37
add comment

2 Answers

There are a few different things you could do here: but without knowing exactly your curl methodology I can just offer a few suggestions.

Make an iterative counting for loop:

for file {1..100000}; do echo "downloading: $file" >&2 ; curl [whatever] ; done This will redirect the "downloading: $file"to STDERR so if you're using a redirect on STDOUT it won't mess it it up but you can still see it on the screen

for file {1..100000}; do echo "downloading: $file" >> progress.out ; curl [whatever] ; done This writes progress to a file so it won't show up on the screen at all, then you can just tail -f progress.out

If your curl isn't iterative:

terminal 1: curl [whatever] | tee progress.out

terminal 2: watch -n5 "grep -c '[unique file delimiter]' progress.out"

tee writes a copy of STDOUT to a file, and you grep count for something that only appears once per HTML doc... maybe a <HEAD> or <HTML> tag or something else. Watch will just run the grep it every 5 seconds to let you know how many you have completed.

share|improve this answer
add comment

GNU Parallel is more or less standard these days:

cat urls | parallel -j30 --eta curl ... > out

Added benefit: multiple curls will be run in parallel.

Watch the intro videos to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.