I am using cURL to download many files and concatenate them to STDOUT. About 100,000 small files. I would like to see progress against the 100,000. Is this possible with curl or by using curl into xargs?

Interested in only standard command line solutions.

  • it is possible to display something like: "downloading [200/100000]". but as you said, you want to print the file contents to stdout. if you got status/progress information, they should be printed to stdout as well. it will mess your stdout up. what do you exactly want? – Kent Dec 14 '12 at 14:37

There are a few different things you could do here: but without knowing exactly your curl methodology I can just offer a few suggestions.

Make an iterative counting for loop:

for file {1..100000}; do echo "downloading: $file" >&2 ; curl [whatever] ; done This will redirect the "downloading: $file"to STDERR so if you're using a redirect on STDOUT it won't mess it it up but you can still see it on the screen

for file {1..100000}; do echo "downloading: $file" >> progress.out ; curl [whatever] ; done This writes progress to a file so it won't show up on the screen at all, then you can just tail -f progress.out

If your curl isn't iterative:

terminal 1: curl [whatever] | tee progress.out

terminal 2: watch -n5 "grep -c '[unique file delimiter]' progress.out"

tee writes a copy of STDOUT to a file, and you grep count for something that only appears once per HTML doc... maybe a <HEAD> or <HTML> tag or something else. Watch will just run the grep it every 5 seconds to let you know how many you have completed.


GNU Parallel is more or less standard these days:

cat urls | parallel -j30 --eta curl ... > out

Added benefit: multiple curls will be run in parallel.

Watch the intro videos to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.