Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am using cURL to download many files and concatenate them to STDOUT. About 100,000 small files. I would like to see progress against the 100,000. Is this possible with curl or by using curl into xargs?

Interested in only standard command line solutions.

share|improve this question

migrated from stackoverflow.com Dec 16 '12 at 19:13

This question came from our site for professional and enthusiast programmers.

    
it is possible to display something like: "downloading [200/100000]". but as you said, you want to print the file contents to stdout. if you got status/progress information, they should be printed to stdout as well. it will mess your stdout up. what do you exactly want? – Kent Dec 14 '12 at 14:37

There are a few different things you could do here: but without knowing exactly your curl methodology I can just offer a few suggestions.

Make an iterative counting for loop:

for file {1..100000}; do echo "downloading: $file" >&2 ; curl [whatever] ; done This will redirect the "downloading: $file"to STDERR so if you're using a redirect on STDOUT it won't mess it it up but you can still see it on the screen

for file {1..100000}; do echo "downloading: $file" >> progress.out ; curl [whatever] ; done This writes progress to a file so it won't show up on the screen at all, then you can just tail -f progress.out

If your curl isn't iterative:

terminal 1: curl [whatever] | tee progress.out

terminal 2: watch -n5 "grep -c '[unique file delimiter]' progress.out"

tee writes a copy of STDOUT to a file, and you grep count for something that only appears once per HTML doc... maybe a <HEAD> or <HTML> tag or something else. Watch will just run the grep it every 5 seconds to let you know how many you have completed.

share|improve this answer

GNU Parallel is more or less standard these days:

cat urls | parallel -j30 --eta curl ... > out

Added benefit: multiple curls will be run in parallel.

Watch the intro videos to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .