2

I am using curl to get a URL then writing to a file, like this:

urls=( 
  'https://www.example1.com'
  'https://www.example2.com'
 )

for i in ${urls[@]}; do
   curl $i &
done
echo 'stuff'

I have deliberately simplified the code, so the exact problem can be tackled.

Output:

stuff
$curlContents1
$curlContents2

I know why this happens, it’s running asynchronously.

What I want to know

  • I want to run this async cmd with the output the same as it would be if I had ran it sync.
  • This is because running it async gives a nice speed boost

Desired output:

$curlContents1
$curlContents2
stuff

more info

  • my actual problem is a bit different…

What I’m doing is downloading videos then taking the last part of the URL and using it as the file name, how can I use parallel in this example?

The write happens before the download as the downloads are the most time consuming part

arr=(
  'https://www.example1.com/stccdtu.mp4’
  'https://www.example2.com/dyubdf.mp4’
 )

for i in ${arr[@]}; do 
    curl $i > `echo $i | sed s'#https://www.example[0-9].com/##'g` &
done

output:

ll

0 stccdtu.mp4 
0 dyubdf.mp4

2 Answers 2

2

With GNU parallel. Basic example:

parallel -j 40 --group --keep-order curl ::: "${urls[@]}"
echo 'stuff'

-j 40 means we assign 40 jobslots, i.e. we allow up to 40 parallel jobs (adjust it to your needs and abilities). If you supply more URLs then the 41st one will be processed after some slot gets available. All URLs will be processed but at any moment there will be at most 40 jobs running in parallel.

Other options used:

--group
Group output. Output from each job is grouped together and is only printed when the command is finished. Stdout (standard output) first followed by stderr (standard error). […]

(source)

which is the default, so usually you don't have to use it explicitly.

--keep-order
-k
Keep sequence of output same as the order of input. Normally the output of a job will be printed as soon as the job completes. […] -k only affects the order in which the output is printed - not the order in which jobs are run.

(source)

Notes:

  • In my example parallel is not in the background and is run synchronously (so echo runs after it); still curls run in parallel, asynchronously.

  • In Debian GNU parallel is in a package named parallel. Basic variant of the tool (from moreutils, at least in Debian) is less powerful.

  • parallel is an external command. If the array is large enough then with parallel … ::: "${urls[@]}" you will hit argument list too long. Use this instead:

    printf '%s\n' "${urls[@]}" | parallel …
    

    It will work because in Bash printf is a builtin and therefore everything before | is handled internally by Bash.

  • ${urls[@]} is properly double-quoted (in your code ${urls[@]} and $i are unquoted, this is wrong).


GNU parallel can call exported Bash functions. This allows us to solve what you called the actual problem:

getvideo() {
curl "$1" > "${1##*/}"
}
export -f getvideo

urls=(
  'https://www.example1.com/stccdtu.mp4'
  'https://www.example2.com/dyubdf.mp4'
 )

parallel -j 40 --group --keep-order getvideo ::: "${urls[@]}"
echo 'stuff'

If you don't know what ${1##*/} does, read this another answer of mine.

5
  • For -jcan you do ${#arr[@]} or `echo ${#arr[@]}`?
    – Nickotine
    Jun 26, 2021 at 22:51
  • Yes seems to work for me
    – Nickotine
    Jun 26, 2021 at 23:04
  • @Nickotine You can. But read the manual: -j 0 means as many as possible. Besides, `echo ${#arr[@]}` (or $(echo ${#arr[@]})) is an over-complicated way to get ${#arr[@]}. In some sense it's similar to echo $(stuff) and similarly bad practice. Jun 26, 2021 at 23:05
  • I only suggested the echo version thinking it might not work… what’s the good practice way in your example?
    – Nickotine
    Jun 26, 2021 at 23:53
  • @Nickotine In what example? In my answer here: -j 0, if this is what you want. Jun 27, 2021 at 8:07
0

Bash shell has the wait command that will pause the script until background jobs are finished.

Waits for each process identified by an ID, which may be a process ID or a job specification, and reports its termination status. If ID is not given, waits for all currently active child processes, and the return status is zero. If ID is a a job specification, waits for all processes in that job's pipeline.

for i in ${urls[@]}; do
   curl $i &
done
wait
echo 'stuff'
4
  • Two things: (1) This prints stuff last, all right; but the output from curls may get interleaved. So the output may be far from "the same as it would be if I had ran it sync". In the "actual problem" the OP tries to write to separate files and then the issue is no more (at least for stdout), but your code does not do this. (2) You replicated the OP's flaws of not quoting right. It may not matter for the example URLs in question but in general not fixing this common bug makes the code mediocre. Jun 26, 2021 at 22:54
  • Yeah for the actual problem this won’t work but works for the example I gave, parallel has sped up my code tremendously… @kamilMaciorowski you only need to worry about quoting when there is white space and even then, if you do IFS=$'\n' (I usually do this) you don’t need to worry about quoting even if there is white space, all those quotes make the code ugly
    – Nickotine
    Jun 26, 2021 at 23:09
  • @Nickotine What about globbing? What about multi-line variables? Get used to quoting and don't bother yourself thinking if you strictly need to quote or can get away without quotes. If your variables change, the lack of quotes may bite you; unless you notice and debug the entire script then. Why not write it right (or at least try to) from the start? Answers are also for users with similar problems and their data may contain spaces, they may not know your tricks. Therefore answers should be robust. I'm not going to argue more. It seems you prefer non-ugly code and I prefer non-buggy code. Jun 26, 2021 at 23:38
  • Hahah well @kamilmaciorowski when I write small scripts for myself I do selective quote, if I was at work or working with others I’d quote :)
    – Nickotine
    Jun 26, 2021 at 23:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .