Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sequential: for i in {1..1000}; do do_something $i; done - too slow

Parallel: for i in {1..1000}; do do_something $i& done - too much load

How to run commands in parallel, but not more than, for example, 20 instances per moment?

Now usually using hack like for i in {1..1000}; do do_something $i& sleep 5; done, but this is not a good solution.

Update 2: Converted the accepted answer into a script: http://vi-server.org/vi/parallel

#!/bin/bash

NUM=$1; shift

if [ -z "$NUM" ]; then
    echo "Usage: parallel <number_of_tasks> command"
    echo "    Sets environment variable i from 1 to number_of_tasks"
    echo "    Defaults to 20 processes at a time, use like \"MAKEOPTS='-j5' parallel ...\" to override."
    echo "Example: parallel 100 'echo \$i; sleep \`echo \$RANDOM/6553 | bc -l\`'"
    exit 1
fi

export CMD="$@";

true ${MAKEOPTS:="-j20"}

cat << EOF | make -f - -s $MAKEOPTS
PHONY=jobs
jobs=\$(shell echo {1..$NUM})

all: \${jobs}

\${jobs}:
        i=\$@ sh -c "\$\$CMD"
EOF

Note that you must replace 8 spaces with 2 tabs before "i=" to make it work.

share|improve this question
add comment

6 Answers

up vote 5 down vote accepted

GNU Parallel is made for this.

seq 1 1000 | parallel -j20 do_something

It can even run jobs on remote computers. Here's an example for re-encoding an MP3 to OGG using server2 and local computer running 1 job per CPU core:

parallel --trc {.}.ogg -j+0 -S server2,: \
     'mpg321 -w - {} | oggenc -q0 - -o {.}.ogg' ::: *.mp3

Watch an intro video to GNU Parallel here:

http://www.youtube.com/watch?v=OpaiGYxkSuQ

share|improve this answer
    
Haven't know about "moreutils" and that there's already a tool for the job. Looking and comparing. –  Vi. Jul 27 '10 at 16:26
1  
The parallel in moreutils is not GNU Parallel and is quite limited in its options. The command above will not run with the parallel from moreutils. –  Ole Tange Sep 28 '10 at 22:49
add comment

Not a bash solution, but you should use a Makefile, possibly with -l to not exceed some maximum load.

NJOBS=1000

.PHONY = jobs
jobs = $(shell echo {1..$(NJOBS)})

all: $(jobs)

$(jobs):
    do_something $@

Then to start 20 jobs at a time do

$ make -j20

or to start as many jobs as possible without exceeding a load of 5

$ make -j -l5
share|improve this answer
    
Looks like the non-hacky solution for now. –  Vi. Jun 17 '10 at 14:01
1  
echo -e 'PHONY=jobs\njobs=$(shell echo {1..100000})\n\nall: ${jobs}\n\n${jobs}:\n\t\techo $@; sleep `echo $$RANDOM/6553 | bc -l`' | make -f - -j20 Now it looks more hacky again. –  Vi. Jun 17 '10 at 14:11
    
@vi: oh my .... –  Benjamin Bannier Jun 17 '10 at 14:12
    
Converted your solution to a script. Now it can be used with ease. –  Vi. Jun 17 '10 at 15:09
add comment

One simple idea:

Check for i modulo 20 and execute the wait shell-command before do_something.

share|improve this answer
    
It will either wait for all current tasks to complete (creating sags in the number of tasks plot) or wait for one specific task that can stall for longer time (again creating sags in this case) –  Vi. Jun 17 '10 at 17:19
    
@Vi: Shell wait is for all background tasks that belong to this shell. –  harrymc Jun 17 '10 at 18:57
add comment

You could use ps to count how many processes you have running, and whenever this drops below a certain threshold you start another process.

Pseudo code:

i = 1
MAX_PROCESSES=20
NUM_TASKS=1000
do
  get num_processes using ps
  if num_processes < MAX_PROCESSES
    start process $i
    $i = $i + 1
  endif
  sleep 1 # add this to prevent thrashing with ps
until $i > NUM_TASKS
share|improve this answer
add comment
for i in {1..1000}; do 
     (echo $i ; sleep `expr $RANDOM % 5` ) &
     while [ `jobs | wc -l` -ge 20 ] ; do 
         sleep 1 
     done
done
share|improve this answer
    
May be while [ `jobs | wc -l` -ge 20]; do? –  Vi. Jun 17 '10 at 13:38
    
sure, but in my sample, I'd then have to compute njobs twice, and performance is quite important in shell scripts that run sleep tasks ;) –  msw Jun 17 '10 at 13:42
    
I mean your version doesn't work as expected . I change sleep 1 to sleep 0.1 and it start to average njobs to 40-50 instead of 20. If there are more than 20 jobs we need to wait for any job gets finished, not just wait 1 second. –  Vi. Jun 17 '10 at 13:57
add comment

posting the script in the question with formatting:

#!/bin/bash

NUM=$1; shift

if [ -z "$NUM" ]; then
    echo "Usage: parallel <number_of_tasks> command"
    echo "    Sets environment variable i from 1 to number_of_tasks"
    echo "    Defaults to 20 processes at a time, use like \"MAKEOPTS='-j5' parallel ...\" to override."
    echo "Example: parallel 100 'echo \$i; sleep \`echo \$RANDOM/6553 | bc -l\`'"
    exit 1
fi

export CMD="$@";

true ${MAKEOPTS:="-j20"}

cat << EOF | make -f - -s $MAKEOPTS
PHONY=jobs
jobs=\$(shell echo {1..$NUM})

all: \${jobs}

\${jobs}:
        i=\$@ sh -c "\$\$CMD"
EOF

Note that you must replace 8 spaces with 2 tabs before "i=".

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.