2

I want to run a multi-parallel task in one bash file like the example code below,

for i in 1 2 3 4 5 6 7 8; do
        setsid python /tmp/t.py ${i} 1>>/tmp/1.log 2>&1 &
done
wait  # first wait

echo "next wait"
for i in 9 10 11 12 13 14 15 16; do
        setsid python /tmp/t.py ${i} 1>>/tmp/1.log 2>&1 &
done
wait  # second wait

As you can see it, is wait possible to do this? I want to run the first 8 tasks and then wait all the tasks to finish, then spawn the next 8 tasks because the RAM is limited, I cannot run all the 16 tasks in one round.

  • Is it possible? The answer is "yes, but...". Is this short answer helpful? I doubt it. What do your tests indicate? Does your code wait? Do you run the script or source it? Have you seen this answer? Does t.py fork? I guess your problem is the code doesn't wait when you need it to, right? (If it did, you wouldn't ask if it's possible). Simple examples do wait, your code may not. Please edit and provide MCVE. Have you considered/tried GNU parallel with -j? – Kamil Maciorowski Jan 20 at 7:27
2

Use option -w or --wait to setsid so that the setsid command waits until the python process ends before ending itself. This means the shell wait command now has child processes to wait for.

2

Something like this:

set -a pids
for i in {1..8}
do
  python /tmp/t.py ${i} 1>>/tmp/1.log 2>&1 &
  pids[${#pids[*]}]=$!  # Remember pid of child
done

# Wait for each child. This loops exits when all children are finished
for pid in ${pids[*]} 
do
  wait $pid
  printf "Sensed end of %d\n" $pid
done

### Continue processing here

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.