3

I want to run a multi-parallel task in one bash file like the example code below,

for i in 1 2 3 4 5 6 7 8; do
        setsid python /tmp/t.py ${i} 1>>/tmp/1.log 2>&1 &
done
wait  # first wait

echo "next wait"
for i in 9 10 11 12 13 14 15 16; do
        setsid python /tmp/t.py ${i} 1>>/tmp/1.log 2>&1 &
done
wait  # second wait

As you can see it, is wait possible to do this? I want to run the first 8 tasks and then wait all the tasks to finish, then spawn the next 8 tasks because the RAM is limited, I cannot run all the 16 tasks in one round.

5
  • Is it possible? The answer is "yes, but...". Is this short answer helpful? I doubt it. What do your tests indicate? Does your code wait? Do you run the script or source it? Have you seen this answer? Does t.py fork? I guess your problem is the code doesn't wait when you need it to, right? (If it did, you wouldn't ask if it's possible). Simple examples do wait, your code may not. Please edit and provide MCVE. Have you considered/tried GNU parallel with -j? Jan 20, 2020 at 7:27
  • Just to clarify: The wait command will wait for the last process to finish. It does not care about the others. So, for the first loop, if #8 finishes before numbers 1-7, the script will continue to the second loop. Which is not what the OP wants. I like the answer @meuh gave by using -w on setsid. Simple & elegant.
    – Scottie H
    Jan 5, 2021 at 0:20
  • @ScottieH After setting -w with setsid, should I also wait?
    – GoingMyWay
    Jan 5, 2021 at 4:24
  • 1
    It wont hurt anything to do it. It will ensure that all the background jobs have stopped before proceeding.
    – Scottie H
    Jan 6, 2021 at 1:40
  • 1
    @ScottieH Thank you.
    – GoingMyWay
    Jan 6, 2021 at 2:34

3 Answers 3

2

Use option -w or --wait to setsid so that the setsid command waits until the python process ends before ending itself. This means the shell wait command now has child processes to wait for.

2

Something like this:

set -a pids
for i in {1..8}
do
  python /tmp/t.py ${i} 1>>/tmp/1.log 2>&1 &
  pids[${#pids[*]}]=$!  # Remember pid of child
done

# Wait for each child. This loops exits when all children are finished
for pid in ${pids[*]} 
do
  wait $pid
  printf "Sensed end of %d\n" $pid
done

### Continue processing here
1

With GNU Parallel it looks like this:

parallel -j8 setsid python /tmp/t.py {} ::: {1..16} > log

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .