qsub -hold_jid -afterok loop not starting

Question

I am trying to run a series of jobs submitted to the cluster, one after the other, by suplpying the following .sh script:

Annotation_Loop.sh :

#!/bin/bash

job=`qsub run_IntersectBed_1.sh 0`
for i in {1..3}
do
    job_next=`qsub -hold_jid $job run_IntersectBed_1.sh $i`
    job=$job_next
done

The first job (before entering the loop) is executed, but the next ones are never started.

I think the script is well written. I change the mode to an executable and run it as

nohup ./Annotation_Loop.sh (I think this is necessary?) ... but the rest is never done.

I tried -W before, using -W depend=afterok:$job in place of -hold_jid $job

#!/bin/bash

    job=`qsub run_IntersectBed_1.sh 0`
    for i in {1..3}
    do
        job_next=`qsub -W depend=afterok:$job run_IntersectBed_1.sh $i`
        job=$job_next
    done

But it returned unknown option -W.

What could be going wrong? :(

Answer 1

The ouput of qsub has the jobid inside a message text. In my case:

$ qsub hello.sh
Your job 8845476 ("hello.sh") has been submitted

You should extract the jobid from this message. E.g:

$ jobid=$(qsub hello.sh | cut -d' ' -f3)
$ echo $jobid
8845481

Maybe your qsub version, has a different message, try it out separatly and then put a cut to get the jobid as a string.

Answer 2

For anyone who stumbles across this years later:

I would use an array job for that to avoid writing a shell script. You need something like run_IntersectBed_1.sh $SGE_TASK_ID and need to submit with qsub -t 1-3 -tc 1 Annotation_Loop.sh

This creates an array of 3 jobs, numbered 1, 2 and 3, which run one at a time (-tc 1). You no longer need to worry about hold_jid.

SGE_TASK_ID is where the integer id of the current job is held. This is what you pass to your code (equivalent of i in your loop).