PBS batch jobs - the qalter command

Question

I've got a giant computation running on a Scientific Linux cluster. At present I have over 600 jobs parked in the queue, waiting for processor time, while a few are running.

I'm trying to use the qalter command on some of the idle but scheduled jobs. I'd like to schedule them for a later time, so that other users can jump part of the queue, sort of as an act of politeness. Is this doable?

For example, JOBNAME 292399 is currently idle, scheduled to be run whenever a spot in the queue opens up.

But if I run qalter -a 10051000 292398 followed by qrerun 292398 I get qrerun: Request invalid for state of job 292398.euler.

From the qalter documentation, I thought 10051000 refers to tomorrow (oct 5th, 10am) but perhaps I'm misunderstanding something?

If I'm going about this the wrong way, please let me know. The main thing I'm looking for is a command that's easily scriptable, so that I can modify when my queued tasks get run. qalter seems good for those purposes if I can get it working. I'd rather avoid running qdel and re qsubbing the computations, as there's a bookkeeping issue on which tasks to restart (vs which ones not to). I want to avoid that kind of bookkeeping.

From googling around I notice some qalter commands have rather different date formats, but the above appears to be correct, as far as I can tell from the man docs.

Any help would be appreciated.

Answer 1

Use qhold to place a hold on a job. When you are ready to run it, use qrls to release the job. You can easily create a cron script for this purpose.

See the PBS Professional User's Guide and the TORQUE Administrator's Guide (TORQUE is mostly compatible with PBS) for more information.

Edit: You can use qalter -a as well, but do not use qrerun: the job is not running and it is not eligible for execution until after the date and time specified in the qalter -a command, so qrerun returns an error.