Every once in a while, I run into this frustrating scenario where jobs that I have submitted don't run, even though there are resources available. This time, I think the problem might be related to one of the nodes crashing while jobs were sitting in the queue. I'm not sure how it's supposed to recover, but jobs will get themselves started after a long delay. I'm wondering if there's a command I can issue to speed things along, e.g. reset the job somehow without killing/resubmitting. Unfortunately, I won't be able to test any ideas recommended until the problem comes up again, but I'll respond whenever I can.
> checkjob 24458
checking job 24458
State: Idle EState: Deferred
Creds: user:amosa group:amosa class:default qos:DEFAULT
WallTime: 00:00:00 of 99:23:59:59
SubmitTime: Tue Aug 7 11:52:03
(Time Queued Total: 1:04:03 Eligible: 00:00:01)
StartDate: -00:45:16 Tue Aug 7 12:10:50
Total Tasks: 1
Req[0] TaskCount: 1 Partition: ALL
Network: [NONE] Memory >= 2000M Disk >= 0 Swap >= 0
Opsys: [NONE] Arch: [NONE] Features: [NONE]
Dedicated Resources Per Task: PROCS: 1 MEM: 2000M
IWD: [NONE] Executable: [NONE]
Bypass: 0 StartCount: 1
PartitionMask: [ALL]
Flags: RESTARTABLE
job is deferred. Reason: RMFailure (cannot start job - RM failure, rc: 15043, msg: 'Execution server rejected request MSG=cannot send job to mom, state=PRERUN')
Holds: Defer (hold reason: RMFailure)
PE: 1.00 StartPriority: 0
cannot select job 24458 for partition DEFAULT (job hold active)
