Every once in a while, I run into this frustrating scenario where jobs that I have submitted don't run, even though there are resources available. This time, I think the problem might be related to one of the nodes crashing while jobs were sitting in the queue. I'm not sure how it's supposed to recover, but jobs will get themselves started after a long delay. I'm wondering if there's a command I can issue to speed things along, e.g. reset the job somehow without killing/resubmitting. Unfortunately, I won't be able to test any ideas recommended until the problem comes up again, but I'll respond whenever I can.
> checkjob 24458 checking job 24458 State: Idle EState: Deferred Creds: user:amosa group:amosa class:default qos:DEFAULT WallTime: 00:00:00 of 99:23:59:59 SubmitTime: Tue Aug 7 11:52:03 (Time Queued Total: 1:04:03 Eligible: 00:00:01) StartDate: -00:45:16 Tue Aug 7 12:10:50 Total Tasks: 1 Req TaskCount: 1 Partition: ALL Network: [NONE] Memory >= 2000M Disk >= 0 Swap >= 0 Opsys: [NONE] Arch: [NONE] Features: [NONE] Dedicated Resources Per Task: PROCS: 1 MEM: 2000M IWD: [NONE] Executable: [NONE] Bypass: 0 StartCount: 1 PartitionMask: [ALL] Flags: RESTARTABLE job is deferred. Reason: RMFailure (cannot start job - RM failure, rc: 15043, msg: 'Execution server rejected request MSG=cannot send job to mom, state=PRERUN') Holds: Defer (hold reason: RMFailure) PE: 1.00 StartPriority: 0 cannot select job 24458 for partition DEFAULT (job hold active)