Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

So I just installed Torque on a small cluster we're setting up, and everything seemed to have gone quite smoothly. I have pbs_mon running on the compute nodes and pbs_server running on the head node, and pbsnodes -a from the head node shows all the nodes in the "free" state.

However, when I submit a trivial job - even the echo 'sleep 30' job - using the qsub command, the things just sit in the queue and never execute. I have four compute nodes and am requesting 4 nodes (unspecified memory/time/ppn).

I thought that maybe I needed to install Maui to get the job scheduling working well, but in hindsight, Torque should be able to schedule and execute jobs by itself, shouldn't it? In any event, I installed Maui (successfully?) on the head node (doing a showq shows my submitted jobs as BLOCKED)... I assume it is working on the head node, since otherwise I wouldn't be able to see my qsub-submitted jobs, right? Do I need to install Maui on each of the compute nodes in order for it to work? And do I need to start maui on those, too?

I really just want to get this working for simple stuff... we'll see about the rest later. I am quite new to this and will appreciate any help that is offered. I have tried doing what is mentioned in the installation guides, and like I said, everything seemed to work, but now it's not behaving like I expected.

Here's the output of running checkjob on a job in the queue...

  job is deferred.  Reason:  RMFailure  (cannot start job - RM failure, rc: 15043, 
  msg: 'Execution server rejected request MSG=cannot send job to mom, state=PRERUN')
  Holds:    Defer  (hold reason:  RMFailure)
  PE:  1.00  StartPriority:  1
  cannot select job 8 for partition DEFAULT (job hold active)

I think that pbs_mom is running on all the compute nodes:

  [root@host1 ~]# service pbs_mom status
  pbs_mom (pid 27824) is running...

  [root@host2 ~]# service pbs_mom status
  pbs_mom (pid 20164) is running...

  [root@host3 ~]# service pbs_mom status
  pbs_mom (pid 23981) is running...

  [root@host4 ~]# service pbs_mom status
  pbs_mom (pid 23996) is running...

I also believe pbs_server is running on the head node... could my problem possibly be that I need to change iptables to make sure that the required ports aren't being blocked? Again, all of the nodes are showing up as free after a pbsnodes -a...

Just in case this contains useful information...

  [root@headnode maui-3.3.1]# pbsnodes -a
  host1
       state = free
       np = 4
       properties = dual470
       ntype = cluster
       status = 
  rectime=1317050602,varattr=,jobs=,state=free,netload=164038242,gres=,
  loadave=0.00,ncpus=4,physmem=8060460kb,availmem=17684340kb,totmem=18349604kb,
  idletime=241170,nusers=2,nsessions=9,sessions=3444 3328 3564 3574 3604 3620 
  3630 3684 6843,uname=Linux hawc1 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 
  19:49:27 BST 2011 x86_64,opsys=linux
       mom_service_port = 15002
       mom_manager_port = 15003
       gpus = 2

  host2
       state = free
       np = 4
       properties = dual480
       ntype = cluster
       status = 
  rectime=1317050602,varattr=,jobs=,state=free,netload=106187454,gres=,
  loadave=0.00,ncpus=8,physmem=8058888kb,availmem=13819772kb,
  totmem=14219264kb,idletime=241201,nusers=1,nsessions=6,sessions=3446 
  3450 3452 3477 3479 3504,uname=Linux hawc2 2.6.32-71.29.1.el6.x86_64 
  #1 SMP Mon Jun 27 19:49:27 BST 2011 x86_64,opsys=linux
       mom_service_port = 15002
       mom_manager_port = 15003
       gpus = 2

  host3
       state = free
       np = 4
       properties = dual480
       ntype = cluster
       status = 
  rectime=1317050602,varattr=,jobs=,state=free,netload=148456196,gres=,
  loadave=0.00,ncpus=8,physmem=8058888kb,availmem=13722692kb,totmem=14219264kb,
  idletime=241161,nusers=2,nsessions=8,sessions=3482 3362 3584 3594 3624 3640 
  3650 3706,uname=Linux hawc3 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 
  19:49:27 BST 2011 x86_64,opsys=linux
       mom_service_port = 15002
       mom_manager_port = 15003
       gpus = 2

  host4
       state = free
       np = 4
       properties = 480C2050
       ntype = cluster
       status = 
  rectime=1317050595,varattr=,jobs=,state=free,netload=154812523,gres=,
  loadave=0.01,ncpus=8,physmem=8058888kb,availmem=13715256kb,totmem=14219264kb,
  idletime=241142,nusers=2,nsessions=9,sessions=3504 3386 3590 3600 3630 3650 
  3655 3712 3829,uname=Linux hawc4 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 
  19:49:27 BST 2011 x86_64,opsys=linux
       mom_service_port = 15002
       mom_manager_port = 15003
       gpus = 2

SOLVED:

So it turns out that iptables was messing with me. I disabled iptables on the compute nodes and added correct entries in the head node to, and now it seems to work OK... I can qsub echo "sleep 30" and see it work. I'm having a new problem, but I will make a new question now... hopefully this one will be simpler!

share|improve this question
    
Check that pbs_mom is running on all the nodes. Looks like it's the issue... –  aland Sep 26 '11 at 16:24
    
@aland: Please check my edit... –  Patrick87 Sep 26 '11 at 16:34

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.