0

I am working to configure slurm on an AWS cluster created with CloudFormation. At boot time some of the nodes get set to a "drain" state, with the stated reason being "Low socketcorethread count". However, all of the nodes in the cluster are the same EC2 instance type, have the same slurm.conf file installed, and come from the same AMI. ONLY SOME of the nodes end up in the "drain" state, and I see no pattern to it. After I manually update the state of the "drain" nodes to "resume", they become available to process tasks. Has anyone seen this before?

ubuntu@ip-10-0-0-10:~$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
compute*     up   infinite      2  drain ip-10-0-0-10,node1
compute*     up   infinite      1   idle node2

ubuntu@ip-10-0-0-10:~$ scontrol show node node1
NodeName=node1 Arch=x86_64 CoresPerSocket=1
   CPUAlloc=0 CPUErr=0 CPUTot=4 CPULoad=0.40 Features=(null)
   Gres=(null)
   NodeAddr=node1 NodeHostName=node1 Version=15.08
   OS=Linux RealMemory=1 AllocMem=0 FreeMem=15062 Sockets=4 Boards=1
   State=IDLE+DRAIN ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A
   BootTime=2019-08-19T21:46:44 SlurmdStartTime=2019-08-19T21:48:36
   CapWatts=n/a
   CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
   Reason=Low socket*core*thread count, Low CPUs [slurm@2019-08-19T21:47:06]

ubuntu@ip-10-0-0-10:~$ sudo scontrol update nodename=node1,ip-10-0-0-10 state=RESUME

ubuntu@ip-10-0-0-10:~$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
compute*     up   infinite      3   idle ip-10-0-0-10,node[1-2]

1 Answer 1

0

Try setting "ReturnToService=2" in slurm.conf if not already set. Then restart slurm services. Here, the value of 2 means

A DOWN node will become available for use upon registration with a valid configuration. The node could have been set DOWN for any reason. A node registers with a valid configuration if its memory, GRES, CPU count, etc. are equal to or greater than the values configured in slurm.conf. (Disabled on Cray ALPS systems.)

The default value is 0.

Source https://slurm.schedmd.com/slurm.conf.html

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .