0

I have encountered the following problem during a first hard shutdown of the department cluster I am in charge of. The system is running SLURM 17.11 and using MariaDB/SQL to store accounting data.

To perform a memory upgrade I had to shut down the control & database server of the cluster, which is using SLURM as scheduler. After a restart the control daemon refused to start since apparently the state save files in /var/spool did not have the correct permissions anymore. So I made a dedicated folder /var/spool/slurm_state for the slurm state files and changed ownership to slurm:slurm. After modifying sulrm.conf to set the proper StateSaveLocation the control daemon started and I could submit test jobs.

I did not, however, copy the old state files to the new location. Thus the new jobs started again at JobID 1. After realising that I quickly terminated slurmctld and changed StateSaveLocation back to /var/spool (with appropriate group and permission changes).

Now the the one test job that has been running when the control daemon has been shut down is stuck in the database with the state set to RUNNING systemverwalter 2 240 9-21:40:55 100.0 RUNNING allgather_latency_240_mpich just accumulating run time for the account.

I have tried to terminate the job via scancel as the user as well as as root, to no avail. Neither has the attempt to put the job on hold using scontrol resulted in the desired outcome.

My question is thus: What should I do to terminate this job? Do I have to modify the database entry by hand, or is there a simpler solution?

Improve this question

2 Answers 2

Reset to default
1

The following command will list the runaway jobs and offer to remove them:

sacctmgr show runawayjobs

Improve this answer
0

Ok. I have found a quite trivial solution to this problem, though I do not think it is one which will always work.

To eliminate such a zombie process proceed as follows:

  1. Start the SLURM account manager via sacctmgr as a user with an Operator account (or root).
  2. Search for runaway jobs by issuing list runawayjobs in the sacctmgr prompt.
  3. If the system recognises one or multiple jobs with no end date, i.e., orphaned (runaway) jobs it will query whether you want to fix it. Confirm with Y.

These steps have resolved my issue after having the runaway job in the sacct reports for 9 days.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.