I have a problem using a GPU in ubuntu server. Nvidia-smi prints
"ERR!!" on FAN and Usage, GPU not being reset, processes are not killed
- I ran three programs using one GPU and found those programs stopped.
- But, when I checked GPU by "gpustat" and "Nvidia-smi", my processes were still using GPU
- and I could see "ERR!" on FAN and USAGE after running Nvidia-smi.
When I googled the problem, I found that I could reset GPU or kill the processes that use that GPU.
- so I tried to kill processes that use that GPU with "sudo kill -9 pid" But It didn't work!! I googled how to do when "sudo kill" does not work.
and I thought they are Zombie processes. So, I found Zombie processes and removed it.
After that, there are not any more Zombie processes when I searched.
But those three processes were not still removed.
- I tried to reset GPU with "Nvidia-smi --gpu-reset -i 0"
- BUT it prints "GPU Reset couldn't run because GPU 00000000:01:00.0 is the primary GPU. " since other people are using other GPUs in that server, I just want to reset index-0 GPU
and I found the problem on Google, and the answer was to kill processes running on that GPU. First problem again!
(I am not used to using GPU. is it a not a good way to run multiple codes on one same GPU? )