2

I have a problem using a GPU in ubuntu server. Nvidia-smi prints

"ERR!!" on FAN and Usage, GPU not being reset, processes are not killed

  • I ran three programs using one GPU and found those programs stopped.
  • But, when I checked GPU by "gpustat" and "Nvidia-smi", my processes were still using GPU
  • and I could see "ERR!" on FAN and USAGE after running Nvidia-smi.

nvidia-smi prints ERR! on FAN and Usage

gpustat still prints three processes are running on gpu

When I googled the problem, I found that I could reset GPU or kill the processes that use that GPU.

  • so I tried to kill processes that use that GPU with "sudo kill -9 pid" But It didn't work!! I googled how to do when "sudo kill" does not work.

and I thought they are Zombie processes. So, I found Zombie processes and removed it.

After that, there are not any more Zombie processes when I searched.

But those three processes were not still removed.

  • I tried to reset GPU with "Nvidia-smi --gpu-reset -i 0"
  • BUT it prints "GPU Reset couldn't run because GPU 00000000:01:00.0 is the primary GPU. " since other people are using other GPUs in that server, I just want to reset index-0 GPU

and I found the problem on Google, and the answer was to kill processes running on that GPU. First problem again!

(I am not used to using GPU. is it a not a good way to run multiple codes on one same GPU? )

1
  • did you find a solution?
    – oezguensi
    May 8 '20 at 18:55
1

I meet the GPU problems, one of which is as same as yours. It seems that Google has no enough resources for GPU Fan ERR! I has spend my time to figure out the following solutions. Most of time they work out, but depends on the GPU status including CUDA Driver,Hardware FAN and PCle Power Cable(probably).

1. Reboot your system

The easiest way is to reboot your system for returning to the normal status. For instance, I reboot my Ubuntu 18.04 as follows.

$ sudo reboot

2.The Solution in the state of the idle GPU

If your GPU Graphics Memory has not occupied with any Deep Learning application, for instance: no the usage amount such as 2100 MiB / 7981 MiB, you can use the following composite commands to solve the GPU Fan Error.

$ sudo rmmod nvidia_uvm
$ sudo modprobe nvidia_uvm
$ sudo reboot

The commands will work out and GPU will go back to the normal status most of the times.

3.The solution in the state of zombie GPU processes

If your Deep Leaning application is completed(or ended) and the GPU was still occupied with zombie processes(or other reasons), the above-mentioned commands could not work out. In this condition, you can solve the issues by inputting the following commands.

Reboot the system

$ sudo reboot

Continue executing the following composite commands.

$ sudo rmmod nvidia_uvm

rmmod: ERROR: Module nvidia_uvm is in use

(Do not care about the above-shown ERROR and continue executing the following command)

$ sudo modprobe nvidia_uvm

$ sudo reboot

My test scenario for the above commands is described as follows.

A. After the single reboot command such as "sudo reboot" is given, the GPU does not work actually. But it is necessary to clear your system.

B. Following A, the above-mentioned composite commands are given, then the GPU works out.

4. Set pesistenced for Nvidia GPU

1). Check the nvidia-persistenced status

$ sudo systemctl status nvidia-persistenced

2). Enable nvidia-persistenced

$ sudo systemctl enable nvidia-persistenced

3). reboot

$ sudo reboot

Cheers.

1
  • Thanks for your kind answer to my question!
    – ylee
    May 13 at 7:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.