0

I am trying to use my Nvidia gpu for scientific purposes. However after a while I started a process for this gpu, my computer shutdowns. It cannot be opened from case button unless PSU power button made off and stayed at this state at least for 30 seconds then became on. I am trying to spot the reason of this situation.

My system overview is:

  • Linux (fedora)
  • Motherboard: Gigabyte GA-970A-UD3
  • Nvidia Gtx 1070 (for gpu tasks)
  • Radeon HD 7850 (for standard purposes)
  • 700W PSU

Here is a list of things that I have tried so far:

  • I did a CPU load test and it does not shut down
  • I removed AMD GPU, it still shuts down
  • It does not shutdown if I give a light calculation to Nvidia GPU for a short period of time
  • If I give heavy calculation it shutdowns very shortly after start
  • I checked Nvidia gpu temp and even it floats max. around 55 C, it shuts down
  • I manually set Nvidia GPU fan speed to %80. It seemed to help because given task run a little bit longer than usual but it did not prevent shutdown
  • I removed extra HDD or fans to make PSU less tired, it did not help at all
  • I added extra PSU using add2psu however still shutdowns
  • Even CPU temp is around 45, it shutdowns if Nvidia GPU starts doing calculations
  • I changed my calculation software completely however it shutdowns
  • I updated Nvidia driver and this did not help
  • I tried random calculations in order to isolate hard disk read process but it still shutdowns
  • I reinstalled OS but the problem persists
  • I checked motherboard but no marks of burn

What is your suggestions? What might be the root cause for this?

Thank you.

  • Who authored your nvidia driver(s)? I'm seeing a number of Fedora-specific tutorials addressing nvidia proprietary drivers installation and Nouveau driver(s) uninstallation. – Ed Salter Jun 6 '18 at 22:00
  • How close are the 2 GPU could they be transferring heat back, and forth? Get a box fan for your window, open the side of the case and blow it in and see if that has any effect. – cybernard Jun 7 '18 at 3:49
  • @Ed Salter I installed official nvidia drivers together with cuda toolkit. No 3rd party repos has involved. – Sefa Jun 7 '18 at 17:35
  • @cybernard I had already removed AMD gpu and had tried but it was not successful. But I will give a chance to air blower. – Sefa Jun 7 '18 at 17:37
0

What are your CPU temps at? Improper usage of thermal compound ex: not enough or misaligned CPU cooler could cause the same issue due to overheating and one last thing to check would be your power supply, I recently had one that was failing under load (gaming) and would reboot my system but when I wasn't gaming would run for hours. Cheapest and easiest to check first would be the PSU if still no relief then look into redoing the thermal compound on the CPU and make sure your CPU cooler is properly seated.

  • I already mentioned in my post that: 1- "I did a CPU load test and it does not shut down" 2- "I added extra PSU using add2psu however still shutdowns" 3- "I manually set Nvidia GPU fan speed to %80 ... but it did not prevent shutdown" – Sefa Jun 20 '18 at 4:13
-1

I have inspected clock frequencies during task execution in Nvidia GPU and found that it really exceeds specification maximum. So in some lower states like P8 it does not shutdown but this over-overclocked frequency prevents it to work in higher states.

I cannot limit clock frequency of Nvidia GPU. However I managed to make it work by limiting power consumption of it but with the cost of %20 relative slower execution speed.

Command:

nvidia-smi --persistence-mode=1
nvidia-smi --power-limit=$POWER_LIMIT
  • Your GPU was overheating then. It's worth it to mention that in your answer. Reducing the power provided to the card will also lower temps. It might have been or still be possible to solve this problem without the performance hit by improving cooling sufficiently. – Cliff Armstrong Jun 20 '18 at 4:44
  • Most probably no, it is not about GPU heating. In my original post, I write that "I manually set Nvidia GPU fan speed to %80. It seemed to help because given task run a little bit longer than usual but it did not prevent shutdown". If I inspect temp, it does not exceeds 61C, which is negligible I think. – Sefa Jun 20 '18 at 18:09
  • Yeah, you wrote that... and it means exactly nothing. Running the fan at full speed is no guarantee of not overheating. On the other hand, I've got two decades of experience as a computer tech telling me your case is a textbook overheating issue. There's absolutely no harm in the GPU running at a higher clock rate... unless it causes the GPU to overheat. That the problem went away when you lowered the GPU clock speed all but proves it was an overheating issue. – Cliff Armstrong Jun 23 '18 at 0:55
  • So you are insisting on overheating. I am expecting you to explain these questions: Why my nvidia gpu temp sensor shows with 0.2ms intervals max 61 C and it immediately shuts down after a higher clock? Even with very low temp ice box adjacent to gpu does not prevent shutdown? Second question is, why limiting power does not prevent temperature increase which goes to 66 C during execution but prevents shutdown? – Sefa Jun 24 '18 at 19:18
  • Because those sensors aren't inside the gpu. It takes time for the heat from the gpu core to physically reach the sensor... no matter how often you take a reading from the sensor it can't overcome this. If the heat increases fast enough, there can be a very large gap between what the sensor shows and the actual temp. This also would explain why you see a higher temp with reduced power... because in that case, the sensor has time to take an accurate reading. And thats not even taking gram temps into account. – Cliff Armstrong Jun 25 '18 at 16:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.