I am running a computer with 4 Nvidia Titan X for training deep neural networks. I have problems with my power consumption. The consumption of Titan X is not supposed to be above 250W but as you can see on the plot, it can be above 350W. When I use nvidia-smi to monitor the power consumption, the parameter enforced power limit is set to 250W as it should be.

Log of power consumption

At first I thought it was not a problem except I cannot run all 4 GPUs simultaneously on my 1200W PSU Corsair Axi. However when I tried a different neural network more computationally expensive on a single GPU I crash the computer. In fact I have burnt the PSU and I have sent it back. I have now bought a bigger PSU (1600W) but I do not want to have the same problem again.

I don't remember how the PSU was plugged but maybe the GPU I used was plugged with something similar to this since Titan x needs 2 pci-e connector and my PSU had only 8 ( 4*2 for GPUs and 1 for CPU it is not enough). If yes the GPU was plugged on a single pci-e output instead of two.

Is it possible that the GPU consumed more than possible on this entry (40A, 480W) and burned the whole PSU ? Can I really enforce the 250W power limit ?

  • 8 pin connector 150w x2 plus 75w from pci-e slot =375w is the max that should be drawn. That is just the rated limited for the wires, some manufacturers break that limit. – cybernard Jun 28 '18 at 16:46
  • Also, 250Watt the max it can draw or the average max it draws (so it can draw more and probably heat up and then throttle down, thus staying on average below the 250W over a period of time). – Hennes Jun 28 '18 at 17:07
  • This thread talks about nvidia-smi bugs related to: a) multiple cards; b) related to startx/stopx; and c) problems when the iGPU is not the first enumerated device (using hdmi dummy plugs to get the device enumerated). Might be worth pulling all but one and testing, reading the thread for brainstorming purposes etc. ( devtalk.nvidia.com/default/topic/1011633/linux/… ) – Yorik Jun 28 '18 at 17:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.