0

I am using deep learning in my research and I added a second 1080ti to my computer, because using only one would not let me finish my experiments in time. Tensorflow will utilize almost 100% GPU and some CPU.

Problem: If I run tensorflow with both GPUs the system turns off after about 30s and will not POST. I have to remove the first gpu to power on again (afterwards I can add the first GPU again).

PC specification:

  • 1x be quiet! Dark Power Pro 11 750W ATX 2.4 (BN252)
  • 1x ASUS Prime X370-Pro (90MB0TD0-M0EAY0)
  • 1x AMD Ryzen 5 1600 (TDP: 65W), 6x 3.20GHz, boxed (YD1600BBAEBOX)
  • 2 x MSI GeForce GTX 1080 Ti (250 W) Gaming X 11G, 11GB GDDR5X, DVI, 2x HDMI, 2x DP (V360-001R)
  • 1x Samsung SSD 850 EVO 250GB, SATA (MZ-75E250B)
  • 3x Seagate IronWolf NAS HDD 10TB, SATA 6Gb/s (ST10000VN0004)
  • 1x G.Skill Aegis DIMM Kit 16GB, DDR4-3000, CL16-18-18-38 (F4-3000C16D-16GISB)

I use PCIEX16_1 and PCIEX16_2 for the GPUs.

What I tried so far:

  • Running tensorflow with each of the GPUs (100% GPU utilization) --> OK for both GPUs
  • Checking the temperatur of both GPUs while using GPUs in parallel: --> OK max temp <80°C
  • Double checked that nothing is overlocked --> OK

Can someone please guide me trough the next steps to figure out the problem?

Thank you all for your help.

  • You first guess (not enough info to really determine a cause) would be overload protection in the PSU. Which leads me to some information which might be edited into your question: How are the graphical cards connected? You have 4 +12V rails. Are they all used? DId you use fewer and split them (not using some and overloading another)? And for this specific PSU, are they configured as 4x 12v rail or as one (there is a button on this PSU to set that). – Hennes Feb 24 at 11:17
  • Also, with connected I means if there is a direct cable going from these 4 PCI-e marked plugs to the 1080's. c1.neweggimages.com/ProductImageCompressAll1280/… and the 4x12v or combine button is the one marked OCK. – Hennes Feb 24 at 11:19
  • @Hennes thank you so much. This solved the problem. "Can you copy it as an answer. You have 4 +12V rails. Are they all used?" --> I now connected 3 +12v railes. Unfortunately, I can not have enough cables to use all 4. One cable has only a 6Pin. Do you think it is wort it to buy a cable even though everything seems to work? (I did not use OCK or am aware what it is used for) – user2672299 Feb 25 at 19:12
  • If you copy it into the answer I will mark it as the solution. :-) "You have 4 +12V rails. Are they all used?" – user2672299 Feb 25 at 19:17
  • Done. Expanded it a bit now I knew more about the setup. also wow, dual 1080Ti. I just use a single one on a single 4K screen. – Hennes Feb 26 at 19:37
1

A PC shutting down without warning can be many things, but two common problems include:

  1. Overheating something. Typically this will trigger #PROCHOT or similar and a full instant power off.
  2. Drawing to much power and triggering the safety in the PSU.

You descripbe a situation which you run max power on two very hungry graphics cards, which leads me to suspect the second problem.

Lookign at your graphics card they draw up to a massive 250 Watt each. Theroretical 75Watt of that may be supplied by the motherboards PCI-e slot. If a card needs more power it may add a 6 pin connector and draw up to 75 Watt from that, or a 8 pin connector and draw upto 150 Watt from it.

You card uses two 8 pins connectors and it likely draws most power from those.
(Drawing from the MB is not really recommended unless you want to heat that up. If the choice is for the PCI-e power connector that is the preferred choice).

How you have two cards and a total of 4 8 pins connectors. So in theory you need: * 4x 150W output from somewhere, or * 2x 300W output and a splitter from pins to 2x 8 pins * Or a similar setup where two 6 pins somehow merge to feed 1 pins.

Now your PSU is a great one. It has no fewer than four such plugs. So in theory you are fine.

but...
It is not guaranteed to be able to supply 600W on a single of those plugs. So you cannot use 1 PCI-e power connector on your modular PSU and split it in four.

If you lack the correct cables and draw from fewer plugs on the PSU then you may overload one of those plugs. That can lead to fancy sparks, or in case of a good PSU, overcurrent protection and instant shutdown.

THe latter is likely what happened on your PC.


It gets more complicated because your PSU als has a button to rebalance between different rails. I'll leave that part out of the answer since it would double or triple the size.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.