3

I'm trying to fix a problem with my company's multi-GPU computer, and I'm at my wit's end.

Whenever I start the computer with 5+ GPUs hooked up (we're trying to use 7), it shuts off before POSTing.

If I plug in 4 GPUs, it works just fine. I have had it running with 7 cards before, but all of a sudden it started doing this.

  • Motherboard: WRX80 Sage II (I do have those two 6-pins on the side of the mobo plugged in)
  • CPU: Threadripper 5955X
  • GPUs: Zotac 4090 (7x) (all of them have Gen 3 riser cables)
  • RAM: G Skill RipJaw 8x 32GB
  • HD: Samsung EVO Plus 500GB
  • PSU: Corsair HX1500i Platinum 80 Plus (4 PSUs in total, 2 GPUs per PSU, and another PSU powers a single card plus mobo)
  • UPS: 4x cyberpower 950W.

Here is what I've tried so far with no luck:

  • I have replaced the motherboard, RAM, CPU, and all the PSUs.
  • I unplugged each GPU (leaving the other 6 in) and powered it on to test all the PCIe slots, GPUs, and riser cables.
  • I tried turning the 4 switches on the motherboard on and off in various configurations. Not sure what they do, but it seemed worth a shot.
  • I updated the BIOS to the newest version.
  • I tried plugging the PSUs directly into the wall to bypass the battery backups.
  • I tried running the computer from the batteries on the UPS (unplugging the UPSs from the wall) to rule out the electrical system.

At one point I did manage to get 5 GPUs running, but I wasn't getting the full power out of the 5th card. Running a benchmark, 4 cards had a score of ~5600, and 5 cards had a score of 6300, which means that the 5th card was only delivering about 50% of the power it should have. Now it's back to only running with 4 or less cards.

I'm open to any ideas.

5
  • 1
    1)What changed from when it was working (with all 7x GPU) to now? Ex: new location, additional parts installed, etc. 2)any idea the amperage on the wall outlet? This site seems to imply US outlets are usually 15A & they'd suggest 20A when they did four 3090's: pugetsystems.com/labs/articles/…
    – gregg
    Commented Nov 26, 2023 at 5:01
  • That's the weird part, literally nothing has changed. It just all of a sudden shut off, then wouldn't turn back on with 5 or more cards plugged in. The computer is split between two 20A breakers, so there is plenty of power. Also keep in mind, the GPUs don't really use much power when booting up the PC.
    – Itsame
    Commented Nov 26, 2023 at 7:53
  • 2
    I've never seen a custom multi PSU PC so I googled to see how that'd work. It seemed to imply you'd need to signal to all PSU's that the PC is on so that implies other cables that aren't just the GPU are needed. 1)Are those in place & what are they; could they be bad? 2)do all PSU's have fans & LED's on? 3)do you have a watt-meter device like a kill-a-watt, if so what does it show for watts when plugged into the PSU that is powering motherboard?
    – gregg
    Commented Nov 26, 2023 at 14:35
  • Are you turning all 4 PSU on at the same time?
    – cybernard
    Commented Nov 28, 2023 at 21:28
  • If literally nothing changed between working and not working, the issue is likely a hardware fault, and you will need to approach it as such. I would guess this is most likely some sort of load issue, and it is possible there may be cascading failures too. You will need to swap parts, and I'm guessing you may need to swap multiple parts. Commented Dec 9, 2023 at 3:36

1 Answer 1

2

Unfortunately, there are overlapping areas of concern that will complicate troubleshooting:

  1. Motherboard firmware compatibility with that many GPUs (>4) tends to be non-trivial. It could be that downgrading your UEFI version could restore some functionality (at the loss of whatever improvements that the updates brought) if the system was working before an update. Consider that the motherboard manual only advertises support for up to 4-way SLI*. Some GPU-dense systems only support >4 GPUs when the additional GPUs are in alternative mode, which can require a tool to change and is almost certainly not compatible with a GeForce product. https://developer.nvidia.com/displaymodeselector
  2. PCIe riser cables are notorious for fragility and compatibility. The more you add, the more likely you are to have a problem. They seem to go bad just from ordinary use even without much handling. Changing BIOS settings so that a lower PCIe signal rate (gen3 instead of gen4) may have a minor improvement.
  3. Although the power supplies and distribution described seem more than sufficient, especially just to complete POST, there may also be a faulty PSU. Suggest reducing the number of GPUs to test each PSU individually, preferably without PCIe risers or with known good risers.

*Although SLI isn’t relevant to your configuration, it may point to the motherboard firmware only being tested with 4 GPUs in display device mode.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .