0

After a search for the best GPU for my budget, I landed on the older K80 due to its attractive VRAM/$ ratio. However, the K80 is a bit quirky as it is a sort of a 2-in-1 card and is passively cooled - it's really only meant to run in a GPU server. So I decided to pick up an older GPU server, the SuperMicro 2027GR-TRF; so far I've installed one K80 but it can support two eventually.

Even so, I've still had some issues around the cooling, which frustrates me because I'm trying to go with the officially supported route for this card. Specifically, if I have the fans set in IPMI to "Optimal", I see one half of the K80's temperature just rise and rise (according to nvidia-smi) until it reaches ~93C and thermals off. Not good! Fortunately, the rig has beefy fans, so if I simply set the fans to "Full" in IPMI it stabilizes at around 51C but is loud enough I put on protective hearing gear. This is not ideal. Unfortunately the BIOS has no fan options, and IPMI only exposes "Full" or "Optimal" (at least that pickable), so I don't have a lot more obvious things to try.

Now, I'm new to the server space but I'm surprised that this doesn't seem to be automatic and I'm trying to understand why. I can see that IPMI sees the GPU1 temp (but does not see a GPU2 temp which maybe one might want with the K80?), so I would expect that to feedback into a closed loop for fan control.

I'm puzzled about this and am looking for some understanding around the issue.

  1. Is closed loop fan control usually the job of BIOS, IPMI, the OS, or user-installed programs? Given the fact the GPU often runs hardest while the CPU is also getting pegged, it seems odd to me that anything other than a dedicated subsystem should be responsible for a critical function like this.
  2. If it is the function of IPMI, is there a standard mechanism to tweak the feedback control? I've seen plenty around controlling the fan speed via setting thresholds, but really what I think I need to set is setting fan RPMs in response to GPU temps - maybe make that more aggressive than it currently is.

Anyways, I'd appreciate it if you can help me solve this so that I can keep my hearing... Thanks!

2 Answers 2

2

I can confirm the issue on Supermicro 2029GP-TR with Nvidia A100 GPU.

Initially, with Fan mode set to "Optimal", IPMI sensors at "Sensors reading" page on IPMI web GUI shows only "GPU1 Temp" sensor with 2 GPU installed. When testing GPUs with gpu-burn platform FANs changed speed according to GPU temperature only for GPU1, and stay on 3000 RPM for GPU2.

The problem was solved by updating IPMI firmware with 'Preserve configuration' and 'Preserve SDR' unchecked.

1
  • Thanks for reporting this on the similar platform!
    – J Trana
    Jul 28, 2021 at 16:25
1

I got it working!

In the past, I tried it on Optimal and I thought I heard a bit of fan speed ratcheting, but it wasn't enough. So I started looking for answers. I had updated IPMI but not BIOS in my quest. Today I kicked it into Optimal via IPMI raw commands and tested it again and ... now it works! It stabilizes right at 60C +/- 1C.

I had noticed in the past that without the NVidia drivers loaded, I don't think I saw GPU1 Temp engage.

So, I'm going to go with a combination of update IPMI and NVidia drivers as the probable cause for it working. I'm pleased to see the system handle this without needing to resort to a manual script, etc. to control this.

2
  • 2
    +1 same issue, IPMI update fixed it. Thanks for documenting this.
    – foreverska
    Jun 18, 2021 at 22:38
  • No problem! Glad it helped.
    – J Trana
    Jun 20, 2021 at 3:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .