Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I need to run some heavy CUDA tests and I can buy something like 4-5 quadro or tesla graphic cards. My question is: what kind of hardware should I buy to use them in a Windows environment with Nsight? I don't think I need a normal desktop pc because they only have something like 2-3 PCI-Express slots. What would I need to set up such a system? If possible I don't want to use linux because I'm confortable with VS and Nsight

share|improve this question
add comment

1 Answer

up vote 2 down vote accepted

If two or three PCI-e slots are not enough you can just buy a motherboard with more slots. You certainly do not want to buy a pre-build desktop. Not just because those often lack proper expandability but also because 4 or 5 CUDA cards are going to use a lot of power. You want a custom build with a big PSU and proper cooling.

Secondly you want to look for a motherboard with PCI-e version 3. Not for the bandwidth (which is twice as high as PCI-e version 2), but for the lower latency.

A quick google search on "motherboard 5 pci express slots" yields several results. One of these is this motherboard with 7 PCI-e slots. *

Mind you, 7 PCI-e slots does not mean you can always use 7 PCI-e cards.
It is likely that you will run into physical limitations. By which I mean thatCUDA cards are often 2 slots wide and that you can not cram one in each slot. Instead you are likely to end up with something like the picture below:

Motherboard picture with 2 wide slots marked

Assuming no other cards (e.g. no RAID, no graphics card, ...) and a wide enough tower you might be able to cram four cards in the example motherboard. If you want more you probably have to work with PCI-e expanders, and even then it is going to be tricky.

It is probably best to work with a few high performance CUDA cards rather than try to use a lot of lower performance ones. Checking Nvidia's list of CUDA cards really makes it easy to select a card. Atm that is a Tesla K20 (two K20's will outperform three Tesla's C2075, 2050 or C2070's). And they are indeed two slots wide.

That probably means that you can use up to three K20's, leaving one slot free.

Note that each of these cards uses 235 Watt power. That is 705 Watt for three of them. And that is without power for the motherboard (say 50 Watt), the CPU (100-ish), RAM (not much per DIMM, but you probably have several of these), fans (3-5 Watt each, assume half a dozen in your tower), harddrives, optional graphical card (not really needed, you can run without a monitor and just RDP in), .... In short: You need a big PSU and a lot of cooling.

All of this ignores minor details, such as which slots are PCI-e v3. Are they all V3, or just one or two slots and is the rest v2? How many connectors are x16 and how many fall back to x8 when you use two slots. (I can look that up for the example board, but that is just an example. Check the manuals of all relevant motherboards to help you make a choice).


BTW, you are not the first to ask this. Some, like Fastra already build similar PCs. That was a a while ago and hardware has moves on, but reading on how they build it is still quite interesting.

enter image description here



* Disclaimer: I do not work for ASRock. That is just to first good link I found.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.