I'm building a multi-GPU machine for scientific calculations. I'm going to pair a Xeon E5-2620v3 CPU with the Asus X99-E WS motherboard to install 3-4 Titan X GPUs. The CPU provides 40 PCIe lanes and the chipset provides additional lanes, so that one can run up to quad x16 setups on a 40-lane CPU. What bothers me is the theoretical performance impact originating from the limited bandwidth. How critical is that for applications with almost 100% GPU-bound loads if GPUs will be constantly loading data from RAM? Should I rather go with 2 CPUs?

  • If it’s “constantly loading” at 1KB/s it won’t matter much. So, the answer is of course: It depends. Also, what limited bandwidth? – Daniel B Oct 18 '16 at 12:45
  • @DanielB the number of PCI-lanes (without switches) limits the data bandwidth for the GPUs. It will be constantly loading gigabytes of memory. – Eli Korvigo Oct 18 '16 at 13:00
  • I’m pretty sure the 2×8GT/s QPI is up to the job. Again, you’re not specifying the desired throughput. If I’m on ISDN, I can also be “constantly loading gigabytes” of data. You need to consider when and how the data is transferred: Constantly over time or one big transfer every once in a while? How much data is really transferred? What’s the overhead involved in transferring the data? – Daniel B Oct 18 '16 at 13:25
  • @DanielB most likely each card will have to load up to 100 GBs of data every 1-5 seconds. – Eli Korvigo Oct 18 '16 at 14:00
  • Intel ARK says: “Max Memory Bandwidth: 59 GB/s”, so that’s a no. You might want to rethink your endeavor. – Daniel B Oct 18 '16 at 14:04

Check guides for cryptocurrency mining farms. That is pretty close to your needs. 1x PCIe throughput is enough for the most calculation-related cases. So you can wire (with rizers) up to 8 GPU per motherboard with standard windows drivers. More gpus require specialized os and drivers support.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.