I want to concurrently run independent instances of a program on our 32 core machine (AMD Threadripper 2990wx, 128GB DDR4 RAM, Ubuntu 18.04). The performance gains are almost null after about 12 processes, and using half the CPU is about as fast as using it entirely (you gain 4%!). I want to identify the source of this scaling bottleneck. Here is a plot of the average speedup:

Poor scaling on AMD Threadripper 2990WX

How should I go about investigating this bottleneck, in terms of external tools or specific things to look for in the code ?

My guess is it has to do with memory access and the NUMA architecture. All cores seem to be working 100% all the time according to htop. I tried experimenting with numactl and assigning a core to each process, without noticeable improvement. Each process uses at most 1GB of memory. It is written in C++, and there is no parallel code (no threads, no mutexes), each instance is totally independent. The program is a scientific simulation where lots of objects are progressively created and regularly updated.

I have also tried using AMD uProf but I am not sure what I need to look for.

Any help would be greatly appreciated.

closed as too broad by Ramhound, Sathyajith Bhat Jul 11 at 5:01

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • "Totally independent"? There is no single I/O or resource bottleneck where all threads write or show progress? – DrMoishe Pippik Jul 11 at 3:18
  • No, there is no single I/O resource bottleneck that I can think of (other than RAM, obviously, but I don't think that qualifies). Once the simulation is over, the process will optionally print a result. – Petipo Jul 11 at 5:39
  • I have edited the question to make it less broad, but the nature of the problem implies that I don't exactly know what I am looking for. The question is about how and what I should investigate to understand the source of this poor scaling. I was hoping someone with a finer knowledge of NUMA could have some interesting insights. – Petipo Jul 11 at 5:44