I want to concurrently run independent instances of a program on our 32 core machine (AMD Threadripper 2990wx, 128GB DDR4 RAM, Ubuntu 18.04). The performance gains are almost null after about 12 processes, and using half the CPU is about as fast as using it entirely (you gain 4%!). I want to identify the source of this scaling bottleneck. Here is a plot of the average speedup:
How should I go about investigating this bottleneck, in terms of external tools or specific things to look for in the code ?
My guess is it has to do with memory access and the NUMA architecture. All cores seem to be working 100% all the time according to htop. I tried experimenting with numactl and assigning a core to each process, without noticeable improvement. Each process uses at most 1GB of memory. It is written in C++, and there is no parallel code (no threads, no mutexes), each instance is totally independent. The program is a scientific simulation where lots of objects are progressively created and regularly updated.
I have also tried using AMD uProf but I am not sure what I need to look for.
Any help would be greatly appreciated.