I'd like to take advantage of 40 physical cores in my 2 * Xeon gold 6230 system with 64GB (2 * 32GB) memory installed. Operating system is Ubuntu 18.04.
The task is solving eigenvalues of random matrices as many as possible. For small matrices, in my test about 300x300, by increasing independent instances of eigensolver workers, the performance has no significant lost. For example, ten worker solving 1000 matrices (total of 1E+4 matrices), twenty workers each solving 1000 matrices (total of 2E+4), and forty workers each solving 1000 matrices (total of 4E+4 matrices), they take approximately the same real time to finish.
But when the matrix is large (2000x2000), MKL performance drops significantly when increase workers. MKL_NUM_THREADS=1 in all tests.
- 1 worker, 10 matrices each: 1m15s to finish (CPU 100%)
- 10 workers, 10 matrices each: 2m23s to finish (CPU 1000%)
- 20 workers, 10 matrices each: 5m34s to finish (CPU 2000%)
20 workers get more than twice worse performance than 10 workers.
Tests are performed in Mathematica 10, Matlab 2019b, python 3.7, and eigen3 (link to intel mkl). Memory usage is below 12%. Test code is simple, for example, the Mathematica code reads:
mat=Table[RandomReal[],{2000},{2000}];
ParallelDo[Do[Eigenvalues[mat],{10}],{i,1,1}]//AbsoluteTiming
ParallelDo[Do[Eigenvalues[mat],{10}],{i,1,10}]//AbsoluteTiming
ParallelDo[Do[Eigenvalues[mat],{10}],{i,1,20}]//AbsoluteTiming
Any idea of improving mkl performance or determining the bottleneck of hardware is appreciate.
update: after replacing the 32GB/CPU with 16GB*2/CPU, MKL performance gets improved
- 01 worker, 10 matrices each: 1m12s to finish
- 10 workers, 10 matrices each: 1m56s to finish
- 20 workers, 10 matrices each: 3m01s to finish
- 40 workers, 10 matrices each: 6m59s to finish
It seems that memory bandwidth is still the bottleneck of multi-process MKL tasks.
