How to increase Python's CPU usage

Question

I use Python to execute some algorithms and no matter which Python I use, and I have tried many versions, the CPU usage goes to 25% max. Why doesn't Python take advantage of the rest of my CPU resources? I changed the priority of the service from normal to high and later to real time, with restarts in between, but nothing changed.

Is there a way to make Python use 50% or even more of my CPU ?

Answer 1

Quite simply, you're running a single threaded application in a system with 4 logical cores - as such, you have one process, using all of the core.

You will (and this is non trivial) need to rewrite the algorithm to be multi-threaded, or see if you can just run 2 or more instances, on specific cores to use more of your CPU. There's no other way.

Answer 2

The Python language predates multi-core CPUs, so it isn't odd that it doesn't use them natively.

Additionally, not all programs can profit from multiple cores. A calculation done in steps, where the next step depends on the results of the previous step, will not be faster using more cores. Problems that can be vectorized (applying the same calculation to large arrays of data) can relatively easy be made to use multiple cores because the individual calculations are independent.

When you are doing a lot of calculations, I'm assuming you're using numpy? If not, check it out. It is an extension written in C that can use optimized linear algebra libraries like ATLAS. It can speed up numerical calculations significantly compared to standard Python.

Having said that, there are several ways to use multiple cores with python.

• Built-in is the multiprocessing module. The multiprocessing.Pool class provides vectorization across multiple CPUs with the map() and related methods. There is a trade-off in here though. If you have to communicate large amounts of data between the processes then that overhead might negate the advantage of multiple cores.
• Use a suitable build of numpy. If numpy is built with a multithreading ATLAS library, it will be faster on large problems.
• Use extension modules like numexpr, parallel python, corepy or Copenhagen Vector Byte Code.

Note that the threading module isn't all that useful in this regard. To keep memory management simple, the global interpreter lock ("GIL") enforces that only one thread at a time can be executing python bytecode. External modules like numpy can use multiple threads internally, though.