18

I use Python to execute some algorithms and no matter which Python I use, and I have tried many versions, the CPU usage goes to 25% max. Why doesn't Python take advantage of the rest of my CPU resources? I changed the priority of the service from normal to high and later to real time, with restarts in between, but nothing changed.

Is there a way to make Python use 50% or even more of my CPU ?

  • is your cpu a multicore cpu? – Journeyman Geek Nov 23 '13 at 9:48
  • yes, it's an i5-480M and in control panel>power options>cpu min/max it's at 100% – Christos K. Nov 23 '13 at 9:50
16

Quite simply, you're running a single threaded application in a system with 4 logical cores - as such, you have one process, using all of the core.

You will (and this is non trivial) need to rewrite the algorithm to be multi-threaded, or see if you can just run 2 or more instances, on specific cores to use more of your CPU. There's no other way.

  • I was afraid that this would be the case, but in cpu usage diagrams in windows task manager i don't see 1 diagram be on it's pick while the algorithm is executed, on the contrary i see all of them with some significant increase. – Christos K. Nov 23 '13 at 9:59
  • 1
    Your system is balancing the load between cores. Still, no two cores are used at once. – gronostaj Nov 23 '13 at 10:05
  • guys you confirmed my fears, seems it's about time i start reading about threading – Christos K. Nov 23 '13 at 10:09
  • @fractal_7: Threading might not bring the benefits you expect. See my answer below. – Roland Smith Nov 23 '13 at 10:48
12

The Python language predates multi-core CPUs, so it isn't odd that it doesn't use them natively.

Additionally, not all programs can profit from multiple cores. A calculation done in steps, where the next step depends on the results of the previous step, will not be faster using more cores. Problems that can be vectorized (applying the same calculation to large arrays of data) can relatively easy be made to use multiple cores because the individual calculations are independent.

When you are doing a lot of calculations, I'm assuming you're using numpy? If not, check it out. It is an extension written in C that can use optimized lineair algebra libraries like ATLAS. It can speed up numerical calculations significantly compared to standard python.

Having said that, there are several ways to use multiple cores with python.

  • Built-in is the multiprocessing module. The multiprocessing.Pool class provides vectorization across multiple CPUs with the map() and related methods. There is a trade-off in here though. If you have to communicate large amounts of data between the processes, that overhead might negate the advantage of multiple cores.
  • Use a suitable build of numpy. If numpy is built with a multithreading ATLAS library, it will be faster on large problems.
  • Use extension modules like numexpr, parallel python, corepy or Copenhagen Vector Byte Code.

Note that the threading module isn't all that useful in this regard. To keep memory management simple, the global interpreter lock ("GIL") enforces that only one thread at a time can be executing python bytecode. External modules like numpy can use multiple threads internally, though.

  • So far i have used python 2.7 ironpython and tried pypy. i'll give numpy a chance. but still i'll have to read before i can make use of any multiprocessing module. – Christos K. Nov 23 '13 at 11:48

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.