1

I have a python program that uses the threading module and runs multithreads that executes a function. The number 8 below is the number of threads in the program but I need to know what is the number of threads my device can handle to run the program as fast as possible.

Here is the concerned part in my python program: 

for i in range(8):
      t = ThreadUrl(q)
      t.setDaemon(True)
      t.start()

Where q = queue.Queue()

My devices CPU data is (using lscpu command in Ubuntu 18.04):

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  2
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               94
Model name:          Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz
Stepping:            3
CPU MHz:             1899.473
CPU max MHz:         3500.0000
CPU min MHz:         800.0000
BogoMIPS:            5184.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            6144K
NUMA node0 CPU(s):   0-7

Can you tell me what is the number of threads my program can handle without causing issues? And how is this computed please?

Improve this question
4
  • "how is this computed please?" - Please clarify this question by editing your question. Please do not reply with a comment.
    – Ramhound
    Mar 4, 2019 at 20:35
  • 1
    You need to tell us about each aspect of your program. For example, if each thread uses 3g of RAM and you have 8gb your practical limit is 2. If each thread requires 10 minutes of download at 90% capacity your limit is 1 because you don't have enough internet speed. You might also have a hard drive bottleneck. CPU is only 1 aspect of many that need to be considered.
    – cybernard
    Mar 4, 2019 at 20:43
  • You may find better answers at stackoverflow.com , a similar site to SuperUser , designed for programmers. This sort of question has definitely been asked before, so do some searching and you'll probably find what you need. SuperUser is more suited for IT rather than programming questions.
    – MountainMan
    Mar 4, 2019 at 23:04
  • As others have commented, assuming nothing else is running, you can run 8 Python threads at the same time on an 8-core processor. If more Python threads are added, they will just steal core time from the existing threads. It appears that Python cannot use Hyperthreading : stackoverflow.com/questions/49943593/…
    – MountainMan
    Mar 4, 2019 at 23:08

1 Answer 1

Reset to default
3

Can you tell me what is the number of threads my program can handle without causing issues?

The i7-6700HQ is a 4 core processor. Each core supports Intel® Hyper-Threading Technology. This means your processor can support 8 threads.

The number 8 below is the number of threads in the program but I need to know what is the number of threads my device can handle to run the program as fast as possible.

Your code is already optimized for the hardware you have.

enter image description here

enter image description here

Improve this answer
4
  • Thanks. What if I run multiple instances of the program, each with 8 threads? Is this possible or the total number is 8 threads no matter how many instance (by multiple instances I mean, run the program from a terminal windows: python myprog.py, then run another run from another terminal, etc.) Can I make each run use 8 threads? and what is the maximum number of instances I can run?
    – user9371654
    Mar 4, 2019 at 20:49
  • It is not possible, with the information you provided, to predict how many instances of your python program you can run. A broad estimate would involve a combination of how much each instance would task your CPU and the amount of memory each instance required. Your Python code would only be able to run 8 threads concurrently, multiple instances of the same code, would not help you process data faster. How you make your Python code faster would is beyond the scope of this question. Any additional questions on how to optimized your Python script will be ignored.
    – Ramhound
    Mar 4, 2019 at 20:57
  • multiple instances of the same code, would not help you process data faster., sure. But my program make web scraping ofr URLs. Therefore, multiple instances will allow me make/scrap more URLs concurrently. Isn't it?
    – user9371654
    Mar 5, 2019 at 15:08
  • You can benchmark that case using any machine you have. You don't need to have an 8 core processor to test your code. I already stated once before, optimization questions, would not be answered.
    – Ramhound
    Mar 5, 2019 at 15:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .