Let me describe our requirement in detail. We are doing some kind of assignment which requires capturing of IP data through NIC card and then processing the captured data in real time. The processing of captured data can be done partially or fully depending on CPU availability. But whatever can be done will be done in real time.
Our architecture is something like as follows. We use one thread to capture IP data from NIC card. And numbers of threads are created to process the captured data in real time and this number can go upto anything (say 500) which are CPU intrinsic threads.
Here is our observations: Please note that we have one thread for IP data capturing. When we create fewer processing threads (say 10) and consume CPU at around 95% in total, there is absolutely no problem in IP data capturing threads. But when we create huge number of processing threads (say 250) and consume CPU at around 80% in total, the capturing thread can't capture IP data properly. In later case, CPU availability is higher, but problem happens.
So we though the problem could be due the number of threads and their scheduling (also context switching). We thought the following kind of CPU distribution among the threads. Using affinity, we will set 25% of cores to capturing thread and 75% of cores to other processing threads.
To start with this, we wrote a simple program on a quad core i7 machine with HT ON. Suppose the logical cores are A1 A2 B1 B2 C1 C2 D1 D2 (A B C D being the real cores). Our sample thread is such that it consumes the all resources on a single HT core. So if I run 8 such threads on total 8 logical cores, CPU consumption shows 100%. Now we tried to assign affinity. If we create 2 cores and set affinity to A1 and B1 (It's not one thread to A1 and other thread to B1, it's combined affinity for two threads), then it works fine and CPU usage shows 25%. But if we set combined affinity to A1 and A2 for two threads, then only 12.5% is consumed (task manager shows only one core is used). But if we set affinity of 4 threads to A1, A2, B1, B2, then CPU usage shows 50%. So the question is why the second case (12.5%) is not working?