Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Looking at the output of top on our server, one of my colleagues told me that the fact that some processes got less than 100 "%CPU" was because I was running too many processes. He added that based on his experience if I run less than 6 processes, then probably all the processes would have 100 "%CPU".

I don't want to be an annoyance to other users, but I doubt what he said is correct. The server has 16 cores and the current load average is between 10 and 11. From what I have learned, it is not overloaded. But I don't know why some processes are just getting less than 100 "%CPU"? Is it really because of me?

Thanks and regards!

Here comes the output of top:

top - 16:34:13 up 32 days,  1:36, 12 users,  load average: 10.61, 10.39, 10.22
Tasks: 380 total,  10 running, 370 sleeping,   0 stopped,   0 zombie
Cpu(s): 55.0%us,  1.7%sy,  0.0%ni, 42.2%id,  0.5%wa,  0.1%hi,  0.4%si,  0.0%st
Mem:  130766620k total, 39859784k used, 90906836k free,   849412k buffers
Swap: 47351548k total,   279456k used, 47072092k free, 19792956k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                        
17197 tim    18  -2 1315m 1.3g 1504 R  100  1.0   4510:11 MLtest                                                                                       
28762 tim    18  -2 1315m 1.3g 1504 R  100  1.0   4633:01 MLtest                                                                                       
29249 tim    18  -2 1315m 1.3g 1504 R  100  1.0   4623:03 MLtest                                                                                       
29560 tim    18  -2 1315m 1.3g 1504 R  100  1.0   4626:59 MLtest                                                                                       
 4904 tim    18  -2 1315m 1.3g 1504 R  100  1.0   4757:12 MLtest                                                                                       
 5143 tim    18  -2 1315m 1.3g 1504 R  100  1.0   4759:40 MLtest                                                                                       
29389 tim    18  -2 1315m 1.3g 1504 R   99  1.0   4622:11 MLtest                                                                                       
 5285 tim    18  -2 1315m 1.3g 1504 R   97  1.0   4758:49 MLtest                                                                                       
 4763 tim    18  -2 1315m 1.3g 1504 R   93  1.0   4754:22 MLtest                                                                                       
 9456 zma    18  -2  206m  85m  11m S   48  0.1  60:46.78 dropbox                                                                                         
 7527 vals   18  -2 1266m 436m  42m S    4  0.3 613:57.10 MATLAB                                                                                          
 2903 root   15  -5     0    0    0 S    1  0.0  19:00.01 rpciod/0                                                                                        
19133 vals   18  -2 1380m 503m  42m S    1  0.4 798:47.99 MATLAB                                                                                          
12454 tim    18  -2 19248 1588 1024 R    1  0.0   0:48.88 top                                                                                             
   12 root   RT  -5     0    0    0 S    1  0.0  35:01.05 migration/3                                                                                     
 2924 root   15  -5     0    0    0 S    1  0.0  27:20.92 nfsiod                                                                                          
12690 jun    18  -2  913m  84m 2684 S    1  0.1 121:55.65 MATLAB                                                                                          
19650 jun    18  -2 19244 1600 1028 S    1  0.0   6:5
share|improve this question
16 cores and 128G physical RAM? That's quite a machine. – Doug Harris Oct 5 '09 at 21:03
It is a server used for computation in our school. – Tim Oct 5 '09 at 21:06
Thanks Doug! Did you type every single space to make the top output aligned so well? – Tim Oct 5 '09 at 21:11
An interesting thing about this question is that I get the sense that Tim is trying to push this server to its limit. Many of the rest of us running services used by actual humans (instead of computation) are worried about responsiveness and, thus, strive for the opposite. Once a server reaches more than 60% capacity, it's time to start adding another server to the mix. I'm not saying that Tim's outlook is wrong -- it's right for this use case. Just interesting is all I'm saying. – Doug Harris Oct 5 '09 at 21:14
Tim -- no, I used emacs to do a couple quick edits. – Doug Harris Oct 5 '09 at 21:15

The old rule of thumbs I use is:

Acceptable load average ≤ number of cores +1

That assures that there is never more than one process waiting while having your resources fully used. However loaded that way, the machine might seem a little unresponsive when used interactively, you might consider to free some resources for this purpose.

The drop you see in %cpu for more than 6 concurrent processes might be caused by a lot of different factors (cpus are not the only shared resource ... ), to know which one(s) you would need to do some profiling on your program.

Anyway, as this machine seems shared between several users for calculation purpose, I would strongly advise to set up some form of job scheduling, even for shell access (you may also introduce some load balancing that way). There's quite a few tools to enable this, the two I use are Torque/PBS and Sun Grid Engine (both OSS projects).

share|improve this answer

From what I see in the top output you posted, I think the mltest can't eat up more cycles than that. It consumes almost 8x 100%. I'm not a CPU architecture specialist, but I think you hitting a limit of that multicore technology. How much cores per CPU does your server have?

BTW, you have a load of 10 or 11 on your "Quad-Quad Machine"? Or is that the average "%CPU" you see? load is the number of processes waiting for a CPU cycle, so 10 is rather high.

Update after the comment:
Thanks for reformatting the top output and for clarification concerning the load value. I kind of disagree with my collegues here - a load of 10 is fairly high, no matter how much cores you have. But you are running a benchmark or CPU burn-in test, aren't you?

For further answers, please see the cross post on ServerFault:

share|improve this answer
The server has 16 cores. Load average is shown at the end of the first line. Are these what you asked? – Tim Oct 5 '09 at 21:05
My understanding of load average is average number of processes waiting for CPU. With 16 cores, I don't think 10 or 11 is too high. On my 4 core servers, I'm happy if load is below 4. – Doug Harris Oct 5 '09 at 21:08
Similar sentiment on serverfault: "If your load level is under your processor count, then you're generally OK." – Doug Harris Oct 5 '09 at 21:11
Thanks Doug! Do you think that some processes are just getting less than 100 "%CPU" is because of me? BTW, what's the difference between SuperUSsr and SeverFault these two sites? Should I post my questions there? – Tim Oct 5 '09 at 21:15
re SuperUser v. ServerFault: SuperUser is focused toward computer enthusiasts, and Serverfault is focused toward sys admins and IT professionals. If enough people feel a move is warranted your question may be migrated to one or another of the trio (these two, plus StackOverflow, for programming.) – JMD Oct 5 '09 at 21:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .