'top' output baffles me

Question

top - 18:22:21 up 98 days, 23:26,  4 users,  load average: 2.73, 2.85, 2.60 
Tasks: 210 total,   1 running, 209 sleeping,   0 stopped,   0 zombie 
Cpu(s):  3.6%us,  2.5%sy,  0.0%ni, 87.7%id,  5.1%wa,  0.0%hi, 1.1%si,  0.0%st 
Mem:  15067172k total,  8230892k used,  6836280k free,  1729472k buffers 
Swap:  1952764k total,        0k used,  1952764k free,  1947068k cached

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND   
29512 root      20   0 6372m 3.0g  10m S   99 21.1  35:33.11 java      
4086 mysql     20   0 1692m 433m 7948 S   11  2.9  48:22.74 mysqld

How is it that on an 87% idle machine there is an 2 < average load < 3 situation?

Also, java and mysql proesses always take up > 100% CPU.

So, there is one indicator of idleness (87.7%) and two other indicators of activity (avg load and CPU%). How come?

PS. This is an 8-core CPU (or 2 CPU quad-core, can't recall), and this Linux is running on top of some virtualization environment.

Answer 1

top output on multi core machines is confusing. 100% represents full usage of a single CPU. Any multithreaded process that can occupy multiple cores will be shown as running at >100%. In your case, since you have 8 cores, you could see up to 800% usage.

Pressing 1 while in top will show you a breakdown of CPU usage by core.

The load average is the average load for the last 1, 5, and 15 minutes. It is scaled to cores. With an 8-core machine, a load average of less than 8 indicates non-full usage.

Load between 2 and 3 is about right for the amount of usage top indicates, and certainly nothing to worry about on an 8-core machine.