I'm running my java program on a 12 cores 24 thread machine. They have several processes that are running simultaneously. It seems that I performed too many processes so that the whole tasks made the machine very slow.
Here is the top information
Tasks: 556 total, 2 running, 554 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.1%us, 0.4%sy, 0.0%ni, 63.2%id, 36.3%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 16295248k total, 16169560k used, 125688k free, 3300k buffers
Swap: 18530296k total, 10867972k used, 7662324k free, 46188k cached
It seems that my processes is memory-consuming-oriented so that almost all the memory was used by them. In the top information what is I don't understand is why only 2 tasks are running instead of 23 (I have dispatched 23 processes).
free -g
total used free shared buffers cached
Mem: 15 15 0 0 0 0
-/+ buffers/cache: 15 0
Swap: 17 10 7
It seems that all the memory was used and it was swapping made the machine slow down.
ps -e -o pid,%cpu,%mem,vsz,rss,comm= --sort=vsz
29707 5.6 4.2 6268732 685660 java
29712 5.2 3.9 6268732 647352 java
...
30269 3.2 4.3 6268732 704676 java
30334 4.8 4.2 6268732 689544 java
There are 23 such java processes. Summing all the %cpu, it is very close to 100%. But the top information indicating that the CPU is not busy.
Cpu(s): 0.1%us, 0.4%sy, 0.0%ni, 63.2%id, 36.3%wa, 0.0%hi, 0.0%si, 0.0%st
I googled what size of vsz and rss but didn't find out. I assume that the unit is in kilo byte. Watching the vsz then the java processes are using 6268732kb*23=144,180,836 =~ 144gb, which seems to be impossible to put in the RAM because it is far more then my RAM (16gb), so only 700000kb*23 =~ 16gb were put into the memory (with the rss info, which is the portion of the data store in the RAM). Because of the frequent swap and context switch made the system slow down.
I don't know my conclusion is correct or not. Please give me some advice and how may I fix the problem.
Add more detail:
vmstat -a -S M
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free inact active si so bi bo in cs us sy id wa st
2 29 16792 124 2105 13152 0 0 29 23 2 0 1 0 95 4 0
I don't know how should I interpret the vmstat data. It's a little bit weird because swpd: the amount of virtual memory used looks high while si and so are 0.
vmstat
to see the actual virtual memory page faults (si
andso
). Usevmstat 1
to show it continuously in a second interval.vmstat
also shows IO activity. Useiotop
for more detailed per-process IO activity monitoring.