I had been running a demo system on Linux with the following kernel:
2.6.32-400.1.1.el5uek #1 SMP Mon Jun 25 20:25:08 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
And as soon as swap happens, the CPU usage on the system goes up to almost 100% busy, among which around 90% is sys CPU usage. I was told that it's expected because the Linux page daemon kicks in and starts writing the memory chunk to swap only when running out of memory(or, free memory is less than 0.5G), which results in a lot of sys CPU usage. "swappiness" is 60 on the system.
Is it expected behavior that the CPU usage goes to 100% when swap happens on Linux?
The reason why I had the above question is, the above behavior has changed when I migrate the demo to another system with the following Linux kernel:
2.6.39-400.128.14.el5uek #1 SMP Sat May 10 00:06:52 PDT 2014 x86_64 x86_64 x86_64 GNU/Linux
The only difference of the vm parameter on the migrated system I noticed is the min_free_kbytes parameter, changed to 524288 on the migrated system vs 51200 in the pre-migrated system.
On the migrated system, when the free memory reaches less than 1.5G, swap happens, then the CPU usage goes up a little bit, something like from 30% to 40%, and the application performance got erratic only for a short period and then everything goes back to normal as before swap happens.
What makes the migrated system behave so different, that CPU usage won't go up to 100% busy when swap happens?
It's a very important demo system and the explanation of the above two questions will be very useful. Any ideas will be very much appreciated.