Recently one of my web servers has got a strange problem that it's load average suddenly increased from 0.01 to 15.00+ and then I can't even logon the server via ssh with a error message "no route to the host xxx.xx.xx.xx".But the problem just lasts for several minutes.The situation comes up about two or three times per day which is quite annoying me.In it's internat lan there're other servers deployed with the same OS and services, they work perfectly fine.I think it's a system problem.So how can I figure out why the problem comes up? Any system command for this kind of problem tracing?Help would appreciated.(OS eviroment:center os 5.2,with web serivce based on LNMP)
Check the following files
There is also a gui System Log Viewer if you can login the graphical interface.
You mention the situation will last several minutes. That may indicate the server actually crashed and reboot itself. Use following command to see if that actually happen
If the server actually reboot, you will see line(s) like following
Next Step 1
Check /var/log/syslog for process crash/kill and kernel message around the abnormal period. That may give some more clue.
Next Step 2
Following is a last resort if you are not able to sit right in front of that server and wait for it to happen. USE CAREFULLY
In /etc/crontab, add following line
Every 1 minute, it will append the system summary with the top 8 highest cpu processes to /var/log/top.log.
It is disabled with # in front. Remove the # will enable it. You may want to change that path to some place you have a lot of space if you want to keep it running for overnight.
DISABLE IT RIGHT AWAY, AFTER YOU REGAIN CONNECTION Disable it by putting the # back or delete the line.
Check /usr/bin/top and /usr/bin/head are correct path for both commands.