Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Recently one of my web servers has got a strange problem that it's load average suddenly increased from 0.01 to 15.00+ and then I can't even logon the server via ssh with a error message "no route to the host xxx.xx.xx.xx".But the problem just lasts for several minutes.The situation comes up about two or three times per day which is quite annoying me.In it's internat lan there're other servers deployed with the same OS and services, they work perfectly fine.I think it's a system problem.So how can I figure out why the problem comes up? Any system command for this kind of problem tracing?Help would appreciated.(OS eviroment:center os 5.2,with web serivce based on LNMP)

share|improve this question
    
Remember that the load average includes time spent waiting on I/O. Your system may be short on RAM and swapping a lot. –  Wyzard Nov 7 '12 at 3:29
add comment

1 Answer

up vote 1 down vote accepted

Check the following files

/var/log/syslog
/var/log/httpd/error_log

There is also a gui System Log Viewer if you can login the graphical interface.

You mention the situation will last several minutes. That may indicate the server actually crashed and reboot itself. Use following command to see if that actually happen

last reboot

If the server actually reboot, you will see line(s) like following

reboot   system boot  3.2.0-0.bpo.3-68 Fri Nov  2 18:25 - 21:46  (03:20)

Also try

last -x

Next Step 1

Check /var/log/syslog for process crash/kill and kernel message around the abnormal period. That may give some more clue.


Next Step 2

Following is a last resort if you are not able to sit right in front of that server and wait for it to happen. USE CAREFULLY

In /etc/crontab, add following line

#* * * * * root /usr/bin/top -b -n 1 | /usr/bin/head -n 15 >> /var/log/top.log

Every 1 minute, it will append the system summary with the top 8 highest cpu processes to /var/log/top.log.

It is disabled with # in front. Remove the # will enable it. You may want to change that path to some place you have a lot of space if you want to keep it running for overnight.

DISABLE IT RIGHT AWAY, AFTER YOU REGAIN CONNECTION Disable it by putting the # back or delete the line.

Check /usr/bin/top and /usr/bin/head are correct path for both commands.

share|improve this answer
    
Thanks for the help.I've checked the last but find no reboot record.Do you know how to find out which process caused the high load average? –  SpawnST Nov 7 '12 at 3:00
    
If you can login very quickly right after the connection come back, do a ps -ef and look for process that have very high cpu time. Assuming those processes didn't die and relaunch. You may also consider installing some server monitoring software to gain more info on cpu, load, swap, ram, disk usage. –  John Siu Nov 7 '12 at 3:12
    
,I've tried that but only see the load average decreasing slowly while no process seems take much resource. –  SpawnST Nov 7 '12 at 5:41
    
Added 2 more step you may try. –  John Siu Nov 7 '12 at 6:17
    
I got this error when I check the logs :BUG: soft lockup - CPU#0 stuck for 195s! [swapper:0] –  SpawnST Nov 7 '12 at 7:12
show 6 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.