Sudden machine reboots observed with ipmiutil watchdog timer

Question

Originally raised on serverfault as this question.

I've got a production openSUSE 11.1 2.6.34.4 Intel server with a standard ipmiutil-2.4.1 installed.

The watchdog is basically a 90 second hardware timer cancelled every 60 seconds from user space by a cron job.

The cause of concern isn't the watchdog reboot itself but the underlying server fault causing it, if any at all. The IPMI watchdog IMO is doing its job to recover the system from an unrecoverable state.

The server appears to have become sluggish/slow. I say this based on 'missed/missing' /var/log/cron, syslog and other application log entries.

This has been happening randomly under load tests. The load is basically SIP traffic to a SIP server. The problem is not readily reproducible but it does happen quite frequently. Its also not deterministic with respect to hardware, time, type of load being run.

I'm at my wits' end and don't know if it's a bad driver, an i/o bug, something along these lines, the SIP application or something else.

I've got volumes of detailed system activity reports i.e. %usr, %nice, %sys, %iowait, %steal, %irq , pgpgin/s, pgpgout/s, fault/s, majflt/s, pgfree/s etc. if needed.

None of the system activity reports indicate something abnormal (although I don't claim to have an exactly trained eye).

Answer 1

I don't think it is a good idea to set a cron job for that critical task. Note that cron might block on other tasks prior to the execution of the watchdog task as described here. From what you describe, assuming that the system did not exhibit any abnormal behavior other than the reboot, it's very likely that the cron job simplely did not run in time.

There are standalone watchdog daemons you can use that would reset the timer every second or so. I suggest you use one of them. See BMC watchdog.