9

While devising an answer to this question I ran into a snag while testing this MySQL Monit ruleset on an Ubuntu 12.04.5 setup:

check process mysqld with pidfile /var/run/mysqld/mysqld.pid
  group mysql
  start program = "/etc/init.d/mysql start"
  stop program = "/etc/init.d/mysql stop"
  if failed host 127.0.0.1 port 3306
    with timeout 15 seconds
  then restart
  if 5 restarts within 5 cycles
  then timeout
  alert email_address@example.com only on { timeout, nonexist }

The issue is I was attempting to invoke start/stop items via /etc/init.d/ — which is more of a CentOS/RedHat system construct — instead of using /usr/sbin/service which would be more appropriate for a Ubuntu/Debian system.

Okay, my bad… But the issue is you see that if 5 restarts within 5 cycles then timeout part? That seems to have bit me hard. With the /etc/init.d/mysql start command not able to work, the system attempted 5 restarts, failed 5 times and then timed out as a result. And the timeout condition seems to result in the MySQL service ruleset being ignored my Monit.

I’ve restarted the Monit service a few times and even rejiggered the ruleset to see if it helps but none of that seems to affect anything.

What can I do to get Monit to pay attention to rulesets it has “unmonitored” due to timeout conditions being met?

6

After doing some digging, it turns out Monit stores system monitoring data in a “state” file. And this “state” file keeps track of what services are being monitored/unmonitored.

So while this is a bit “brute force”-ish, it definitely works. If a service becomes “unmonitored” due to something like a timeout, then just remove the Monit state file from the system like this:

sudo rm /var/lib/monit/state

And then restart Monit like this and all should be good:

sudo service monit restart

FWIW, on other systems/setups the Monit “state” file might be saved as state or monit.state or even .monit.state (with a dot/period . prepending it) in another directory. Be sure to determine exactly where that “state” file is being saved when you actually attempt to implement this fix.

3
  • 1
    On a related note, when trying to find this state file on amazon linux, I was (finally) able to find it at /root/.monit.state
    – Scott
    Jun 26 '17 at 14:58
  • @Scott Good point! Adjusted my answer to reflect the fact that the “state” file might be located elsewhere with a different filename entirely. Jun 26 '17 at 18:15
  • 1
    You should be able to find the location of your statefile in the Monit config file (/etc/monit/monitrc) with e.g. grep statefile /etc/monit/monitrc
    – user51928
    May 19 '18 at 0:31
5

Monit includes commands to enable and disable monitoring of all or specific services.

If a service has become unmonitored you can re-enable monitoring with e.g. monit monitor mysql or monit monitor all.

Note you must have the Monit HTTP interface enabled for these commands to work.

2
  • “Note you must have the Monit HTTP interface enabled for these commands to work.” That makes no sense. May 19 '18 at 1:36
  • 1
    The Monit CLI uses the HTTP interface to communicate with the Monit daemon. "Note that if HTTP support is disabled, the Monit CLI interface will have reduced functionality, as most CLI commands (such as "monit status") needs to communicate with the Monit background process via the HTTP interface. We strongly recommend having HTTP support enabled." mmonit.com/monit/documentation/monit.html#MONIT-HTTPD Now whether or not that makes sense is a question for the developers :)
    – user51928
    May 20 '18 at 19:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.