I'm currently running a personal project that basically monitors a few things around the clock. One is a process that uses PhantomJS, and the other scans some other APIs, and sends emails if some certain situation occurs.
When all is up and running, it works exactly how I want, but unfortunately, at the moment, every 2-3 days, something occurs normally in the dead of night, and the API scanner just freezes. It's written in Python, and runs on a tick of every 3 seconds, and when I check the log next day, no crash, nothing looks out of the ordinary, but for some reason, it's stopped running.
When I've time, I will eventually get around to fixing that, but because it only occurs once after 3, or 4 days running, it's been very difficult to reproduce.
For the PhantomJS stuff... every now and again, that just blows up. Again, there's going to be cleanup happening there, fixing these issues, and potentially replace it completely.
But, in the meantime, what I'd love to do, is restart these every 24 hours a day. It's definitely not in any way perfect, but for what I'm currently doing, it would solve 99% of my issues.
In a perfect world, I'd have a list of processes, paired with a list of PIDs. Once every 24 hours, at a time I've proscribed, I want this controller to run through every one of the PIDs, send a regular kill signal (one of them talks to a DB, so I'd like it to close safely, if possible), then wait a proscribed period of time (say 30 seconds), then send a
sudo kill -9 signal to the PID for fatal crash situations, then run the startup command again.
I've looked into maybe using
Supervisor for this, but it didn't seem (at first skim) to do what I'm looking for.
I'm completely ok with maybe using 2-3 different programs for this (cron + supervisord + python script, for instance), instead of looking for one, if there's isn't the perfect single solution.