0

I'm currently running a personal project that basically monitors a few things around the clock. One is a process that uses PhantomJS, and the other scans some other APIs, and sends emails if some certain situation occurs.

When all is up and running, it works exactly how I want, but unfortunately, at the moment, every 2-3 days, something occurs normally in the dead of night, and the API scanner just freezes. It's written in Python, and runs on a tick of every 3 seconds, and when I check the log next day, no crash, nothing looks out of the ordinary, but for some reason, it's stopped running.

When I've time, I will eventually get around to fixing that, but because it only occurs once after 3, or 4 days running, it's been very difficult to reproduce.

For the PhantomJS stuff... every now and again, that just blows up. Again, there's going to be cleanup happening there, fixing these issues, and potentially replace it completely.

But, in the meantime, what I'd love to do, is restart these every 24 hours a day. It's definitely not in any way perfect, but for what I'm currently doing, it would solve 99% of my issues.

In a perfect world, I'd have a list of processes, paired with a list of PIDs. Once every 24 hours, at a time I've proscribed, I want this controller to run through every one of the PIDs, send a regular kill signal (one of them talks to a DB, so I'd like it to close safely, if possible), then wait a proscribed period of time (say 30 seconds), then send a sudo kill -9 signal to the PID for fatal crash situations, then run the startup command again.

I've looked into maybe using Supervisor for this, but it didn't seem (at first skim) to do what I'm looking for.

I'm completely ok with maybe using 2-3 different programs for this (cron + supervisord + python script, for instance), instead of looking for one, if there's isn't the perfect single solution.

1

My preferred solution is a cron-driven python script which primarily takes care of starting the monitored processes when they are not running. It can itself obtain and store the child processes PIDs (and maybe other metadata if needed, for example a last OK iteration timestamp) for later use or just monitor such files that the child processes themselves create.

In the first iteration I'd take advantage of the relatively predictable failure patterns/timelines and modify the respective child processes to simply exit cleanly after a specified number of iterations or total runtime. The monitoring process would just restart them.

Down the road the monitoring script could be enhanced to do more detailed sanity analysis of the running monitored processes and kill them if needed as well. For example by checking their last OK iteration timestamp I was mentioning above. This would take care of less predictive failure patterns.

For the actual killing of more complex process structures you might want to check my answer to this question: https://stackoverflow.com/questions/30780487/python-script-to-monitor-process-and-sub-processes.

  • "specified number of iterations of total runtime" - Now that's something I just never thought to check at all. Will probably implement that straight away, before putting the full checker in place. The rest of your answer is definitely where my head was at, and I'll mark it correct if nothing better shows. Thanks Dan. – seaders Jul 6 '15 at 13:32
  • Ugh - I had a typo, should be "specified number of iterations or total runtime"... Whichever is more convenient. – Dan Cornilescu Jul 6 '15 at 13:35
  • Yer ok! I read it as "number of iterations in current runtime", which while may be implied anyway, still makes perfect sense. I already log current time, and total time running so this definitely is a useful one to check to see if there's a correlation. – seaders Jul 6 '15 at 13:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.