1

My scenario:

I've some python scripts on a raspberry pi. They behave like a service, they run forever without exiting and process signals. They are meant to run in the background, writing some values to a MariaDB. They run unattended, without a logged in user.

However, sometimes they crash. I'm not sure why and to me it looks like it's really hard to find out what happens (random crashes, no pattern, no clue, sometimes they run for months, sometimes they crash every day).

As the signals this scripts deal with are not really important, i'd love to just:

  1. write a shell script that kills all running python scripts and starts them again
  2. use crontab to run this script twice a day, so if something crashes i don't lose too much data and i don't have to care about checking and restarting them

I've written the shell script "restart.sh" and executing it in bash works great:

#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
kill $(pgrep python3)
sleep 2
nohup python3 script1.py &
sleep 2
nohup python3 script2.py &
sleep 2
nohup python3 script3.py &
sleep 2
exit

I've also modified my crontab with sudo crontab -e, containing this:

SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

0 9 * * * /home/pi/restart.sh
0 15 * * * /home/pi/restart.sh

My Problem:

Testing the crontab, i can see that the restart.sh is executed but only the first command, the kill command, works. All running python3 scripts are killed, but not started again.

I tried adding PATH and SHELL, as that seems to be common problems, but that didn't change anything.

I also tried first to edit crontab -e without sudo, same problem.

What am i missing here? Why aren't the nohup commands executed like they are if i run ./restart.sh manually in the terminal? What can i do to get this scripts started using crontab?

9
  • what's the purpose of sleep 2 along with &
    – alecxs
    Oct 28, 2021 at 19:59
  • @alecxs: & starts the script in background, nohup allone does not do that. sleep 2 is just a delay of 2 seconds between the commands - maybe not really needed, but as some of the scripts need a moment to fully launch on the raspberry, i thought it can't hurt.
    – xph
    Oct 28, 2021 at 20:05
  • Using & in a non-interactive script without wait is always wrong (in some sense). Another recommendation: Avoid outdated cron, it has no place in the 20s of the 21. century. Use systemd .timer units instead. They will give you precise success / failure tracking, proper log management and retention, configurable retries on failures, security hardening, handling of dependencies on (other) services and system states and much more. Oct 28, 2021 at 20:16
  • @AndrejPodzimek: How could wait be any help with & for a script that is intended to never finish? I might have a look at the mentioned systemd.timer_units, but honestly - i have no need for anything else than running a script twice a day, exactly what cron was made for. But if that helps to start this scripts it might be worth a try.
    – xph
    Oct 28, 2021 at 20:26
  • 1
    @AndrejPodzimek: You miss the point that i don't care why or when this python scripts crashes. They crash, it's fine. Yes, changing the complete method of running scripts might work. But i still believe what i'm trying to do here is not wanting to "exceed 60 km/h with my bike" and you can't provide any information to my origin question. You don't provide any helpful information at all on how one could start a script twice a day using your mentioned method. Maybe you want to write an answer that shows how systemd could be used for that? Would be more helpful than "you are doing this wrong".
    – xph
    Oct 29, 2021 at 4:09

1 Answer 1

1

It might be cleaner to use pkill rather than pgrep but I don't think that's the problem.

I'm not sure from the information you supply precisely why your script isn't working as expected (it looks OK to me).

However I'm not sure using '&' in scripts called from cron commands is the right thing to do. My first alternative suggestion would be to use the fact that cron natively runs things in the background, and have a separate cron job per python script rather than doing it all at once.

If I understand correctly, the symptom is that the python programs crash completely rather than hanging, and on the assumption it would be preferable to keep then running if they are happy, you could use a start-if-not-running script (I use bash):

#!/bin/bash
value=$( ps -ef | grep -ic "$1" )
if [ $value -lt 2 ]
then
    python3 "$1"
fi

and then in your crontab:

15 * * * * bash /path/to/script/start-if-not-running.bash script1.py
30 * * * * bash /path/to/script/start-if-not-running.bash script2.py
45 * * * * bash /path/to/script/start-if-not-running.bash script3.py

Explanation:

ps -ef searches the entire command line of the process table (pgrep and pkill by default only search the first 13 characters)

$value -lt 2 means if there are less than two matches then go ahead and start a new script (there will always be one match - that of the grep command itself)

I'd also strongly recommend making sure your raspberry pi emails you the outputs from cron commands if you haven't got that set up already (e.g. https://medium.com/swlh/setting-up-gmail-and-other-email-on-a-raspberry-pi-6f7e3ad3d0e) as an aid to debugging.

1
  • Thank you very much for your answer! To be more specific: I've seen both, python programs crash and hangs. It seems like they crash more often, but they could also be hanging - that's why i want to kill running ones and restart them. But meanwhile i have the problem that the sensor hardware and wiring, that delivers the signals for the python programs, is literally crashed, beause someone ripped all apart by accident... So if i find the time, i'll need to re-build it. And then your mentioned things will be super helpful! I'm sure this helps me a lot to get this working better. Thanks!
    – xph
    Dec 19, 2021 at 13:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.