Many cron -f processes on unresponsive server

Question

My Debian server had again become unresponsive and needed to be restartet (SSH login impossible, no longer transmitted log data to another server's MySQL Database, no new FTP connections possible, but the Wowza Mediaserver continues to run and stream video without problems).

However, I was still logged in through an existing SSH connection, and able to enter shell commands. When entering "ps auxf", I had this output (OUTPUT 1):

 USER       PID %CPU %MEM    VSZ   RSS TTY  STAT START   TIME COMMAND
 root     14994  0.0  0.0  71200  3520 ?        S    00:01   0:00  \_/usr/sbin/CRON -f
 root     15007  0.0  0.0   4328   748 ?        Ss   00:01   0:00  |   \_ /bin/sh -c /usr/sbin/iotop --accumulated --batch --delay=86100 --proc
 root     15014  0.0  0.0  60208 17548 ?        S    00:01   0:00  |       \_ /usr/bin/python /usr/sbin/iotop --accumulated --batch --delay=861
 root     18667  0.0  0.0  71092  3504 ?        D    08:50   0:00  \_ /usr/sbin/CRON -f
 root     18668  0.0  0.0  71092  3472 ?        D    08:50   0:00  \_ /usr/sbin/CRON -f
 root     18669  0.0  0.0  71092  3472 ?        D    08:50   0:00  \_ /usr/sbin/CRON -f
 root     18670  0.0  0.0  71092  3428 ?        D    08:50   0:00  \_ /usr/sbin/CRON -f
 root     18671  0.0  0.0  71200  3520 ?        D    08:50   0:00  \_ /usr/sbin/CRON -f

And dozens more like the last few lines. Also, when entering ps after 15 minutes, the all of the old CRON -f entries (e.g. PID 18667, start time 08:50) still showed up as running processes.

Obviously, many cronjobs were being run, but only the first cronjob (which exists in my crontab) has a script contents (/usr/sbin/iotop). The others have no command, and are in inactive mode (state "D", versus the first one having state "S").

My server runs about 5-6 PHP script cronjobs once a minute, most of which complete within about 15-40 seconds and are not resource intensive. Their ps aux output looks like this (OUTPUT 2):

 root     19024  0.0  0.0  42224  2644 ?        S    09:03   0:00  \_ /usr/sbin/CRON -f
 root     19026  0.0  0.0   4328   712 ?        Ss   09:03   0:00  |   \_ /bin/sh -c /usr/bin/php -q /usr/local/script.php

Therefore, I tentatively conclude that the first output (OUTPUT 1) does not show that my PHP scripts became unresponsive, accumulated massively, and caused problems. However, the "ps auxf" output in OUTPUT 1 showed about 5-6 new CRON -f entries per minute, all of which continued to run, which may refer to cronjob attempts to run my scripts, but not being able to run them and then getting stuck, producing this massive ps auxf output?

Questions:

1. What is the meaning of those dozens of CRON -f lines? What do they refer to?

2. Can this ps auxf output explain anything about why my server became unresponsive?

Answer 1

For me this turns out to be due to something broken in the email system, so all those cron processes were waiting to send their results. I restarted postfix and it stopped happening.