How to kill a program if it did not produce any output in a given timeout?

Question

I want to detect when some process is stuck, but it doesn't give me any clues on the matter. All I know is that if it doesn't produce any output in a given timeout (let's say 30 seconds), it means that it's probably blocked.

I know the coreutils' timeout program, but it's based on the full program exit time, not on the last line of output time. I'd love if something like this worked:

timeout --stdout --stderr 30s my-program

So is there any way to do that? How can I do it?

Answer 1

The code

Save this as tkill (make it executable and adjust your PATH if needed):

#!/bin/bash

_terminate_children() {
        trap "exit 143" SIGTERM && kill -- "-$$"
}

trap _terminate_children SIGINT SIGTERM

tout="$1"
shift
eval "$@" | tee >(while :; do
   read -t "$tout"
   case $? in
      0) : ;;
      1) break ;;
      *) _terminate_children ;;
   esac
done)
exit "${PIPESTATUS[0]}"

---

Basic usage

tkill 30 some_command

The first argument (30 here) is the timeout in seconds.

---

Notes

• tkill expects some_command to generate text (not binary) output.
• tkill probes stdout of the given command. To include stderr redirect it like in the last advanced example below.

---

Advanced usage

These are valid examples:

tkill 9 foo -option value
tkill 9 "foo -option value"  # equivalent to the above
tkill 5 "foo | bar"
tkill 5 'foo | bar'
tkill 5 'foo | bar | baz'    # tkill monitors baz
tkill 5 'foo | bar' | baz    # baz reads from tkill
tkill 3 "foo; bar"
tkill 6 "foo && bar || baz"
tkill 7 "some_command 2>&1"

Use Bash syntax in these quotes.

---

Exit status

• If some_command exits by itself then its exit status will be reused as the exit status of tkill; tkill 5 true returns 0; tkill 5 false returns 1; tkill 5 "true; false" returns 1.
• If the given timeout expires or tkill gets interrupted by SIGINT or SIGTERM then the exit status will be 143.

---

Fragments of code explained

• eval makes the advanced examples possible.
• tee allows us to analyze stdin while still passing a copy of it to stdout.
• read -t is responsible for applying the timeout, its exit status is used to determine what to do next.
• Command(s) being monitored are killed when needed with this solution.
• Exit status of monitored command(s) is retrieved with this solution.

---

Quirks

• eval makes the advanced examples possible but you need to remember it does this by evaluating its arguments. Example (somewhat artificial): if you had a file literally named |, then tkill 9 ls * would expand * in the current shell, | would appear as an argument to eval and it would be interpreted as a pipe operator. In this case tkill 9 'ls *' is better (but note it expands nothing in the current shell). It's similar with watch (I mean e.g. watch ls * vs watch 'ls *').
• The evaluated command gets piped to tee, it does not write directly to the terminal. Some commands alter their behavior depending on whether their stdout is a terminal or not. They may colorize and/or columnize their output to a terminal, but not their output to a regular file or a pipe. E.g. ls --color=auto and tkill 9 'ls --color=auto' give different output.

Answer 2

So, basically something like this:

#!/bin/bash
tmp1=/tmp/tmp-$$-1
tmp2=/tmp/tmp-$$-2
touch $tmp1
touch $tmp2

time_out=30

typeset -i i

i=0
my-program > $tmp1 &
pgmpid=$!

while ps $pgmpid > /dev/null ; do
    sleep 1
    if diff $tmp1 $tmp2 > /dev/null ; then
        i=i+1
        if [ $i -gt $time_out ] ; then
            kill $pgmpid
        fi
    else
        i=0
        cp $tmp1 $tmp2 
    fi
done

rm -f $tmp1 $tmp2

Answer 3

Run the program in the background while copying the output to a file. After 30 seconds, if the file is empty, terminate the program, else bring it back in the foreground.

my-program | tee temp-file & 
sleep 30
[ -s temp-file ] && kill $! || fg $!