5

I want to detect when some process is stuck, but it doesn't give me any clues on the matter. All I know is that if it doesn't produce any output in a given timeout (let's say 30 seconds), it means that it's probably blocked.

I know the coreutils' timeout program, but it's based on the full program exit time, not on the last line of output time. I'd love if something like this worked:

timeout --stdout --stderr 30s my-program

So is there any way to do that? How can I do it?

  • I like the idea, but I can't point to an existing tool. This should not be hard to do per se. – tripleee Mar 21 '18 at 13:31
  • You may find this useful. – Wobbly Mar 21 '18 at 14:08
4

The code

Save this as tkill (make it executable and adjust your PATH if needed):

#!/bin/bash

_terminate_children() {
        trap "exit 143" SIGTERM && kill -- -$$
}

trap _terminate_children SIGINT SIGTERM

tout="$1"
shift
eval $@ | tee >(while :; do
   read -t "$tout"
   case $? in
      0) : ;;
      1) break ;;
      *) _terminate_children ;;
   esac
done)
exit ${PIPESTATUS[0]}

Basic usage

tkill 30 some_command

The first argument (30 here) is the timeout in seconds.


Notes

  • tkill expects some_command to generate text (not binary) output.
  • tkill probes stdout of the given command. To include stderr redirect it like in the last advanced example below.

Advanced usage

These are valid examples:

tkill 9 foo -option value
tkill 9 "foo -option value"  # equivalent to the above
tkill 5 "foo | bar"
tkill 5 'foo | bar'
tkill 5 'foo | bar | baz'    # tkill monitors baz
tkill 5 'foo | bar' | baz    # baz reads from tkill
tkill 3 "foo; bar"
tkill 6 "foo && bar || baz"
tkill 7 "some_command 2>&1"

Use Bash syntax in these quotes.


Exit status

  • If some_command exits by itself then its exit status will be reused as the exit status of tkill; tkill 5 true returns 0; tkill 5 false returns 1; tkill 5 "true; false" returns 1.
  • If the given timeout expires or tkill gets interrupted by SIGINT or SIGTERM then the exit status will be 143.

Fragments of code explained

  • eval makes the advanced examples possible.
  • tee allows us to analyze stdin while still passing a copy of it to stdout.
  • read -t is responsible for applying the timeout, its exit status is used to determine what to do next.
  • Command(s) being monitored are killed when needed with this solution.
  • Exit status of monitored command(s) is retrieved with this solution.
| improve this answer | |
0

So, basically something like this:

#!/bin/bash
tmp1=/tmp/tmp-$$-1
tmp2=/tmp/tmp-$$-2
touch $tmp1
touch $tmp2

time_out=30

typeset -i i

i=0
my-program > $tmp1 &
pgmpid=$!

while ps $pgmpid > /dev/null ; do
    sleep 1
    if diff $tmp1 $tmp2 > /dev/null ; then
        i=i+1
        if [ $i -gt $time_out ] ; then
            kill $pgmpid
        fi
    else
        i=0
        cp $tmp1 $tmp2 
    fi
done

rm -f $tmp1 $tmp2
| improve this answer | |
  • This should do the trick, although I was expecting some kinda one-liner TBH... Thanks anyway! – Yajo Mar 25 '18 at 19:22
0

Run the program in the background while copying the output to a file. After 30 seconds, if the file is empty, terminate the program, else bring it back in the foreground.

my-program | tee temp-file & 
sleep 30
[ -s temp-file ] && kill $! || fg $!
| improve this answer | |
  • With this, if after 30 seconds it becomes frozen, you're outta luck, right? – Yajo Mar 25 '18 at 19:21
  • Yeah. Seems I misunderstood the question. – Lubo Kanev Mar 26 '18 at 19:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.