Been having a problem with application that keeps getting killed. This is running a RHEL5 server. We ran strace against the process and at the end of the trace file I get a bunch of +++ killed by SIGKILL +++ messages. SIGKILL can't be caught so I'm trying to find a way to track down why the process is getting killed.
I tried using some recommendations from Red Hat such as
auditctl -a exit,always -S kill
which didn't seem to work. I didn't see any entries for kill in the audit.log.
Another recommendation from Red Hat was to use systemtap. I installed it via yum and tried to run stap with the sigmon.stp script that was provided. The sigmon.stp is on this page: Sigmon.stp I used the command provided in the comments section and here is the output:
# stap sigmon.stp -x 12647 SIGKILL -v
Pass 1: parsed user script and 85 library script(s) using 146868virt/23716res/3024shr/21404data kb, in 110usr/10sys/116real ms.
semantic error: while resolving probe point: identifier 'kernel' at /usr/share/systemtap/tapset/signal.stp:60:37
source: probe __signal.send.send_sigqueue = kernel.function("send_sigqueue")
^
semantic error: missing x86_64 kernel/module debuginfo under '/lib/modules/2.6.18-308.13.1.el5/build'
semantic error: while resolving probe point: identifier '__signal' at :39:21
source: probe signal.send = __signal.send.send_sigqueue,
^
semantic error: no match
semantic error: while resolving probe point: identifier 'signal' at sigmon.stp:30:7
source: probe signal.send
^
Pass 2: analyzed script: 1 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 150348virt/28920res/4696shr/24884data kb, in 30usr/40sys/81real ms.
Pass 2: analysis failed. Try again with another '--vp 01' option.
Missing separate debuginfos, use: debuginfo-install kernel-2.6.18-308.13.1.el5.x86_64
Has anyone had luck using the stap command? Is there another way I can find out what's killing my process?