Been having a problem with application that keeps getting killed. This is running a RHEL5 server. We ran strace against the process and at the end of the trace file I get a bunch of +++ killed by SIGKILL +++ messages. SIGKILL can't be caught so I'm trying to find a way to track down why the process is getting killed.

I tried using some recommendations from Red Hat such as

auditctl -a exit,always -S kill 

which didn't seem to work. I didn't see any entries for kill in the audit.log.

Another recommendation from Red Hat was to use systemtap. I installed it via yum and tried to run stap with the sigmon.stp script that was provided. The sigmon.stp is on this page: Sigmon.stp I used the command provided in the comments section and here is the output:

# stap sigmon.stp -x 12647 SIGKILL -v
Pass 1: parsed user script and 85 library script(s) using 146868virt/23716res/3024shr/21404data kb, in 110usr/10sys/116real ms.
semantic error: while resolving probe point: identifier 'kernel' at /usr/share/systemtap/tapset/signal.stp:60:37
        source: probe __signal.send.send_sigqueue = kernel.function("send_sigqueue")
semantic error: missing x86_64 kernel/module debuginfo under '/lib/modules/2.6.18-308.13.1.el5/build'
semantic error: while resolving probe point: identifier '__signal' at :39:21
        source: probe signal.send = __signal.send.send_sigqueue,
semantic error: no match
semantic error: while resolving probe point: identifier 'signal' at sigmon.stp:30:7
        source: probe signal.send 
Pass 2: analyzed script: 1 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 150348virt/28920res/4696shr/24884data kb, in 30usr/40sys/81real ms.
Pass 2: analysis failed.  Try again with another '--vp 01' option.
Missing separate debuginfos, use: debuginfo-install kernel-2.6.18-308.13.1.el5.x86_64  

Has anyone had luck using the stap command? Is there another way I can find out what's killing my process?

  • Does your process use a LOT of memory? The kernel will kill processes using too much memory if they start affecting the performance of the system. – terdon Mar 27 '13 at 20:04
  • 1
    "What killed my process?" -> "killed by SIGKILL" – Diogo Mar 27 '13 at 20:11
  • 2
    Ah, but the intended question is probably "which program dispatched the sigkill?" – Hennes Mar 27 '13 at 20:19
  • Any clues in the system logs? – suspectus Mar 27 '13 at 22:04
  • The process is java based, so yes it uses a fair amount of memory. We have bumped up the memory and the issue remains. I've read about oom killer, think I'll try disabling the oom killer tomorrow to see if I can rule that out. But I wasn't able to find any trace of oom killer in the logs. And yes to Hennes, I am looking to narrow down who dispatched the sigkill. – zjherner Mar 27 '13 at 22:06

SystemTap looks like it can't find the debug symbols for your kernel. Have you installed the required kernel debug packages for the kernel you are running? The SystemTap guide says:

To deploy SystemTap, you need to install the SystemTap packages along with the corresponding set of -devel, -debuginfo and -debuginfo-common packages for your kernel. If your system has multiple kernels installed, and you wish to use SystemTap on more than one kernel kernel, you will need to install the -devel and -debuginfo packages for each of those kernel versions.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.