7

Been having a problem with application that keeps getting killed. This is running a RHEL5 server. We ran strace against the process and at the end of the trace file I get a bunch of +++ killed by SIGKILL +++ messages. SIGKILL can't be caught so I'm trying to find a way to track down why the process is getting killed.

I tried using some recommendations from Red Hat such as

auditctl -a exit,always -S kill 

which didn't seem to work. I didn't see any entries for kill in the audit.log.

Another recommendation from Red Hat was to use systemtap. I installed it via yum and tried to run stap with the sigmon.stp script that was provided. The sigmon.stp is on this page: Sigmon.stp I used the command provided in the comments section and here is the output:

# stap sigmon.stp -x 12647 SIGKILL -v
Pass 1: parsed user script and 85 library script(s) using 146868virt/23716res/3024shr/21404data kb, in 110usr/10sys/116real ms.
semantic error: while resolving probe point: identifier 'kernel' at /usr/share/systemtap/tapset/signal.stp:60:37
        source: probe __signal.send.send_sigqueue = kernel.function("send_sigqueue")
                                                    ^
semantic error: missing x86_64 kernel/module debuginfo under '/lib/modules/2.6.18-308.13.1.el5/build'
semantic error: while resolving probe point: identifier '__signal' at :39:21
        source: probe signal.send = __signal.send.send_sigqueue,
                                    ^
semantic error: no match
semantic error: while resolving probe point: identifier 'signal' at sigmon.stp:30:7
        source: probe signal.send 
                      ^
Pass 2: analyzed script: 1 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 150348virt/28920res/4696shr/24884data kb, in 30usr/40sys/81real ms.
Pass 2: analysis failed.  Try again with another '--vp 01' option.
Missing separate debuginfos, use: debuginfo-install kernel-2.6.18-308.13.1.el5.x86_64  

Has anyone had luck using the stap command? Is there another way I can find out what's killing my process?

6
  • Does your process use a LOT of memory? The kernel will kill processes using too much memory if they start affecting the performance of the system.
    – terdon
    Mar 27, 2013 at 20:04
  • 1
    "What killed my process?" -> "killed by SIGKILL"
    – Diogo
    Mar 27, 2013 at 20:11
  • 2
    Ah, but the intended question is probably "which program dispatched the sigkill?"
    – Hennes
    Mar 27, 2013 at 20:19
  • Any clues in the system logs?
    – suspectus
    Mar 27, 2013 at 22:04
  • The process is java based, so yes it uses a fair amount of memory. We have bumped up the memory and the issue remains. I've read about oom killer, think I'll try disabling the oom killer tomorrow to see if I can rule that out. But I wasn't able to find any trace of oom killer in the logs. And yes to Hennes, I am looking to narrow down who dispatched the sigkill.
    – zjherner
    Mar 27, 2013 at 22:06

1 Answer 1

2

SystemTap looks like it can't find the debug symbols for your kernel. Have you installed the required kernel debug packages for the kernel you are running? The SystemTap guide says:

To deploy SystemTap, you need to install the SystemTap packages along with the corresponding set of -devel, -debuginfo and -debuginfo-common packages for your kernel. If your system has multiple kernels installed, and you wish to use SystemTap on more than one kernel kernel, you will need to install the -devel and -debuginfo packages for each of those kernel versions.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .