Who's crashing my kernel?

Question

Symptoms

Suddenly, a message saying "A problem in the kernel package has been detected" started appearing after I login upon boot. A new message is shown every second, incessantly (translation below).

Translation:

A problem was reported

A problem in the kernel package has been detected

I don't know what's causing these messages to appear. How I get to obtain details about system crashes?

Specs

I haven't updated the kernel in the last 12 days (3.19.7-200.fc21.x86_64). Booting from an older kernel doesn't stop the warnings.

I have installed 5 new packages today: subversion-1.8.11-1.fc21.x86_64, gitk-2.1.0-4.fc21.noarch, git-gui-2.1.0-4.fc21.noarch, subversion-libs-1.8.11-1.fc21.x86_64 and libserf-1.3.7-2.fc21.x86_64

I have installed a few gnome extensions but I used them for a few hours with no problem before rebooting. I disabled the extensions and the problem persists.

What I tried

I believe these notification messages are part of abrt. But when I tried to obtain more details, abrt-cli list doesn't show anything for the current month.

dmesg doesn't show anything suspicious (or maybe I'm misiterpreting it. I'll post a log).

As suggested on a comment, I checked /var/log/messages, /var/log/syslog, and /var/log/kern.log:

The latter two aren't present. tail /var/log/messages contains a lot (more than a thousand) of the following, repeated over and over again (with different timestamps):

May 26 16:39:28 [hostname] abrt-dump-journal-oops: Reported 1 kernel oopses to Abrt
May 26 16:39:30 [hostname] abrt-dump-journal-oops: abrt-dump-journal-oops: Found oopses: 1
May 26 16:39:30 [hostname] abrt-dump-journal-oops: abrt-dump-journal-oops: Creating problem directories
May 26 16:39:30 [hostname] abrt-server: Deleting problem directory oops-2015-05-26-16:39:30-585-0 (dup of oops-2015-04-28-15:49:00-21380-1)
May 26 16:39:30 [hostname] gnome-session: abrt-applet: repeated problem in kernel, not showing the notification

The problem detected at 2015-04-28-15:49:00 via abrt-cli list is:

id dadaa8ca8525cf44b21c438b086cc731ac73c2cd
reason:         WARNING: CPU: 0 PID: 21350 at fs/block_dev.c:67 bdev_inode_switch_bdi+0x87/0x90()
time:           Ter 28 Abr 2015 15:49:02 BRT
cmdline:        BOOT_IMAGE=/boot/vmlinuz-3.19.3-100.fc20.x86_64 root=UUID=45f0c704-ada0-411d-95ba-50169ce0994a ro rd.md=0 rd.lvm=0 rd.dm=0 rd.luks=0 vconsole$
package:        kernel
count:          1529
Directory:  /var/tmp/abrt/oops-2015-04-28-15:49:00-21380-1
Relatado:   https://retrace.fedoraproject.org/faf/reports/bthash/392cacbf6958e88053298dbce758bf6865c4db3f

Answer 1

First of all, your kernel is not crashing. If it were crashing, your system would completely freeze up and you would be unable to use it.

There are several types of problems that can occur in the kernel.

• A warning (WARN), bug (BUG) or OOPS can occur when the kernel's built-in self-checks detect a situation that might lead to system instability, or loss of data in the future. However, generally speaking, these issues do not cause an (immediate) crash of the system. Generally, OOPSes are the most severe of these, and will lead to any associated userspace processes receiving a SIGKILL ("thou shalt die", not "please go away") signal from the kernel.

• A panic is where the system is so hosed that it refuses to go on. This is where the kernel simply stops executing (after printing a stack trace, if it's able to do so), and yields control to.... nothing. Usually. Although if you have a crash kernel, sometimes the broken kernel will attempt to load a second kernel whose purpose is to gather information on the cause of the crash, and try to write it to disk. In general it is not possible for even a very robust crash kernel to fully recover the state of the system to be usable and stable again without rebooting.

In my mind, a crash is synonymous with a panic. There are many situations where a WARN or BUG can be safely ignored, with a very low probability of data loss. If your system continues to run after these "problems" are being reported, it is almost definitely not a panic.

You haven't given me enough of your logs (particularly dmesg) for me to be able to tell the reason for this particular crash, but in general, when the kernel itself is reporting problems, it will manifest in the dmesg kernel ring buffer. Literally execute the command dmesg on the console to view the kernel ring buffer.

It seems like, in your case, you might have experienced a one-time oops that is being incorrectly handled by the abrt crash event notification system (or the GNOME user interface infrastructure that displays it to you).

May 26 16:39:30 segtic-1c505e gnome-session: abrt-applet: repeated problem in kernel, not showing the notification

So it thinks it's not showing it to you because it's a repeated problem, yet it continues to bombard you with the same error. So, either abrt-applet thinks it's not bombarding you but is actually doing so anyway, or there's another program that handles kernel errors (maybe a different applet that also deals with abrt?) that does not detect repeated issues and is slamming you with the same one over and over.

So there are several problems here:

1. You haven't given me any dmesg logs that indicate a repeated problem. The ACPI thing you showed might be the source of one error, but that happened very early on in boot, and it isn't happening again and again.

2. The error reporting infrastructure seems broken. I think, at some level, abrt knows that it's a repeated message for the same event (or a sequence of independent events that are identical in cause), but somehow the notifications are getting through the system and to your UI anyway.

3. Obviously, it is an issue that you are having some kind of crash or OOPS or BUG or WARN related to the Linux kernel in the first place. But since the kernel logs you've posted have been minimal and not especially concerning, the root of the problem seems elusive right now. If it is complaining about that ACPI issue from early boot, it should really learn to shut up; the fact is that motherboard ACPI DSDTs are almost always horribly mangled and broken, and the OS just needs to learn to deal with that the best it can. There's nothing you can do about it as an end-user. It's not like your mobo manufacturer would still be releasing BIOS updates to try and improve their DSDT correctness (well, it's pretty unlikely that they would, anyway).

Or maybe the issue is completely unrelated to ACPI and the actual problem report just isn't making it to the kernel ringbuffer. This would be quite a strange one indeed, and not something I've experienced before. For that matter, I'm not sure what mechanism abrt is using to detect the existence of the error if it's not parsing dmesg.

When it comes to Linux kernel issues and the way they're being reported in the UI, there's rarely an easy path to diagnosing it. It's the nature of the beast.