This now appears to be a bug with the combination of MD and XFS after finding similar kernel panic reports elsewhere. A bug report with Ubuntu has been filed here ( https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1929591 ) but there's been no response on that so I'll probably just give up and move on at this point -- Back up all my data, format the drives, and replace it with a ZFS array, completely ditching MD and XFS.
Update 1/14/21 and 6/16/21 below. Original post:
I recently upgraded my NAS server from an old i7-2600K to a Ryzen 3700X. Both systems use software MD RAID 6 on 7 4TB HDDs, running Ubuntu 20.04. The MD RAID was transferred as-is to a fresh Ubuntu install. However, I am having an issue where this new build is periodically rebooting, approximately every 2-4 days.
I had the exact same issue on my new VM server I built at the same time (using a 5950X and running Proxmox), but that was fixed with disabling C-States in BIOS and adding processor.max_cstate=5 rcu_nocbs=0-31 to the kernel args -- it currently has an uptime of 20 days which I am thrilled about. Doing the same thing on this machine did not fix the issue however.
I was not able to ever catch the kernel panic text until I setup a serial console server on a Raspberry Pi. It appears to be a recursive kernel panic which fills up any crash logs, which explains why my attempts at setting up kdump / linux-crashdump were never successful and always hard-locked with an out-of-memory error.
The kernel panic appears to be a bug in the MD driver of Linux. But I have no idea why it wasn't occurring on the same software on my old Intel hardware.
I have tried 2 different sets of RAM (and memtest86 reports both sets are OK), and many BIOS versions -- makes no difference. I originally was using Ubuntu 20.04 because I always use LTS, but decided to reinstall with Ubuntu 20.10 for a slightly newer kernel version (5.8) to see if that fixed it -- also same error.
Specs:
- CPU: AMD Ryzen 7 3700X
- Mobo: ASRock X570 Pro4 ATX
- Various BIOS versions attempted, inc. latest 3.60 down to 2.20
- RAM: GSkill Ripjaws V 32 GB DDR4-3200 F4-3200C16D-32G
- Also used older set of Mushkin Enhanced Blackline 32GB DDR4-2400
- GPU: Zotac Geforce GTX 1650 OC SFF
- SSD: Inland Premium 1TB M.2
- HDD: 7x Western Digital Red (Plus) 4TB WD40EFRX
Running on this machine is Docker, including Plex, which is using the Nvidia Docker extensions with the proprietary 450.80 driver for hardware encoding (NVENC). I don't think the Nvidia driver is affecting this?
Some info:
$ lsb_release -ds
Ubuntu 20.10
$ uname -a
Linux babylon 5.8.0-33-generic #36-Ubuntu SMP Wed Dec 9 09:14:40 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$
$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.8.0-33-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro console=tty1 console=ttyS0,115200 processor.max_cstate=1 rcu_nocbs=0-15
$
$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid6 sdc[7] sda[1] sdb[2] sde[3] sdd[4] sdg[6] sdf[5]
19534435840 blocks super 1.2 level 6, 512k chunk, algorithm 2 [7/7] [UUUUUUU]
bitmap: 4/30 pages [16KB], 65536KB chunk
unused devices: <none>
$
$ sudo mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Sat Jul 7 18:47:41 2018
Raid Level : raid6
Array Size : 19534435840 (18629.49 GiB 20003.26 GB)
Used Dev Size : 3906887168 (3725.90 GiB 4000.65 GB)
Raid Devices : 7
Total Devices : 7
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Tue Jan 5 22:26:49 2021
State : clean
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : bitmap
Name : america:0
UUID : 279552e4:7db4a375:40a2a9bd:53f3c643
Events : 34528
Number Major Minor RaidDevice State
7 8 32 0 active sync /dev/sdc
1 8 0 1 active sync /dev/sda
2 8 16 2 active sync /dev/sdb
3 8 64 3 active sync /dev/sde
4 8 48 4 active sync /dev/sdd
5 8 80 5 active sync /dev/sdf
6 8 96 6 active sync /dev/sdg
$
$ nvidia-smi
Tue Jan 5 22:28:23 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GTX 1650 Off | 00000000:08:00.0 Off | N/A |
| 50% 33C P0 N/A / 75W | 0MiB / 3911MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
Serial Console Log at point of panic:
[161342.301305] BUG: stack guard page was hit at 00000000fc60f228 (stack is 00000000875efe77..000000003f38a379)
[161342.301306] kernel stack overflow (double-fault): 0000 [#1] SMP NOPTI
[161342.301306] CPU: 10 PID: 465 Comm: md0_raid6 Tainted: P OE 5.8.0-33-generic #36-Ubuntu
[161342.301307] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X570 Pro4, BIOS P3.60 12/01/2020
[161342.301307] RIP: 0010:slab_free_freelist_hook+0x35/0x120
[161342.301308] Code: 89 d5 41 54 49 89 f4 53 48 89 fb 48 83 ec 08 48 8b 02 4c 8b 36 48 c7 06 00 00 00 00 48 c7 02 00 00 00 00 48 85 c0 49 0f 44 c6 <48> 89 45 d0 eb 06 4c 3b 7d d0 74 5d 8b 53 20 4d 89 f7 49 8d 34 16
[161342.301308] RSP: 0018:ffffa86b00c6fff8 EFLAGS: 00010246
[161342.301309] RAX: ffff98edc21cac40 RBX: ffff98ef0b407800 RCX: 0000000000000001
[161342.301310] RDX: ffffa86b00c70040 RSI: ffffa86b00c70038 RDI: ffff98ef0b407800
[161342.301310] RBP: ffffa86b00c70028 R08: 0000000000000001 R09: ffffffff85854500
[161342.301311] R10: ffff98edc21ca100 R11: 0000000000000001 R12: ffffa86b00c70038
[161342.301311] R13: ffffa86b00c70040 R14: ffff98edc21cac40 R15: ffff98e9b53d74d8
[161342.301311] FS: 0000000000000000(0000) GS:ffff98ef0ec80000(0000) knlGS:0000000000000000
[161342.301312] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[161342.301312] CR2: ffffa86b00c6ffe8 CR3: 00000007fa766000 CR4: 0000000000340ee0
[161342.301312] Call Trace:
[161342.301313] ? mempool_kfree+0xe/0x10
[161342.301313] ? kfree+0xb8/0x220
[161342.301313] ? mempool_kfree+0xe/0x10
[161342.301313] ? mempool_free+0x2f/0x80
[161342.301314] ? md_end_io+0x4b/0x70
[161342.301314] ? bio_endio+0xe6/0x150
[161342.301314] ? bio_chain_endio+0x2d/0x40
[161342.301315] ? md_end_io+0x5d/0x70
[161342.301315] ? bio_endio+0xe6/0x150
[161342.301315] ? bio_chain_endio+0x2d/0x40
[161342.301315] ? md_end_io+0x5d/0x70
[161342.301316] ? bio_endio+0xe6/0x150
[161342.301316] ? bio_chain_endio+0x2d/0x40
[161342.301316] ? md_end_io+0x5d/0x70
[161342.301316] ? bio_endio+0xe6/0x150
[161342.301317] ? bio_chain_endio+0x2d/0x40
[161342.301317] ? md_end_io+0x5d/0x70
[161342.301317] ? bio_endio+0xe6/0x150
[161342.301317] ? bio_chain_endio+0x2d/0x40
...
[161342.301379] ? md_end_io+0x5d/0x70
[161342.301379] ? bio_endio+0xe6/0x150
[161342.301380] ? bio_chain_endio+0x2d/0x40
[161342.301380] ? md_end_io+0x5d/0x70
[161342.301380] ? bio_endio+0xe6/0x150
[161342.301380] ? bio_ch
[161342.301381] Lost 296 message(s)!
[ 0.000000] Linux version 5.8.0-33-generic (buildd@lgw01-amd64-036) (gcc (Ubuntu 10.2.0-13ubuntu1) 10.2.0, GNU ld (GNU Binutils for Ubuntu) 2.35.1) #36-Ubuntu SMP Wed Dec 9 09:14:40 UTC 2020 (Ubuntu 5.8.0-33.36-generic 5.8.17)
I can reproduce this exact same kernel panic with RIP: 0010:slab_free_freelist_hook+0x35/0x120 every time it reboots now. They all look exactly the same.
I'm at a loss as what to do at this point. I have a replacement motherboard coming (a Gigabyte B550M Aorus Pro) along with a LSI 9211-8i that I will instead connect the HDDs to. Maybe that will fix it? Is there something else in software/BIOS that I can do?
UPDATE 1/14/2021:
I swapped out the ASRock X570 Pro4 motherboard for the Gigabyte B550M Aorus Pro, as well as moving all the HDDs to an LSI 9211-8ik PCIe card. However, after some promising looking uptime, another similar kernel panic :(
[406005.583315] BUG: stack guard page was hit at 000000007cbff150 (stack is 000000003b7072a2..00000000dac5ed08)
[406005.583315] kernel stack overflow (double-fault): 0000 [#1] SMP NOPTI
[406005.583315] CPU: 15 PID: 514 Comm: md0_raid6 Tainted: P OE 5.8.0-36-generic #40-Ubuntu
[406005.583316] Hardware name: Gigabyte Technology Co., Ltd. B550M AORUS PRO/B550M AORUS PRO, BIOS F1 05/19/2020
[406005.583316] RIP: 0010:slab_free_freelist_hook+0x35/0x120
[406005.583316] Code: 89 d5 41 54 49 89 f4 53 48 89 fb 48 83 ec 08 48 8b 02 4c 8b 36 48 c7 06 00 00 00 00 48 c7 02 00 00 00 00 48 85 c0 49 0f 44 c6 <48> 89 45 d0 eb 06 4c 3b 7d d0 74 5d 8b 53 20 4d 89 f7 49 8d 34 16
[406005.583316] RSP: 0018:ffffa620c06e3ff8 EFLAGS: 00010246
[406005.583317] RAX: ffff9aaf36f54720 RBX: ffff9ab34b407800 RCX: 0000000000000001
[406005.583317] RDX: ffffa620c06e4040 RSI: ffffa620c06e4038 RDI: ffff9ab34b407800
[406005.583317] RBP: ffffa620c06e4028 R08: 0000000000000001 R09: ffffffffb9c54500
[406005.583318] R10: ffff9aaf36f54fe0 R11: 0000000000000001 R12: ffffa620c06e4038
[406005.583318] R13: ffffa620c06e4040 R14: ffff9aaf36f54720 R15: ffff9ab2925cbd10
[406005.583318] FS: 0000000000000000(0000) GS:ffff9ab34edc0000(0000) knlGS:0000000000000000
[406005.583318] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[406005.583318] CR2: ffffa620c06e3fe8 CR3: 00000005d52ac000 CR4: 0000000000340ee0
[406005.583319] Call Trace:
[406005.583319] ? mempool_kfree+0xe/0x10
[406005.583319] ? kfree+0xb8/0x220
[406005.583319] ? mempool_kfree+0xe/0x10
[406005.583319] ? mempool_free+0x2f/0x80
[406005.583319] ? md_end_io+0x4b/0x70
[406005.583319] ? bio_endio+0xe6/0x150
Anyone have any suggestions?
Thanks!
UPDATE 6/16/2021:
So yeah this is totally still happening. Another serial console capture:
babylon login: [1457468.880947] BUG: stack guard page was hit at 000000007aef1a4a (stack is 00000000af9c61cd..000000007ccda653)
[1457468.880948] kernel stack overflow (double-fault): 0000 [#1] SMP NOPTI
[1457468.880948] CPU: 3 PID: 512 Comm: md0_raid6 Tainted: P OE 5.8.0-53-generic #60-Ubuntu
[1457468.880949] Hardware name: Gigabyte Technology Co., Ltd. B550M AORUS PRO/B550M AORUS PRO, BIOS F13h 04/23/2021
[1457468.880949] RIP: 0010:slab_free_freelist_hook+0x35/0x120
[1457468.880950] Code: 89 d5 41 54 49 89 f4 53 48 89 fb 48 83 ec 08 48 8b 02 4c 8b 36 48 c7 06 00 00 00 00 48 c7 02 00 00 00 00 48 85 c0 49 0f 44 c6 <48> 89 45 d0 eb 06 4c 3b 7d d0 74 5d 8b 53 20 4d 89 f7 49 8d 34 16
[1457468.880951] RSP: 0018:ffffbcda805efff8 EFLAGS: 00010246
[1457468.880952] RAX: ffff9bfb8ccc42a0 RBX: ffff9bfcdb407800 RCX: 0000000000000001
[1457468.880952] RDX: ffffbcda805f0040 RSI: ffffbcda805f0038 RDI: ffff9bfcdb407800
[1457468.880953] RBP: ffffbcda805f0028 R08: 0000000000000001 R09: ffffffff90841600
[1457468.880953] R10: ffff9bfb8ccc4f40 R11: 0000000000000001 R12: ffffbcda805f0038
[1457468.880953] R13: ffffbcda805f0040 R14: ffff9bfb8ccc42a0 R15: ffff9bf766967940
[1457468.880954] FS: 0000000000000000(0000) GS:ffff9bfcdeac0000(0000) knlGS:0000000000000000
[1457468.880954] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1457468.880955] CR2: ffffbcda805effe8 CR3: 00000003a65ee000 CR4: 0000000000340ee0
[1457468.880955] Call Trace:
[1457468.880955] ? mempool_kfree+0xe/0x10
[1457468.880956] ? kfree+0xb8/0x220
[1457468.880956] ? mempool_kfree+0xe/0x10
[1457468.880956] ? mempool_free+0x2f/0x80
[1457468.880956] ? md_end_io+0x4b/0x70
[1457468.880957] ? bio_endio+0xe6/0x150
[1457468.880957] ? bio_chain_endio+0x2d/0x40
[1457468.880957] ? md_end_io+0x5d/0x70
[1457468.880958] ? bio_endio+0xe6/0x150
[1457468.880958] ? bio_chain_endio+0x2d/0x40
[1457468.880958] ? md_end_io+0x5d/0x70
[1457468.880959] ? bio_endio+0xe6/0x150
[1457468.880959] ? bio_chain_endio+0x2d/0x40
[1457468.880959] ? md_end_io+0x5d/0x70
[1457468.880959] ? bio_endio+0xe6/0x150
[1457468.880960] ? bio_chain_endio+0x2d/0x40
[1457468.880960] ? md_end_io+0x5d/0x70
[1457468.880960] ? bio_endio+0xe6/0x150
[1457468.880960] ? bio_chain_endio+0x2d/0x40
[1457468.880961] ? md_end_io+0x5d/0x70
[1457468.880961] ? bio_endio+0xe6/0x150
[1457468.880961] ? bio_chain_endio+0x2d/0x40
[1457468.880962] ? md_end_io+0x5d/0x70
[1457468.880962] ? bio_endio+0xe6/0x150
[1457468.880962] ? bio_chain_endio+0x2d/0x40
[1457468.880962] ? md_end_io+0x5d/0x70
[1457468.880963] ? bio_endio+0xe6/0x150
[1457468.880963] ? bio_chain_endio+0x2d/0x40
[1457468.880963] ? md_end_io+0x5d/0x70
[1457468.880963] ? bio_endio+0xe6/0x150
[1457468.880964] ? bio_chain_endio+0x2d/0x40
[1457468.880964] ? md_end_io+0x5d/0x70
[1457468.880964] ? bio_endio+0xe6/0x150
[1457468.880965] ? bio_chain_endio+0x2d/0x40
[1457468.880965] ? md_end_io+0x5d/0x70
[1457468.880965] ? bio_endio+0xe6/0x150
[1457468.880965] ? bio_chain_endio+0x2d/0x40
[1457468.880966] ? md_end_io+0x5d/0x70
[1457468.880966] ? bio_endio+0xe6/0x150
[1457468.880966] ? bio_chain_endio+0x2d/0x40
[1457468.880966] ? md_end_io+0x5d/0x70
[1457468.880967] ? bio_endio+0xe6/0x150
[1457468.880967] ? bio_chain_endio+0x2d/0x40
[1457468.880967] ? md_end_io+0x5d/0x70
[1457468.880968] ? bio_endio+0xe6/0x150
[1457468.880968] ? bio_chain_endio+0x2d/0x40
[1457468.880968] ? md_end_io+0x5d/0x70
[1457468.880968] ? bio_endio+0xe6/0x150
[1457468.880969] ? bio_chain_endio+0x2d/0x40
[1457468.880969] ? md_end_io+0x5d/0x70
[1457468.880969] ? bio_endio+0xe6/0x150
[1457468.880969] ? bio_chain_endio+0x2d/0x40
[1457468.880970] ? md_end_io+0x5d/0x70
[1457468.880970] ? bio_endio+0xe6/0x150
[1457468.880970] ? bio_chain_endio+0x2d/0x40
[1457468.880971] ? md_end_io+0x5d/0x70
[1457468.880971] ? bio_endio+0xe6/0x150
[1457468.880971] ? bio_chain_endio+0x2d/0x40
[1457468.880971] ? md_end_io+0x5d/0x70
[1457468.880972] ? bio_endio+0xe6/0x150
[1457468.880972] ? bio_chain_endio+0x2d/0x40
[1457468.880972] ? md_end_io+0x5d/0x70
[1457468.880972] ? bio_endio+0xe6/0x150
[1457468.880973] ? bio_chain_endio+0x2d/0x40
[1457468.880973] ? md_end_io+0x5d/0x70
[1457468.880973] ? bio_endio+0xe6/0x150
[1457468.880973] ? bio_chain_endio+0x2d/0x40
[1457468.880974] ? md_end_io+0x5d/0x70
[1457468.880974] ? bio_endio+0xe6/0x150
[1457468.880974] ? bio_chain_endio+0x2d/0x40
[1457468.880975] ? md_end_io+0x5d/0x70
[1457468.880975] ? bio_endio+0xe6/0x150
[1457468.880975] ? bio_chain_endio+0x2d/0x40
[1457468.880975] ? md_end_io+0x5d/0x70
[1457468.880976] ? bio_endio+0xe6/0x150
[1457468.880976] ? bio_chain_endio+0x2d/0x40
[1457468.880976] ? md_end_io+0x5d/0x70
[1457468.880976] ? bio_endio+0xe6/0x150
[1457468.880977] ? bio_chain_endio+0x2d/0x40
[1457468.880977] ? md_end_io+0x5d/0x70
[1457468.880977] ? bio_endio+0xe6/0x150
[1457468.880978] ? bio_chain_endio+0x2d/0x40
[1457468.880978] ? md_end_io+0x5d/0x70
[1457468.880978] ? bio_endio+0xe6/0x150
[1457468.880978] ? bio_chain_endio+0x2d/0x40
[1457468.880979] ? md_end_io+0x5d/0x70
[1457468.880979] ? bio_endio+0xe6/0x150
[1457468.880979] ? bio_chain_endio+0x2d/0x40
[1457468.880979] ? md_end_io+0x5d/0x70
[1457468.880980] ? bio_endio+0xe6/0x150
[1457468.880980] ? bio_chain_endio+0x2d/0x40
[1457468.880980] ? md_end_io+0x5d/0x70
[1457468.880981] ? bio_endio+0xe6/0x150
[1457468.880981] ? bio_chain_endio+0x2d/0x40
[1457468.880981] ? md_end_io+0x5d/0x70
[1457468.880981] ? bio_endio+0xe6/0x150
[1457468.880982] ? bio_chain_endio+0x2d/0x40
[1457468.880982] ? md_end_io+0x5d/0x70
[1457468.880982] ? bio_endio+0xe6/0x150
[1457468.880982] ? bio_chain_endio+0x2d/0x40
[1457468.880983] ? md_end_io+0x5d/0x70
[1457468.880983] ? bio_endio+0xe6/0x150
[1457468.880983] ? bio_chain_endio+0x2d/0x40
[1457468.880984] ? md_end_io+0x5d/0x70
[1457468.880984] ? bio_endio+0xe6/0x150
[1457468.880984] ? bio_chain_endio+0x2d/0x40
[1457468.880984] ? md_end_io+0x5d/0x70
[1457468.880985] ? bio_endio+0xe6/0x150
[1457468.880985] ? bio_chain_endio+0x2d/0x40
[1457468.880985] ? md_end_io+0x5d/0x70
[1457468.880985] ? bio_endio+0xe6/0x150
[1457468.880986] ? bio_chain_endio+0x2d/0x40
[1457468.880986] ? md_end_io+0x5d/0x70
[1457468.880986] ? bio_endio+0xe6/0x150
[1457468.880987] ? bio_chain_endio+0x2d/0x40
[1457468.880987] ? md_end_io+0x5d/0x70
[1457468.880987] ? bio_endio+0xe6/0x150
[1457468.880987] ? bio_chain_endio+0x2d/0x40
[1457468.880988] ? md_end_io+0x5d/0x70
[1457468.880988] ? bio_endio+0xe6/0x150
[1457468.880988] ? bio_chain_endio+0x2d/0x40
[1457468.880988] ? md_end_io+0x5d/0x70
[1457468.880989] ? bio_endio+0xe6/0x150
[1457468.880989] ? bio_chain_endio+0x2d/0x40
[1457468.880989] ? md_end_io+0x5d/0x70
[1457468.880989] ? bio_endio+0xe6/0x150
[1457468.880990] ? bio_chain_endio+0x2d/0x40
[1457468.880990] ? md_end_io+0x5d/0x70
[1457468.880990] ? bio_endio+0xe6/0x150
[1457468.880991] ? bio_chain_endio+0x2d/0x40
[1457468.880991] ? md_end_io+0x5d/0x70
[1457468.880991] ? bio_endio+0xe6/0x150
[1457468.880991] ? bio_chain_endio+0x2d/0x40
[1457468.880992] ? md_end_io+0x5d/0x70
[1457468.880992] ? bio_endio+0xe6/0x150
[1457468.880992] ? bio_chain_endio+0x2d/0x40
[1457468.880992] ? md_end_io+0x5d/0x70
[1457468.880993] ? bio_endio+0xe6/0x150
[1457468.880993] ? bio_chain_endio+0x2d/0x40
[1457468.880993] ? md_end_io+0x5d/0x70
[1457468.880994] ? bio_endio+0xe6/0x150
[1457468.880994] ? bio_chain_endio+0x2d/0x40
[1457468.880994] ? md_end_io+0x5d/0x70
[1457468.880994] ? bio_endio+0xe6/0x150
[1457468.880995] ? bio_chain_endio+0x2d/0x40
[1457468.880995] ? md_end_io+0x5d/0x70
[1457468.880995] ? bio_endio+0xe6/0x150
[1457468.880995] ? bio_chain_endio+0x2d/0x40
[1457468.880996] ? md_end_io+0x5d/0x70
[1457468.880996] ? bio_endio+0xe6/0x150
[1457468.880996] ? bio_chain_endio+0x2d/0x40
[1457468.880997] ? md_end_io+0x5d/0x70
[1457468.880997] ? bio_endio+0xe6/0x150
[1457468.880997] ? bio_chain_endio+0x2d/0x40
[1457468.880997] ? md_end_io+0x5d/0x70
[1457468.880998] ? bio_endio+0xe6/0x150
[1457468.880998] ? bio_chain_endio+0x2d/0x40
[1457468.880998] ? md_end_io+0x5d/0x70
[1457468.880998] ? bio_endio+0xe6/0x150
[1457468.880999] ? bio_chain_endio+0x2d/0x40
[1457468.880999] ? md_end_io+0x5d/0x70
[1457468.880999] ? bio_endio+0xe6/0x150
[1457468.881000] ? bio_chain_endio+0x2d/0x40
[1457468.881000] ? md_end_io+0x5d/0x70
[1457468.881000] ? bio_endio+0xe6/0x150
[1457468.881000] ? bio_chain_endio+0x2d/0x40
[1457468.881001] ? md_end_io+0x5d/0x70
[1457468.881001] ? bio_endio+0xe6/0x150
[1457468.881001] ? bio_chain_endio+0x2d/0x40
[1457468.881001] ? md_end_io+0x5d/0x70
[1457468.881002] ? bio_endio+0xe6/0x150
[1457468.881002] ? bio_chain_endio+0x2d/0x40
[1457468.881002] ? md_end_io+0x5d/0x70
[1457468.881003] ? bio_endio+0xe6/0x150
[1457468.881003] ? bio_chain_endio+0x2d/0x40
[1457468.881003] ? md_end_io+0x5d/0x70
[1457468.881003] ? bio_endio+0xe6/0x150
[1457468.881004] ? bio_chain_endio+0x2d/0x40
[1457468.881004] ? md_end_io+0x5d/0x70
[1457468.881004] ? bio_endio+0xe6/0x150
[1457468.881004] ? bio_chain_endio+0x2d/0x40
[1457468.881005] ? md_end_io+0x5d/0x70
[1457468.881005] ? bio_endio+0xe6/0x150
[1457468.881005] ? bio_chain_endio+0x2d/0x40
[1457468.881006] ? md_end_io+0x5d/0x70
[1457468.881006] ? bio_endio+0xe6/0x150
[1457468.881006] ? bio_chain_endio+0x2d/0x40
[1457468.881006] ? md_end_io+0x5d/0x70
[1457468.881007] ? bio_endio+0xe6/0x150
[1457468.881007] ? bio_chain_endio+0x2d/0x40
[1457468.881007] ? md_end_io+0x5d/0x70
[1457468.881007] ? bio_endio+0xe6/0x150
[1457468.881008] ? bio_chain_endio+0x2d/0x40
[1457468.881008] ? md_end_io+0x5d/0x70
[1457468.881008] ? bio_endio+0xe6/0x150
[1457468.881009] ? bio_chain_endio+0x2d/0x40
[1457468.881009] ? md_end_io+0x5d/0x70
[1457468.881009] ? bio_endio+0xe6/0x150
[1457468.881009] ? bio_chain_endio+0x2d/0x40
[1457468.881010] ? md_end_io+0x5d/0x70
[1457468.881010] ? bio_endio+0xe6/0x150
[1457468.881010] ? bio_chain_endio+0x2d/0x40
[1457468.881010] ? md_end_io+0x5d/0x70
[1457468.881011] ? bio_endio+0xe6/0x150
[1457468.881011] ? bio_chain_endio+0x2d/0x40
[1457468.881011] ? md_end_io+0x5d/0x70
[1457468.881011] ? bio_endio+0xe6/0x150
[1457468.881012] ? bio_chain_endio+0x2d/0x40
[1457468.881012] ? md_end_io+0x5d/0x70
[1457468.881012] ? bio_endio+0xe6/0x150
[1457468.881013] ? bio_chain_endio+0x2d/0x40
[1457468.881013] ? md_end_io+0x5d/0x70
[1457468.881013] ? bio_endio+0xe6/0x150
[1457468.881013] ? bio_chain_endio+0x2d/0x40
[1457468.881014] ? md_end_io+0x5d/0x70
[1457468.881014] ? bio_endio+0xe6/0x150
[1457468.881014] ? bio_chain_endio+0x2d/0x40
[1457468.881014] ? md_end_io+0x5d/0x70
[1457468.881015] ? bio_endio+0xe6/0x150
[1457468.881015] ? bio_chain_endio+0x2d/0x40
[1457468.881015] ? md_end_io+0x5d/0x70
[1457468.881016] ? bio_endio+0xe6/0x150
[1457468.881016] ? bio_chain_endio+0x2d/0x40
[1457468.881016] ? md_end_io+0x5d/0x70
[1457468.881016] ? bio_endio+0xe6/0x150
[1457468.881017] ? bio_chain_endio+0x2d/0x40
[1457468.881017] ? md_end_io+0x5d/0x70
[1457468.881017] ? bio_endio+0xe6/0x150
[1457468.881017] ? bio_chain_endio+0x2d/0x40
[1457468.881018] ? md_end_io+0x5d/0x70
[1457468.881018] ? bio_endio+0xe6/0x150
[1457468.881018] ? bio_chain_endio+0x2d/0x40
[1457468.881019] ? md_end_io+0x5d/0x70
[1457468.881019] ? bio_endio+0xe6/0x150
[1457468.881019] ? bio_chain_endio+0x2d/0x40
[1457468.881019] ? md_end_io+0x5d/0x70
[1457468.881020] ? bio_endio+0xe6/0x150
[1457468.881020] ? bio_chain_endio+0x2d/0x40
[1457468.881020] ? md_end_io+0x5d/0x70
[1457468.881020] ? bio_endio+0xe6/0x150
[1457468.881021] ? bio_chain_endio+0x2d/0x40
[1457468.881021] ? md_end_io+0x5d/0x70
[1457468.881021] ? bio_endio+0xe6/0x150
[1457468.881022] ? bio_chain_endio+0x2d/0x40
[1457468.881022] ? md_end_io+0x5d/0x70
[1457468.881022] ? bio_endio+0xe6/0x150
[1457468.881022] ? bio_chain_endio+0x2d/0x40
[1457468.881023] ? md_end_io+0x5d/0x70
[1457468.881023] ? bio_endio+0xe6/0x150
[1457468.881023] ? bio_chain_endio+0x2d/0x40
[1457468.881023] ? md_end_io+0x5d/0x70
[1457468.881024] ? bio_endio+0xe6/0x150
[1457468.881024] ? bio_chain_endio+0x2d/0x40
[1457468.881024] ? md_end_io+0x5d/0x70
[1457468.881025] ? bio_endio+0xe6/0x150
[1457468.881025] ? bio_chain_endio+0x2d/0x40
[1457468.881025] ? md_end_io+0x5d/0x70
[1457468.881025] ? bio_endio+0xe6/0x150
[1457468.881026] ? bio_chain_endio+0x2d/0x40
[1457468.881026] ? md_end_io+0x5d/0x70
[1457468.881026] ? bio_endio+0xe6/0x150
[1457468.881026] ? bio_chain_endio+0x2d/0x40
[1457468.881027] ? md_end_io+0x5d/0x70
[1457468.881027] ? bio_endio+0xe6/0x150
[1457468.881027] ? bio_chain_endio+0x2d/0x40
[1457468.881028] ? md_end_io+0x5d/0x70
[1457468.881028] ? bio_endio+0xe6/0x150
[1457468.881028] ? bio_chain_endio+0x2d/0x40
[1457468.881028] ? md_end_io+0x5d/0x70
[1457468.881029] ? bio_endio+0xe6/0x150
[1457468.881029] ? bio_chain_endio+0x2d/0x40
[1457468.881029] ? md_end_io+0x5d/0x70
[1457468.881029] ? bio_endio+0xe6/0x150
[1457468.881030] ? bio_chain_endio+0x2d/0x40
[1457468.881030] ? md_end_io+0x5d/0x70
[1457468.881030] ? bio_endio+0xe6/0x150
[1457468.881030] ? bi
[1457468.881031] Lost 306 message(s)!
[ 0.000000] Linux version 5.8.0-53-generic (buildd@lgw01-amd64-020) (gcc (Ubuntu 10.2.0-13ubuntu1) 10.2.0, GNU ld (GNU Binutils for Ubuntu) 2.35.1) #60-Ubuntu SMP Thu May 6 07:46:32 UTC 2021 (Ubuntu 5.8.0-53.60-generic 5.8.18)
[ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-5.8.0-53-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro console=tty1 console=ttyS0,115200 processor.max_cstate=5 rcu_nocbs=0-15
