[truadm@foreman-esxi ~]$ uname -a Linux foreman-esxi.bis.pasteur.fr 4.18.0-358.el8.x86_64 #1 SMP Mon Jan 10 13:11:20 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
using kmod-mpispi from centos-release-kmods-rebuild repository:
[truadm@foreman-esxi ~]$ modinfo mptspi filename: /lib/modules/4.18.0-358.el8.x86_64/extra/drivers/message/fusion/mptspi.ko.xz version: 3.04.20 license: GPL description: Fusion MPT SPI Host driver author: LSI Corporation rhelversion: 8.6 srcversion: 06BEC6AB1BDDA75AC1C1B0F alias: pci:v00001000d00000040sv*sd*bc*sc*i* alias: pci:v0000117Cd00000030sv*sd*bc*sc*i* alias: pci:v00001000d00000030sv*sd*bc*sc*i* depends: mptscsih,mptbase,scsi_transport_spi name: mptspi vermagic: 4.18.0-358.el8.x86_64 SMP mod_unload modversions parm: mpt_saf_te: Force enabling SEP Processor: enable=1 (default=MPTSCSIH_SAF_TE=0) (int) [truadm@foreman-esxi ~]$ rpm -qf /lib/modules/4.18.0-358.el8.x86_64/extra/drivers/message/fusion/mptspi.ko.xz kmod-mptspi-4.18.0.80-5.el8s.x86_64
[truadm@foreman-esxi ~]$ sudo lsinitrd /boot/initramfs-4.18.0-358.el8.x86_64.img |grep mptspi -rw-r--r-- 1 root root 11124 Jan 27 22:44 usr/lib/modules/4.18.0-358.el8.x86_64/extra/drivers/message/fusion/mptspi.ko.xz -rw-r--r-- 1 root root 11784 Jan 10 14:25 usr/lib/modules/4.18.0-358.el8.x86_64/kernel/drivers/message/fusion/mptspi.ko.xz
Upgrading to latest kernel from 4.18.0-358.el8.x86_64 to 4.18.0-365.el8.x86_64
[truadm@foreman-esxi ~]$ sudo lsinitrd /boot/initramfs-4.18.0-365.el8.x86_64.img |grep mptspi -rw-r--r-- 1 root root 11124 Jan 27 22:44 usr/lib/modules/4.18.0-358.el8.x86_64/extra/drivers/message/fusion/mptspi.ko.xz -rw-r--r-- 1 root root 11784 Feb 5 08:07 usr/lib/modules/4.18.0-365.el8.x86_64/kernel/drivers/message/fusion/mptspi.ko.xz lrwxrwxrwx 1 root root 78 Feb 5 08:07 usr/lib/modules/4.18.0-365.el8.x86_64/weak-updates/drivers/message/fusion/mptspi.ko.xz -> ../../../../../4.18.0-358.el8.x86_64/extra/drivers/message/fusion/mptspi.ko.xz
-> yields unbootable VM
kmod-mptspi for 4.18.0-365.el8.x86_64 is not yet available and a blocker with a red flag should prevent the kernel upgrade, imho. This is one of the driver affected by the non stable kABI? "update/rebuild" showed on https://sigs.centos.org/kmods/kabi/c8s/ ? If that is possible, build and release affected kmod when the stream8 kernel are released.
Thanks
Tru
We do have a process that automatically triggers rebuilds for new kernel releases if required. This process determined that no rebuild is required for kmod-mptspi going from -358 to -365. I just manually confirmed: No rebuild is required.
Hence there seems to be a bug/error somewhere else. My first guess was that the modified mptspi.ko.xz is not included in the initramfs but only the one provided by the kernel itself. But this is obviously not the case as shown by the info you provided.
Can you please provide the error message you get trying to boot 4.18.0-365.el8.x86_64? I assume that for some reasons the mptspi.ko.xz in subdirectory kernel is used instead of the one in weak-updates. But I want to verify that first.
Here are the screenshot of the console VM for 4.18.0-365.el8.x86_64, I could only have the last page of the console, no video recording, sorry!. <img alt="00-grub.png" src="/centos-sig-kmods/sig/issue/raw/files/786ab36d3e9e7f0f467942fe6f76ac7a3145cecec84e882e8d85e0501aecdeb8-00-grub.png" /><img alt="01-cmdline.png" src="/centos-sig-kmods/sig/issue/raw/files/6e10f3d16f8bd4ee89bf4d1251a4a93a3077a2e957925d61efdae405cbeb6cd2-01-cmdline.png" /><img alt="03-crash.png" src="/centos-sig-kmods/sig/issue/raw/files/16ab94004fa26b8ee468060124b271c0aa6ddb0c03578d4d322005cfe1cb6115-03-crash.png" />
Thanks for the provided info. The last picture is the most interesting. It lists mptspi(OE+) as linked in module. This is the one provided by the kmod-mptspi not by the kernel itself (O means out of tree, E unsigned). I.e. my assumption that the wrong mptspi.ko is loaded is wrong.
mptspi(OE+)
Now the interesting question is why the mptspi module crashes when running -365 kernel but works when running the -358 kernel. None of the symbols required by mptspi have changed. And indeed the module is being loaded but then crashes. Is there any interesting output before the last screenshot?
Anyway, easiest check is probably to first try a version that is explicitly built against -365. Can you please download such version from CBS and let me know if this fixes your issue?
it's is getting worse: the newly rebuilt driver install itself in 358 and 368 and crash both on boot at the (same) place and override the kmod-mptspi for 358.
[root@foreman-esxi boot]# lsinitrd initramfs-4.18.0-358.el8.x86_64.img|grep mptspi -rw-r--r-- 1 root root 11124 Jan 27 22:44 usr/lib/modules/4.18.0-358.el8.x86_64/extra/drivers/message/fusion/mptspi.ko.xz -rw-r--r-- 1 root root 11784 Jan 10 14:25 usr/lib/modules/4.18.0-358.el8.x86_64/kernel/drivers/message/fusion/mptspi.ko.xz lrwxrwxrwx 1 root root 78 Feb 5 08:07 usr/lib/modules/4.18.0-358.el8.x86_64/weak-updates/drivers/message/fusion/mptspi.ko.xz -> ../../../../../4.18.0-365.el8.x86_64/extra/drivers/message/fusion mptspi.ko.xz -rw-r--r-- 1 root root 11128 Feb 5 08:07 usr/lib/modules/4.18.0-365.el8.x86_64/extra/drivers/message/fusion/mptspi.ko.xz [root@foreman-esxi boot]# lsinitrd initramfs-4.18.0-365.el8.x86_64.img|grep mptspi -rw-r--r-- 1 root root 11124 Jan 27 22:44 usr/lib/modules/4.18.0-358.el8.x86_64/extra/drivers/message/fusion/mptspi.ko.xz -rw-r--r-- 1 root root 11128 Feb 5 08:07 usr/lib/modules/4.18.0-365.el8.x86_64/extra/drivers/message/fusion/mptspi.ko.xz -rw-r--r-- 1 root root 11784 Feb 5 08:07 usr/lib/modules/4.18.0-365.el8.x86_64/kernel/drivers/message/fusion/mptspi.ko.xz lrwxrwxrwx 1 root root 78 Feb 5 08:07 usr/lib/modules/4.18.0-365.el8.x86_64/weak-updates/drivers/message/fusion/mptspi.ko.xz -> ../../../../../4.18.0-358.el8.x86_64/extra/drivers/message/fusion mptspi.ko.xz [root@foreman-esxi boot]#
Luckyly I have a working copy of the previous ramdisk as initramfs-4.18.0-358.el8.x86_64.img.ok to reboot into a running stream8 version...
<img alt="04-rebuilt-kmod-365.png" src="/centos-sig-kmods/sig/issue/raw/files/e0429e0e62de89cc674b82320c66933a1eb49b7cdb00d1eb001f736c1db8c181-04-rebuilt-kmod-365.png" /><img alt="05-rebuilt-kmod-358.png" src="/centos-sig-kmods/sig/issue/raw/files/e438f7a6ea286e6d84bbd04063996e938d8dcc3685e99e6d45809cc781ba779f-05-rebuilt-kmod-358.png" />
[root@foreman-esxi boot]# rpm -q kmod-mptspi kernel kmod-mptspi-4.18.0.80-5.el8s.x86_64 kmod-mptspi-4.18.0.80-6.el8s.x86_64 kernel-4.18.0-348.2.1.el8_5.x86_64 kernel-4.18.0-358.el8.x86_64 kernel-4.18.0-365.el8.x86_64 [root@foreman-esxi boot]# rpm -ql kmod-mptspi-4.18.0.80-5.el8s.x86_64|grep mptspi.ko /lib/modules/4.18.0-358.el8.x86_64/extra/drivers/message/fusion/mptspi.ko.xz [root@foreman-esxi boot]# rpm -ql kmod-mptspi-4.18.0.80-6.el8s.x86_64|grep mptspi.ko /lib/modules/4.18.0-365.el8.x86_64/extra/drivers/message/fusion/mptspi.ko.xz
imho, - kmod-mptspi-4.18.0.80-6.el8s.x86_64 should not have been install into /boot/initramfs-4.18.0-358.el8.x86_64.img (kernel release 365 > 358) - kmod-mptspi-4.18.0.80-6.el8s.x86_64 and kmod-mptspi-4.18.0.80-5.el8s.x86_64 being both installed is fine, but not sure which one was loaded.
It makes sense that a weak-updates link from -358 to -365 is being created. The module compiled for -365 is compatible with kernel -358 as no required kernel symbol have been changed between these two kernel releases. That's the main idea of the weak-updates mechanism. And another proof that the mptspi.ko compiled for -358 or -365 should indeed work for kernel -365 or -358.
You can check which module was loading by checking modules.dep in /lib/modules/<kernel>/. The kmod-mptspi installs a depmod.d config which causes the weak-updates version to always be loaded if available. This is required to overwrite the version provided by the kernel.
This is a very weird error. Which error message do you get trying to boot -358 and -365 with no kmod-mptspi installed, i.e. using the one provided by the kernel itself? Or does it simply not boot due to missing device support with no error message?
Edit: Another test that might help me pin down the issue is to only uninstall kmod-mptspi-4.18.0.80-5.el8s.x86_64 and then try to booth kernel -365 with only kmod-mptspi-4.18.0.80-6.el8s.x86_64 installed. Thanks!
I needed the support provided by the kmod-mptspi driver during el8 installation. I suppose it is still the case today.
[truadm@foreman-esxi ~]$ lspci -k ... 00:10.0 SCSI storage controller: Broadcom / LSI 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 01) Kernel driver in use: mptspi Kernel modules: mptspi ... [truadm@foreman-esxi ~]$ lspci -n -s 00:10.0 00:10.0 0100: 1000:0030 (rev 01) [truadm@foreman-esxi ~]$ rpm -qf /lib/modules/4.18.0-358.el8.x86_64/extra/drivers/message/fusion/mptspi.ko.xz kmod-mptspi-4.18.0.80-5.el8s.x86_64 [truadm@foreman-esxi ~]$ modinfo /lib/modules/4.18.0-358.el8.x86_64/extra/drivers/message/fusion/mptspi.ko.xz |grep alias alias: pci:v00001000d00000040sv*sd*bc*sc*i* alias: pci:v0000117Cd00000030sv*sd*bc*sc*i* alias: pci:v00001000d00000030sv*sd*bc*sc*i* [truadm@foreman-esxi ~]$ modinfo /lib/modules/4.18.0-358.el8.x86_64/kernel/drivers/message/fusion/mptspi.ko.xz|grep alias alias: pci:v00001000d00000030sv000015ADsd*bc*sc*i*
testing without kmod-mptspi-4.18.0.80-5.el8s.x86_64
[root@foreman-esxi boot]# rpm -q kmod-mptspi kmod-mptspi-4.18.0.80-6.el8s.x86_64 [root@foreman-esxi boot]# lsinitrd initramfs-4.18.0-358.el8.x86_64.img|grep mptspi -rw-r--r-- 1 root root 11784 Jan 10 14:25 usr/lib/modules/4.18.0-358.el8.x86_64/kernel/drivers/message/fusion/mptspi.ko.xz lrwxrwxrwx 1 root root 78 Feb 5 08:07 usr/lib/modules/4.18.0-358.el8.x86_64/weak-updates/drivers/message/fusion/mptspi.ko.xz -> ../../../../../4.18.0-365.el8.x86_64/extra/drivers/message/fusion mptspi.ko.xz -rw-r--r-- 1 root root 11128 Feb 5 08:07 usr/lib/modules/4.18.0-365.el8.x86_64/extra/drivers/message/fusion/mptspi.ko.xz [root@foreman-esxi boot]# lsinitrd initramfs-4.18.0-365.el8.x86_64.img|grep mptspi -rw-r--r-- 1 root root 11128 Feb 5 08:07 usr/lib/modules/4.18.0-365.el8.x86_64/extra/drivers/message/fusion/mptspi.ko.xz -rw-r--r-- 1 root root 11784 Feb 5 08:07 usr/lib/modules/4.18.0-365.el8.x86_64/kernel/drivers/message/fusion/mptspi.ko.xz
-365 boots with only kmod-mptspi -6
[truadm@foreman-esxi ~]$ uname -a Linux foreman-esxi.bis.pasteur.fr 4.18.0-365.el8.x86_64 #1 SMP Thu Feb 10 16:11:23 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux [truadm@foreman-esxi ~]$ modinfo mptspi filename: /lib/modules/4.18.0-365.el8.x86_64/extra/drivers/message/fusion/mptspi.ko.xz version: 3.04.20 license: GPL description: Fusion MPT SPI Host driver author: LSI Corporation rhelversion: 8.6 srcversion: 06BEC6AB1BDDA75AC1C1B0F alias: pci:v00001000d00000040sv*sd*bc*sc*i* alias: pci:v0000117Cd00000030sv*sd*bc*sc*i* alias: pci:v00001000d00000030sv*sd*bc*sc*i* depends: mptscsih,mptbase,scsi_transport_spi name: mptspi vermagic: 4.18.0-365.el8.x86_64 SMP mod_unload modversions parm: mpt_saf_te: Force enabling SEP Processor: enable=1 (default=MPTSCSIH_SAF_TE=0) (int)
and 358 crashes with the 365 kmod...
<img alt="06-358-with-kmod-6.png" src="/centos-sig-kmods/sig/issue/raw/files/cdc71cbcdc90653f9e7331fb758cbc0012f6ee60c723f0fa5704878de02d9f10-06-358-with-kmod-6.png" />
summary: - kernel-358 with kmod-mptspi-5 (for 358) boots - kernel-358 with kmod-mptspi-6 (for 365) crash - kernel-365 with kmod-mptspi-6 (for 365) boots
kernel 358 or 365 with both kmod instaled crashes
@tru Thanks for the testing!
To summarize: kernel-358 with kmod-mptspi build for kernel-358: Works kernel-365 with kmod-mptspi build for kernel-365: Works kernel-358 with kmod-mptspi build for kernel-365: Panic kernel-365 with kmod-mptspi build for kernel-358: Panic
So the conclusion is: There is some change between kernel-358 and kernel-365 that is not visible from used kernel symbols.
We do know a way to fix this now. However I want to further investigate further to understand the issue and be able to detect such incompatibilities.
This issue is especially bad as the weak-modules mechanism also detects the kmod as compatible. This means a fix is not easy to implement. I would need some way to tell depmod to use mptspi.ko from weak-updates subdirectory and not the one from subdirectory kernel, but use from subdirectory extra if available. This could be done by modifying the search command of depmod. However this is afaik a global option and can not be set per module.
@tru Do you mind running one more test for me? I can't really test myself as I have to device depending on mptspi.
Please install the version found here and test booting both kernel-358 and kernel-365. Thanks!
Just to be sure: remove all previous kmod-mptspi and only install https://cbs.centos.org/kojifiles/work/tasks/5819/2685819/kmod-mptspi-4.18.0.80-5.1.el8s.x86_64.rpm for both kernels 358 and 365. Correct?
Yes. Uninstall the other kmod-mptspi versions first. Thanks!
[truadm@foreman-esxi ~]$ uname -a Linux foreman-esxi.bis.pasteur.fr 4.18.0-365.el8.x86_64 #1 SMP Thu Feb 10 16:11:23 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux [truadm@foreman-esxi ~]$ rpm -q kmod-mptspi package kmod-mptspi is not installed [truadm@foreman-esxi ~]$ find /lib/modules -name mptspi.ko\* /lib/modules/4.18.0-348.2.1.el8_5.x86_64/kernel/drivers/message/fusion/mptspi.ko.xz /lib/modules/4.18.0-358.el8.x86_64/kernel/drivers/message/fusion/mptspi.ko.xz /lib/modules/4.18.0-365.el8.x86_64/kernel/drivers/message/fusion/mptspi.ko.xz [truadm@foreman-esxi ~]$ sudo lsinitrd /boot/initramfs-4.18.0-358.el8.x86_64.img|grep -i mptspi -rw-r--r-- 1 root root 11784 Jan 10 14:25 usr/lib/modules/4.18.0-358.el8.x86_64/kernel/drivers/message/fusion/mptspi.ko.xz [truadm@foreman-esxi ~]$ sudo lsinitrd /boot/initramfs-4.18.0-365.el8.x86_64.img|grep -i mptspi -rw-r--r-- 1 root root 11784 Feb 5 08:07 usr/lib/modules/4.18.0-365.el8.x86_64/kernel/drivers/message/fusion/mptspi.ko.xz [truadm@foreman-esxi ~]$
Installing:
[truadm@foreman-esxi ~]$ sudo dnf install -y kmod-mptspi-4.18.0.80-5.1.el8s.x86_64.rpm ... [truadm@foreman-esxi ~]$ find /lib/modules -name mptspi.ko\* /lib/modules/4.18.0-348.2.1.el8_5.x86_64/kernel/drivers/message/fusion/mptspi.ko.xz /lib/modules/4.18.0-358.el8.x86_64/kernel/drivers/message/fusion/mptspi.ko.xz /lib/modules/4.18.0-358.el8.x86_64/extra/drivers/message/fusion/mptspi.ko.xz /lib/modules/4.18.0-365.el8.x86_64/kernel/drivers/message/fusion/mptspi.ko.xz /lib/modules/4.18.0-365.el8.x86_64/weak-updates/lib/modules/4.18.0-358.el8.x86_64/extra/drivers/message/fusion/mptspi.ko.xz [truadm@foreman-esxi ~]$ sudo lsinitrd /boot/initramfs-4.18.0-358.el8.x86_64.img|grep -i mptspi -rw-r--r-- 1 root root 11784 Jan 10 14:25 usr/lib/modules/4.18.0-358.el8.x86_64/kernel/drivers/message/fusion/mptspi.ko.xz [truadm@foreman-esxi ~]$ sudo lsinitrd /boot/initramfs-4.18.0-365.el8.x86_64.img|grep -i mptspi -rw-r--r-- 1 root root 11784 Feb 5 08:07 usr/lib/modules/4.18.0-365.el8.x86_64/kernel/drivers/message/fusion/mptspi.ko.xz
kmod-mptspi was NOT added to the ramdisks.. the machine will panic at reboot.
It seems initramfs has not been re-generated. This path also look very awkward.
/lib/modules/4.18.0-365.el8.x86_64/weak-updates/lib/modules/4.18.0-358.el8.x86_64/extra/drivers/message/fusion/mptspi.ko.xz
Something obviously went wrong there. Anyway I had a closer look and it seems the explanation I thought about is wrong. This test is not required anymore, sorry.
However it would be great if you could provide the full kernel trace for any of the configurations that result in a kernel panic. The screenshots provided so far all only show the last part of the trace. I.e. for one of these two, or both if possible: kernel-358 with kmod-mptspi build for kernel-365 (kmod-mptspi-4.18.0.80-6.el8s.x86_64) kernel-365 with kmod-mptspi build for kernel-358 (kmod-mptspi-4.18.0.80-5.el8s.x86_64)
as expected 365 panicked <img alt="07-365-no-extra.png" src="/centos-sig-kmods/sig/issue/raw/files/a6fa7975d3f30b264493e8a600f8c5920857d1b33a62c7d1689f0cb0b13826e0-07-365-no-extra.png" /><img alt="08-365-no-extra-timeout.png" src="/centos-sig-kmods/sig/issue/raw/files/587ebeb6cd300c3c509882557e950fc272f86429294bf7fce7aa153e79b814c0-08-365-no-extra-timeout.png" /><img alt="09-365-no-extra-rdshell.png" src="/centos-sig-kmods/sig/issue/raw/files/d2a250778d762bc55951437ce978d416fec44874b3b67d8d921a2d147a7f9b2d-09-365-no-extra-rdshell.png" /><img alt="10-358-no-extra.png" src="/centos-sig-kmods/sig/issue/raw/files/8741cb4e2e95371b1484d780cc428fc70de7fa8722c96ae7fa8a5c42039980c7-10-358-no-extra.png" />
358 panicked <img alt="11-358-no-extra-timeout.png" src="/centos-sig-kmods/sig/issue/raw/files/35c3af8cc6f428dd823f7ea9e5e5ca3919c3da228b4d7a5a8b107ac46f173695-11-358-no-extra-timeout.png" /><img alt="12-358-no-extra-rdhell.png" src="/centos-sig-kmods/sig/issue/raw/files/a265ed23d6e066c5a3f429ea83dec90ceeafdc259ccd167d588a3d84ee129791-12-358-no-extra-rdhell.png" />
[truadm@foreman-esxi ~]$ rpm -q kmod-mptspi kmod-mptspi-4.18.0.80-5.el8s.x86_64 [truadm@foreman-esxi ~]$ sudo lsinitrd /boot/initramfs-4.18.0-365.el8.x86_64.img|grep -i mptspi -rw-r--r-- 1 root root 11124 Jan 27 22:44 usr/lib/modules/4.18.0-358.el8.x86_64/extra/drivers/message/fusion/mptspi.ko.xz -rw-r--r-- 1 root root 11784 Feb 5 08:07 usr/lib/modules/4.18.0-365.el8.x86_64/kernel/drivers/message/fusion/mptspi.ko.xz lrwxrwxrwx 1 root root 78 Feb 5 08:07 usr/lib/modules/4.18.0-365.el8.x86_64/weak-updates/drivers/message/fusion/mptspi.ko.xz -> ../../../../../4.18.0-358.el8.x86_64/extra/drivers/message/fusion mptspi.ko.xz [truadm@foreman-esxi ~]$ sudo lsinitrd /boot/initramfs-4.18.0-358.el8.x86_64.img|grep -i mptspi -rw-r--r-- 1 root root 11124 Jan 27 22:44 usr/lib/modules/4.18.0-358.el8.x86_64/extra/drivers/message/fusion/mptspi.ko.xz -rw-r--r-- 1 root root 11784 Jan 10 14:25 usr/lib/modules/4.18.0-358.el8.x86_64/kernel/drivers/message/fusion/mptspi.ko.xz
I will try to add a serial console to the vm and output to a text file
<img alt="365-kmod-358.serial.txt" src="/centos-sig-kmods/sig/issue/raw/files/60853ac3f16fe3734b0cc8d00c4b53d819fdecddcf28697d32f4c538a77f52ce-365-kmod-358.serial.txt" />
[truadm@foreman-esxi ~]$ sudo dnf install -y kmod-mptspi.x86_64 Last metadata expiration check: 0:43:15 ago on Mon 14 Feb 2022 11:10:05 PM CET. Dependencies resolved. =================================================================================================================================================================================================================== Package Architecture Version Repository Size =================================================================================================================================================================================================================== Installing: kmod-mptspi x86_64 4.18.0.80-5.el8s centos-kmods-rebuild 32 k Transaction Summary =================================================================================================================================================================================================================== Install 1 Package Total download size: 32 k Installed size: 29 k Downloading Packages: kmod-mptspi-4.18.0.80-5.el8s.x86_64.rpm 270 kB/s | 32 kB 00:00 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 137 kB/s | 32 kB 00:00 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Running scriptlet: kmod-mptspi-4.18.0.80-5.el8s.x86_64 1/1 Preparing : 1/1 Installing : kmod-mptspi-4.18.0.80-5.el8s.x86_64 1/1 Running scriptlet: kmod-mptspi-4.18.0.80-5.el8s.x86_64 1/1 Verifying : kmod-mptspi-4.18.0.80-5.el8s.x86_64 1/1 Installed: kmod-mptspi-4.18.0.80-5.el8s.x86_64 Complete! [truadm@foreman-esxi ~]$ rpm -q kmod-mptspi kmod-mptspi-4.18.0.80-5.el8s.x86_64 [truadm@foreman-esxi ~]$ sudo lsinitrd /boot/initramfs-4.18.0-365.el8.x86_64.img|grep -i mptspi -rw-r--r-- 1 root root 11124 Jan 27 22:44 usr/lib/modules/4.18.0-358.el8.x86_64/extra/drivers/message/fusion/mptspi.ko.xz -rw-r--r-- 1 root root 11784 Feb 5 08:07 usr/lib/modules/4.18.0-365.el8.x86_64/kernel/drivers/message/fusion/mptspi.ko.xz lrwxrwxrwx 1 root root 78 Feb 5 08:07 usr/lib/modules/4.18.0-365.el8.x86_64/weak-updates/drivers/message/fusion/mptspi.ko.xz -> ../../../../../4.18.0-358.el8.x86_64/extra/drivers/message/fusion mptspi.ko.xz [truadm@foreman-esxi ~]$ sudo lsinitrd /boot/initramfs-4.18.0-358.el8.x86_64.img|grep -i mptspi -rw-r--r-- 1 root root 11124 Jan 27 22:44 usr/lib/modules/4.18.0-358.el8.x86_64/extra/drivers/message/fusion/mptspi.ko.xz -rw-r--r-- 1 root root 11784 Jan 10 14:25 usr/lib/modules/4.18.0-358.el8.x86_64/kernel/drivers/message/fusion/mptspi.ko.xz
booting with /boot/initramfs-4.18.0-365.el8.x86_64.img yield the serial console attached.
Thanks for the trace! The panic is happening within the PCI subsystem. More precisely in the function __pci_register_driver. Its ABI is stable, kernel(__pci_register_driver) = 0xe85eaefe, but it obviously is not.
Not sure whether this behavior is considered a bug by Red Hat. I opened a bug report to check that 1.
Anyway we need to think how to do rebuilds to fix such issues even if all kABI compatibility checks pass.
Metadata Update from @pjgeorg: - Issue assigned to pjgeorg
Let's see how https://bugzilla.redhat.com/show_bug.cgi?id=2054595#c22 is resolved for stream8, how the kmod could handle this back and forth compatibility within the same dot release. That will be a live test of what stream early adopter could go through on a critical driver issue, imho.
According to RH the bug should be fixed in kernel-4.18.0-373.el8 for stream and kernel-4.18.0-372.1.1.el8 for el. However we now have a new bug report #26 and all tests I have done so far indicate that this bug has not yet been fixed. However in the other bu, the issue is not traceable that easily. Hence I want to ask you for some information (I do not have the hardware to reproduce any of the bugs myself).
Have you updated your kernel version to something >= -373? If yes, which kmod did you use? kmod-mptspi-4.18.0.80-6.el8s.x86_64 or kmod-mptspi-4.18.0.80-5.el8s.x86_64 ?
In case you have been using kmod-mptspi-4.18.0.80-6.el8s.x86_64, can you please try booting the following combination and provide a trace in case it crashes: kernel-373 (or greater) with kmod-mptspi build for kernel-358 (kmod-mptspi-4.18.0.80-5.el8s.x86_64)
You have to uninstall kmod-mptspi-4.18.0.80-6.el8s.x86_64 for this test.
Thanks!
We have now done rebuilds for -365 and -373 of all kernel modules requiring the affected kernel symbol.
Metadata Update from @pjgeorg: - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.