Upgraded to unsupported kernel version accidently. Fedora Server 40, kernel 6.11, zfs 2.2.6

[osirase]

Hiya,

I was under the assumption that the kernel would be locked via https://openzfs.github.io/openzfs-docs/Getting%20Started/Fedora/index.html, an incorrect assumption on my behalf.

The latest kernel update introduced instability as expected. I see the latest supported kernel for zfs 2.2.6 is 6.10. Unfortunately this kernel is no longer in my history and isn't available in the updates repo from fedora.

I'm thinking of temporarily uninstalling zfs and waiting for 2.3.0 to release then reinstalling. Is this a safe approach?

I'm currently getting some dkms errors when installing, something about the source not being available. I will attach the exact logs later when I have access again.

If anyone has any other ideas of how I might rescue this system I'm all ears. For now I've reclined to accepting my zfs pool will have to be down to a newer release that supports newer kernels comes out.

[mskarbek]

You can download latest 6.10 for fc40 from Koji: https://koji.fedoraproject.org/koji/buildinfo?buildID=2566685

[osirase]

Ok, I've ran this on a test VM:

user@localhost:~$ uname -r
6.10.14-200.fc40.x86_64
user@localhost:~$ ll
total 141388
-rw-r--r--. 1 user user  181709 Oct 29 12:31 kernel-6.10.14-200.fc40.x86_64.rpm
-rw-r--r--. 1 user user 18273077 Oct 29 12:32 kernel-core-6.10.14-200.fc40.x86_64.rpm
-rw-r--r--. 1 user user 17916061 Oct 29 12:40 kernel-devel-6.10.14-200.fc40.x86_64.rpm
-rw-r--r--. 1 user user 66158745 Oct 29 12:32 kernel-modules-6.10.14-200.fc40.x86_64.rpm
-rw-r--r--. 1 user user 39226549 Oct 29 12:33 kernel-modules-core-6.10.14-200.fc40.x86_64.rpm
-rw-r--r--. 1 user user  3009233 Oct 29 12:33 kernel-modules-extra-6.10.14-200.fc40.x86_64.rpm

However I'm still a bit worried as it tried to pull in the latest kernel-devel as a dependency regardless:

sudo dnf install -y zfs
Last metadata expiration check: 0:00:09 ago on Tue 29 Oct 2024 12:46:29.
Dependencies resolved.
======================================================================================
 Package                      Arch   Version                            Repo     Size
======================================================================================
Installing:
 kernel-devel                 x86_64 6.11.4-201.fc40                    updates  21 M
 zfs                          x86_64 2.2.6-1.fc40                       zfs     729 k
Installing dependencies:

Although it seems fine.. I'm not getting the other dkms errors that I do on my production system. I'll collect those next when I'm on that system.

[osirase]

I'm on my production server at the moment, set the kernel to 6.10.14. Added an extra download that was showing up as a dependency for sudo dnf install zfs which was kernel-devel-matched-6.10.14-200.fc40.x86_64.rpm

Current kernel installation:

sudo dnf list --installed | grep -i kernel | grep -i 10.14
kernel.x86_64                             6.10.14-200.fc40                   @@commandline
kernel-core.x86_64                        6.10.14-200.fc40                   @@commandline
kernel-devel.x86_64                       6.10.14-200.fc40                   @@commandline
kernel-devel-matched.x86_64               6.10.14-200.fc40                   @@commandline
kernel-modules.x86_64                     6.10.14-200.fc40                   @@commandline
kernel-modules-core.x86_64                6.10.14-200.fc40                   @@commandline
kernel-modules-extra.x86_64               6.10.14-200.fc40                   @@commandline
user@server:~$ uname -r
6.10.14-200.fc40.x86_64

The errors I'm seeing from sudo dnf install zfs

  Installing       : perl-4:5.38.2-506.fc40.x86_64                                                                                                                   199/201 
  Running scriptlet: zfs-dkms-2.2.6-1.fc40.noarch                                                                                                                    200/201 
Running pre installation script: /var/tmp/rpm-tmp.gWI5vs. Parameters: 1

Error! zfs/2.2.6: Missing the module source directory or the symbolic link pointing to it.
Manual intervention is required!
Removing zfs dkms modules version 2.2.6 from all kernels.

Error! Could not locate dkms.conf file.
File: /var/lib/dkms/zfs/2.2.6/source/dkms.conf does not exist.

Error! zfs/2.2.6 is broken!
Missing the source directory or the symbolic link pointing to it.
Manual intervention is required!

  Installing       : zfs-dkms-2.2.6-1.fc40.noarch                                                                                                                    200/201 
  Running scriptlet: zfs-dkms-2.2.6-1.fc40.noarch                                                                                                                    200/201

[osirase]

Well, it's still randomly soft-locking when the zfs pool is accessed. At least I'm on a supported kernel now. I'll uninstall zfs again and wait until I get some more advice on how to proceed. Not sure where logs are that might be useful.

[osirase]

I noticed I have .12 as my last stable zfs dkms directory:

user@server:/var/lib/dkms/zfs$ ll
total 24
drwxr-xr-x. 1 root root 282 Oct 29 14:25 2.2.6
lrwxrwxrwx. 1 root root  36 Sep 20 22:35 kernel-6.10.10-200.fc40.x86_64-x86_64 -> 2.2.6/6.10.10-200.fc40.x86_64/x86_64
lrwxrwxrwx. 1 root root  36 Sep 29 13:52 kernel-6.10.11-200.fc40.x86_64-x86_64 -> 2.2.6/6.10.11-200.fc40.x86_64/x86_64
lrwxrwxrwx. 1 root root  36 Oct 12 13:39 kernel-6.10.12-200.fc40.x86_64-x86_64 -> 2.2.6/6.10.12-200.fc40.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Sep  5 16:39 kernel-6.10.3-200.fc40.x86_64-x86_64 -> 2.2.6/6.10.3-200.fc40.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Sep  5 16:42 kernel-6.10.6-200.fc40.x86_64-x86_64 -> 2.2.6/6.10.6-200.fc40.x86_64/x86_64
lrwxrwxrwx. 1 root root  35 Sep  7 12:51 kernel-6.10.7-200.fc40.x86_64-x86_64 -> 2.2.6/6.10.7-200.fc40.x86_64/x86_64

So installed as follows 6.10.12-200

curl -LO https://kojipkgs.fedoraproject.org//packages/kernel/6.10.12/200.fc40/x86_64/kernel-6.10.12-200.fc40.x86_64.rpm
curl -LO https://kojipkgs.fedoraproject.org//packages/kernel/6.10.12/200.fc40/x86_64/kernel-core-6.10.12-200.fc40.x86_64.rpm
curl -LO https://kojipkgs.fedoraproject.org//packages/kernel/6.10.12/200.fc40/x86_64/kernel-devel-6.10.12-200.fc40.x86_64.rpm
curl -LO https://kojipkgs.fedoraproject.org//packages/kernel/6.10.12/200.fc40/x86_64/kernel-devel-matched-6.10.12-200.fc40.x86_64.rpm
curl -LO https://kojipkgs.fedoraproject.org//packages/kernel/6.10.12/200.fc40/x86_64/kernel-modules-6.10.12-200.fc40.x86_64.rpm
curl -LO https://kojipkgs.fedoraproject.org//packages/kernel/6.10.12/200.fc40/x86_64/kernel-modules-core-6.10.12-200.fc40.x86_64.rpm
curl -LO https://kojipkgs.fedoraproject.org//packages/kernel/6.10.12/200.fc40/x86_64/kernel-modules-extra-6.10.12-200.fc40.x86_64.rpm

dnf install ./kernel*.rpm

sudo grubby --set-default /boot/vmlinuz-6.10.12-200.fc40.x86_64

Fingers crossed no dkms missing stuff errors...

[osirase]

Slightly different but similar errors:

  Running scriptlet: zfs-dkms-2.2.6-1.fc40.noarch                                                                                                                    200/201 
Running pre installation script: /var/tmp/rpm-tmp.8pcO0s. Parameters: 1

Error! zfs/2.2.6: Missing the module source directory or the symbolic link pointing to it.
Manual intervention is required!
Removing zfs dkms modules version 2.2.6 from all kernels.

Error! Could not locate dkms.conf file.
File: /var/lib/dkms/zfs/2.2.6/source/dkms.conf does not exist.

Error! zfs/2.2.6 is broken!
Missing the source directory or the symbolic link pointing to it.
Manual intervention is required!

  Installing       : zfs-dkms-2.2.6-1.fc40.noarch                                                                                                                    200/201 
  Running scriptlet: zfs-dkms-2.2.6-1.fc40.noarch                                                                                                                    200/201 
Running post installation script: /var/tmp/rpm-tmp.DS1Y6t. Parameters: 1
Adding zfs dkms modules version 2.2.6 to dkms.

Error! DKMS tree already contains: zfs-2.2.6
You cannot add the same module/version combo more than once.
Installing zfs dkms modules version 2.2.6 for the current kernel.

Module zfs-2.2.6 for kernel 6.10.12-200.fc40.x86_64 (x86_64):
Before uninstall, this module version was ACTIVE on this kernel.
Deleting /lib/modules/6.10.12-200.fc40.x86_64/extra/zfs.ko.xz
Deleting /lib/modules/6.10.12-200.fc40.x86_64/extra/spl.ko.xz
Running depmod.... done.

Installing /lib/modules/6.10.12-200.fc40.x86_64/extra/zfs.ko.xz
Installing /lib/modules/6.10.12-200.fc40.x86_64/extra/spl.ko.xz