kernel-devel and kernel-headers matching kernel-core not available

[mtalexan]

Describe the bug

When running either rpm-ostree update or ujust update, I get the report that there's nothing to update.
However I need to build a custom out of tree kernel driver for the lighting on my laptop keyboard (Asus Predator device), so I need the kernel-devel installed. I tried this and then discovered it didn't match the current kernel-core that was installed by default.

After installing kernel-core and kernel-devel:

$ rpm -q kernel-core
kernel-core-6.9.12-205.fsync.fc40.x86_64

$ rpm -q kernel-devel
kernel-devel-6.10.12-200.fc40.x86_64

$ rpm -q kernel-headers
kernel-headers-6.10.3-200.fc40.x86_64

And when I try to install the matching version for kernel-core explicitly:

$ rpm-ostree install kernel-devel-$(uname -r) kernel-headers-$(uname -r)
Checking out tree 5b620b6... done
Enabled rpm-md repositories: copr:copr.fedorainfracloud.org:matte-schwartz:sunshine copr:copr.fedorainfracloud.org:rodoma92:kde-cdemu-manager copr:copr.fedorainfracloud.org:rodoma92:rmlint copr:copr.fedorainfracloud.org:rok:cdemu updates fedora rpmfusion-free-updates-testing rpmfusion-free-updates rpmfusion-free code updates-archive
Importing rpm-md... done
rpm-md repo 'copr:copr.fedorainfracloud.org:matte-schwartz:sunshine' (cached); generated: 2024-09-13T09:52:44Z solvables: 2
rpm-md repo 'copr:copr.fedorainfracloud.org:rodoma92:kde-cdemu-manager' (cached); generated: 2024-04-20T12:39:00Z solvables: 12
rpm-md repo 'copr:copr.fedorainfracloud.org:rodoma92:rmlint' (cached); generated: 2024-05-10T10:16:28Z solvables: 4
rpm-md repo 'copr:copr.fedorainfracloud.org:rok:cdemu' (cached); generated: 2024-09-18T12:52:44Z solvables: 23
rpm-md repo 'updates' (cached); generated: 2024-10-11T03:33:46Z solvables: 28845
rpm-md repo 'fedora' (cached); generated: 2024-04-14T18:51:11Z solvables: 74881
rpm-md repo 'rpmfusion-free-updates-testing' (cached); generated: 2024-10-09T14:33:29Z solvables: 2
rpm-md repo 'rpmfusion-free-updates' (cached); generated: 2024-10-09T14:33:22Z solvables: 164
rpm-md repo 'rpmfusion-free' (cached); generated: 2024-04-20T12:11:51Z solvables: 422
rpm-md repo 'code' (cached); generated: 2024-10-11T21:14:29Z solvables: 459
rpm-md repo 'updates-archive' (cached); generated: 2024-10-10T02:51:28Z solvables: 47039
error: Packages not found: kernel-devel-6.9.12-205.fsync.fc40.x86_64, kernel-headers-6.9.12-205.fsync.fc40.x86_64

This is a brand new install of the latest Bazzite, with the only change to the repo being the addition of the VSCode RPM repo. There is no matching kernel-devel or kernel-headers for the fsync kernel-core available in any repos.

What did you expect to happen?

I expected there to be a kernel-devel and kernel-headers matching the default shipped kernel-core. On Bazzite this is a custom UBlue fsync version (apparently) and doesn't keep up with the latest from upstream Fedora, so UBlue would be responsible for providing the associated kernel-devel and kernel-headers to match their custom kernel-core.

Output of rpm-ostree status

State: idle
Deployments:
● ostree-image-signed:docker://ghcr.io/ublue-os/bazzite-nvidia-open:stable
Digest: sha256:f0a715ec44064ebfa14504db0c72b3790542ce3bcd3d7f3b9fb068e6a04197e7
Version: 40.20240930.0 (2024-09-30T15:25:37Z)
LayeredPackages: code grubby kernel-devel kernel-devel-uname-r sunshine zsh

ostree-image-signed:docker://ghcr.io/ublue-os/bazzite-nvidia-open:stable
Digest: sha256:f0a715ec44064ebfa14504db0c72b3790542ce3bcd3d7f3b9fb068e6a04197e7
Version: 40.20240930.0 (2024-09-30T15:25:37Z)
LayeredPackages: code grubby kernel-devel sunshine zsh

Hardware

$ inxi
CPU: 14-core (6-mt/8-st) 12th Gen Intel Core i9-12900H (-MST AMCP-)
speed/min/max: 771/400/4900:5000:3800 MHz
Kernel: 6.9.12-205.fsync.fc40.x86_64 x86_64 Up: 27m
Mem: 8.37/31.04 GiB (27.0%) Storage: 92.53 GiB/-4784 Procs: 460 Shell: Zsh
inxi: 3.3.34

$ cat /sys/devices/virtual/dmi/id/product_name
Predator PH315-55s

Extra information or context

The particulars of the device don't really matter, the problem is that there seems to be no repos to provide updates or support for the custom fsync kernel-core that ships with Bazzite.
It's reasonable that the kernel-header and kernel-devel matching the kernel-core aren't installed by default in the Bazzite ostree refs, and it's certainly easier to distribute the custom kernel-core purely by including it directly there rather than providing it via an RPM repo on copr or something, but some method of getting the kernel-devel and kernel-headers matching it are a minimum necessity of having a custom kernel in the first place.

[mtalexan]

Incidentally, this brought up basically the same topic and was closed almost immediately without any resolution whatso ever: #899

[mtalexan]

I'm not at all sure where the fsync kernel-core being used is coming from, but at a rough guess it looks like maybe this copr could add what's needed, presuming it keeps old enough builds that it would still have the 6.9 kernel (the oldest kernel version Fedora itself still supports). https://copr.fedorainfracloud.org/coprs/sentry/kernel-fsync/

[mtalexan]

Yeah ok. This is a clear initial comfig bug in UBlue. Any ostrees shipping with the fsync kernel require the copr repo to be included in the /etc/yum.repos.d/ files, and the default kernels from the fefora-updates.repo to be disabled. Exactly as the https://copr.fedorainfracloud.org/coprs/sentry/kernel-fsync/ explains. If you want to add a ujust command to toggle the Fedora kernels back on that could make sense so people could then install newer kernels without fsync patches, but the default has to be to use the matching kernel packages from the copr.

[mtalexan]

I see why the copr repo isn't installed and setup as the default. The copr only keeps the most recent build, none of the past ones, which means it has only the 6.11 kernel right now. The current latest UBlue/Bazzite-provided kernel-core-6.9.12-205.fsync.fc40.x86_64 is relatively old, and is not supported by the copr anymore (probably hasn't been for >1 year).

I guess if UBlue is going to ship with the fsync kernel and not keep to the bleeding edge like the upstream copr supports, it's going to have to figure out how to support a cached version of all the associated kernel packages so it has the older fsync versions available. Maybe UBlue needs to setup it's own copr just to host cached copies of things?

In the meantime, it's not possible to build kernel drivers for any fsync UBlue images as far as I can tell. The minimum necessary kernel-devel doesn't exist anymore and (apparently) wasn't cached before it was removed from the upstream copr. From all appearances, it looks like even the UBlue akmods layer isn't functional for fsync kernels (it doesn't actually appear to be functional for anything to be honest, relying on GitHub releases of container images to keep cached copies of the kernel RPMs but the most recent release being from April 2024 and therefore way out of date).

[mtalexan]

Looking more at the UBlue akmods, it appears there's actually nightly updates to the akmods container images that just aren't published as releases. So doing this would get the set of RPMs from the latest for the fsync kernel:

$ podman pull ghcr.io/ublue-os/fsync-kernel:40
...
$ podman container create --name=foo ghcr.io/ublue-os/fsync-kernel:40
...
$ podman container cp foo:/tmp/rpms $(pwd)
...
$ ls rpms/
kernel-6.11.2-201.fsync.fc40.x86_64.rpm        kernel-devel-matched-6.11.2-201.fsync.fc40.x86_64.rpm  kernel-modules-core-6.11.2-201.fsync.fc40.x86_64.rpm
kernel-core-6.11.2-201.fsync.fc40.x86_64.rpm   kernel-headers-6.11.2-201.fsync.fc40.x86_64.rpm        kernel-modules-extra-6.11.2-201.fsync.fc40.x86_64.rpm
kernel-devel-6.11.2-201.fsync.fc40.x86_64.rpm  kernel-modules-6.11.2-201.fsync.fc40.x86_64.rpm        kernel-uki-virt-6.11.2-201.fsync.fc40.x86_64.rpm

This is running into a similar issue as the copr though, it only has the latest available kernel version.