There seems at least one thread [1] on the mailing list that talks about getting kernel modules built on Atomic, but nothing that I could find has been created here to figure out the best path forward on supporting kernel modules.
The use case that I am personally trying to fix is getting WireGuard working on Atomic. I maintain the RHEL/CentOS/Fedora Copr for WireGuard [2] and DKMS works pretty well for providing the WireGuard kernel module on non-atomic based installs. I have had some WireGuard users try and use the Copr RPMs on Atomic Host which resulted in failure. One of them was manually copying the .ko file over to get things working [3] and this isn't a really good solution long term.
I now find myself in the same situation where I would like to use Wireguard on Atomic, so I am trying to figure out how to come up with a short term solution to this issue as the WireGuard devs work on getting WG pushed upstream into the mainline kernel. Even if that does happen somewhat soon, this problem will still impact others that need third party kernel modules working such as Nvidia drivers or third party monitoring services such as Sysdig [4]
I started looking at creating a system container / systemd unit file that builds a new container on boot if a new kernel is detected. The current idea is it would do the following:
1) Check to see if a new kernel module is needed 2) Bind mount the kernel-devel source from the Atomic Host into the container 3) Check out the current WireGuard source and build the module (manually or via DKMS) 4) Enable's the newly built kernel module via system container [5]
kernel-devel
Some issues with this idea come to light quickly as the current Fedora Atomic Host snapshot has mismatching kernel and kernel-devel packages so building the module this way might not always work. I haven't fully figured out if this is the right path forward on getting kernel modules built. It seems that some folks have done somewhat similar things for CoreOS to get kernel module support for Nvidia based GPUs. [6] [7]
kernel
I turned to IRC to get some pointers on my above idea to solve this problem and @walters recommended that I start an issue.
[1] https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2017-November/msg00041.html [2] https://copr.fedorainfracloud.org/coprs/jdoss/wireguard/ [3] https://lists.zx2c4.com/pipermail/wireguard/2017-August/001656.html [4] https://sysdig.com/blog/dig-into-atomic-host/ [5] https://github.com/giuseppe/hellomod [6] https://github.com/ryanolson/CoreOS-GPU [7] https://github.com/src-d/coreos-nvidia
Some issues with this idea come to light quickly as the current Fedora Atomic Host snapshot has mismatching kernel and kernel-devel packages so building the module this way might not always work.
A possible workaround here is to use the koji cli inside the container to run koji download-build --arch=x86_64 $(rpm -q --qf '%{N}-%{V}-%{R}\n' kernel) where the rpm -q command is querying the host's rpm db (or find some other way to pass that information in).
koji download-build --arch=x86_64 $(rpm -q --qf '%{N}-%{V}-%{R}\n' kernel)
rpm -q
The core thing here is that since there's no kABI, you are not guaranteed that a module will work as the host evolves. This is true of non-Atomic Fedora systems today, and has been true basically forever - nothing new here. What changes is how the host is built/delivered.
Mechanically there's two parts to this. First, there's the question of where the module gets built. That in turn affects how it's deployed (the second issue).
DKMS builds it on the client. There are also "akmod" variants that have the modules pre-built for multiple kernels.
Particularly "at scale", I think it makes the most sense to pre-build kernel modules rather than doing it per-host. If you set up a COPR or whatever, you can query what kernels "will be" available in the future by looking at Koji builds. If your built kernel module package then has a Requires on the exact kernel version it was built for, then rpm-ostree client side layering should automatically select the right one as the primary releases pick up updated kernels.
Requires
That's going to work fairly well today because FAH/Silverblue etc. do not carry distinct kernels.
Now everything I just said is fundamentally distinct from making a DKMS-style system work with rpm-ostree which it seems is what you were looking at. For that...let's perhaps talk in https://github.com/projectatomic/rpm-ostree/issues/1091 ?
Just to cross-link there's now work targeted mostly at the nVidia driver for Silverblue but it's obviously somewhat generic to akmods: https://bugzilla.redhat.com/show_bug.cgi?id=1667014
https://pagure.io/workstation-ostree-config/pull-request/119
Login to comment on this ticket.