#4 kmods package structure
Closed 2 months ago by pjgeorg. Opened 4 months ago by pjgeorg.

Before adding the first kmods, we should discuss the general structure we want to use for kmod packages.

There are several different approaches we could take. I already looked at some other projects that have already been packaging kmods for some time. One of the major differences often is whether you provide a single package or split it into a userland and kernel-module package.

I currently favor using a very similar approach to rpmfusion. See https://rpmfusion.org/Packaging/KernelModules/Kmods2 for details. But without using kmodtool as it seems to be used to handle features we do not need in EL anyway.

With this approach providing a kernel module, e.g. the isci in-kernel module, would look like this:

  1. Write an isci-driver.spec file, which produces a isci-driver-%version.el8.%arch.rpm.

  2. Write an isci-kmod.spec file, which produces a kmod-isci-%version.el8.%arch.rpm and kmod-isci-%kernel_version-%version.el8.%arch.rpm. Where the first is only a metapackage and the second actually containg the .ko.

The user would then install this driver via dnf install isci-driver.

I just created two projects, isci-driver and isci-kmod, here on pagure to show what this means in practice. It's always better to have an example. I'll delete these two projects once agreed on package structure. There might be some errors concerning metadata, please ignore these.

Open question concerning the userland package:

  • Do we have to add a license file for in-kernel modules? These packages have the same license as the kernel, for which the license file is bundled with the kernel-core rpm.

There is a lot to ponder there.

On the license side of things, the current Fedora guidelines require a license to be installed for each package. I'd be inclined to follow that pattern so we're in the pattern should we add any non-GPL kmods down the line.

There is a lot to ponder there.

Yes, there is. We should take the time needed to set it up poperly once. And probably also document it.

On the license side of things, the current Fedora guidelines require a license to be installed for each package. I'd be inclined to follow that pattern so we're in the pattern should we add any non-GPL kmods down the line.

Following the Fedora guidelines my understanding is that in case of splitting into a userland and kernel-module package (i.e. two spec files) we have to include the license file in both packages. The only exception from this rule is for subpackages. A package depending on another package including the license file is not sufficient. Both packages must be the result of the same source RPM / spec file.

I see this as an argument to not split a driver into two packages.

The other option is to still use a similar approach as rpmfusion, but have the kernel module as a subpackage and the documentation (and other userland stuff) in the base package.
Disadvantage of this approach is that we are rebuilding the base package everytime we have to rebuild for a new kernel release, although the documentation (and other userland stuff) does not depend on the kernel version. It seems that avoiding this issue is the main argument for rpmfusion to do the split into two packages in the first place.

Hence we might also just use one single package (with no subpackages) including the license and kernel module. In most cases userland tools associated with this kernel module are bundled in an extra package anyway. I.e. for wireguard there is wireguard-tools, which is already in EPEL.

For an example of using a single package I've been looking at kmod-kvdo which is included in EL8. Surprisingly this package does not follow the Fedora guidelines: No license file is included.

I will try to write a kmod-isci.spec producing a single kmod-isci-%version.el8.%arch.rpm later today and push to pagure so we can then compare both approaches. I hope this helps the discussion.

I will try to write a kmod-isci.spec producing a single kmod-isci-%version.el8.%arch.rpm later today and push to pagure so we can then compare both approaches. I hope this helps the discussion.

Please find the example here.

I'm wondering if we can loop in our friends at ELRepo to see what their thoughts on this are? They've got yet another packaging setup and I'd be curious to hear their thoughts.

I think it's definitely worth reaching out to the people at ELRepo to hear their opinion.

I guess it's best to reach out to them via the elrepo-devel mailing list?
I'll subscribe to the mailing list and then post there to ask for thoughts linking to this issue here.

Afaik their approach is similar to the single package kmod-isci.

In a parallel vein, I've opened up https://bugzilla.redhat.com/show_bug.cgi?id=1976971 to see about getting the list of supported IDs into the packages by default. That should help make it easier if someone (probably me) wants to write up a dnf plugin to find kmods that match your installed hardware.

Phil Perry wrote:

I'm unable to provide input at pagure.io as I don't have an account, but
happy to help here. My apologies for the lengthy email.

I guess you guys have some fundamental decisions to make early on which
will largely dictate the path you then follow. For us, back in the early
days of RHEL5, we investigated various methods of maintaining and
packaging out of tree kernel modules before settling on the current kmod
model. Jon Masters was instrumental in helping us with that, having read
his white paper and other documentation on kABI tracking kmods and the
RHEL Driver Update Programme, and he was very supportive of our early
work. We also considered dkms and other solutions at the time but kABI
tracking kmods were seen as the Red Hat approved way to package external
modules for RHEL.

In theory, using kmods for RHEL is a no brainer. You have a stable kABI,
build the package once and it works for the 10 year lifespan of RHEL. In
reality, that's not the case. Only a select number of kernel symbols are
on the stable list, and guaranteed not to change. In reality, I'm yet to
see any kernel module that uses only symbols from the kABI stable
list, so every kmod package has the potential for kABI breakage, thus
requiring a rebuild against a newer kernel.

How often do things break? quite often, especially in the early days of
the full support cycle phase where Red Hat is more actively backporting
stuff into the RHEL kernel - in other words, the first 5 years which is
the support cycle phase that Stream corresponds with. To give a recent
example, going from RHEL 8.3 -> 8.4, there were ~65 internal kernel
releases (-240 to -305), and during those 65 releases, every single kmod
package that elrepo ships broke (~50 in total) and required a rebuild.
For us, this involved a mass rebuild of all packages against the new 8.4
release kernel, many packages requiring source code updates and a
handful of packages required further fixes to build. In Stream you will
see many (all?) of those kernel updates land in your git repos, so I
guess you will be rebuilding far more regularly. You will experience
that breakage in a constant flow bit by bit whereas we see it all in one
big hit at point release time. I think it's easier for us as we can
assume things will just work for 6 months whereas you will never know
what's going to break from one week to the next, other than it will
break at some point.

So you will need to make some fundamental decisions. Lets look at your
kmod-isci as an example of the kind of things you will need to consider:

  1. What source code will you use for your package. The isci is an in
    kernel module which Red Hat does not build in RHEL. You could take the
    source from the Stream kernel, but it's unlikely to be maintained as RH
    don't build it - why would they backport updates / fixes etc to an
    unsupported module? Or you could backport newer maintained code from
    upstream (kernel.org). Which option you choose may depend on the driver
    in question so may change from package to package. How will you handle
    backporting of security fixes/updates in either scenario?

  2. What packaging model will you use. Given the kABI churn in the Stream
    kernel, is kmod the best format? Have you considered akmods or dkms as
    alternatives?

  3. Once you have your chosen source code and packaging models, how often
    will you update and rebuild packages? Will you rebuild against every
    kernel release or will you only rebuild upon kABI breakage (assuming you
    use kmods). How will you determine/measure breakage? How will you test
    the kernel module without appropriate hardware to know if it works or
    not? How will you track source code changes if you are only rebuilding
    on breakage. How will you indicate which kernel version a package is
    intended to be used with / is built against?

  4. You will need to make and ship Driver Update Disk images for storage
    modules like isci. You may need to make new DUD images for every compose
    as I believe Stream makes new installation images for every new compose??

You mention "low maintenance" above. IMHO none of this is low
maintenance, which is a large part of the reason elrepo is unable to
support Stream. For RHEL, the choice of kmods is obvious given the
stable kABI. Likewise on fedora, with no stable kABI, akmods (or dkms)
make far more sense. Stream is kind of somewhere in the middle. Even if
you went the akmod route, you'd still need to build binary driver
packages for your driver disk images for installations (I have no idea
if akmods can be used in DUD images).

  1. How will you handle Secure Boot signing? As I understand you will be
    unable to sign with the official RH/CentOS project key, you may want to
    generate your own SIG signing key and have users import that to their
    MOK list as elrepo does. We developed our own process for this - I have
    no idea what process Red Hat uses to sign their kmod modules.

In terms of packaging, we have largely followed Red Hat's lead and used
their kmod packages as examples for each RHEL release. These have
changed markedly from RHEL 5 to 6 to 7 to 8. I have no idea what to
expect for RHEL9 yet, but historically it has always been a massive
piece of work for us to develop for a new release. If you are using a
model not too dissimilar to us, I am hoping you will have already ironed
out many of the bugs for us! For example, kmod and dracut have always
been problematic for us. What tends to happen is Red Hat forks packages
from fedora to go into a RHEL beta (or now Stream). Then 3+ years passes
and the same process happens again for the next RHEL release. In those
intervening years, the fedora developers invariably manage to break all
the bits that kABI-tracking kmods depend upon as (a) they don't
care/understand about the kABI in RHEL, and (b) even if they did care,
they have no way to test their changes haven't inadvertently broken it.
So when you start developing for 9, expect kABI tracking not to work,
and expect to spend a long time trying to figure out what broke and
when, as you'll be looking back through 3+ years of commits. We normally
start developing against the initial RHEL beta and aim to have things
fixed by the x.1 point release. We often run our own internally patched
versions for the first 6-12 months until we can get things fixed in
RHEL. My advise would be to start developing for Stream 9 as soon as
possible and test the pants out of everything kABI related (we regularly
build multiple kernel variants from the beta kernel to simulate testing
kmods against kernel updates). It would be a huge positive to us if you
guys have done this development work in Stream before RHEL 9 ships :-)

Regarding packaging, our kmod packages generally only contain kernel
module(s), and associated conf files and documentation/licence files
etc. Any firmwares or other binary/userland files are generally shipped
in separate packages which can be linked with Requires as appropriate.
Of course you are constrained by a whole load of limitations which
simply do not concern us, limiting what you can package and how you
should package it. 12+ years ago, elrepo was founded out of frustration
of not being able to contribute to the CentOS project. CentOS had the
Dasha project (google still has a couple references to it [1,2]) but it
quickly became apparent it was a non-starter, hence the birth of elrepo.
Hopefully you guys can make a better success of the kmod SIG for Stream.

Best wishes,

Phil

[1] https://people.centos.org/arrfab/Events/Fosdem-2009/centos-introduction.pdf

[2] https://www.spinics.net/lists/centos-docs/msg01336.html

I've been working on the WireGuard kmod packaging off and on for a while now using the official Red Hat kmod packaging macros.

You can see what that looks like here: https://pagure.io/wireguard-kmod-el-rpm/blob/master/f/wireguard-kmod.spec

There are certainly interesting bits in there!

Is there a bugzilla I can follow for the comment on line 1?

I don't recall one, it's been a while...

Looks like nvidia has their kmod specs posted as well for possible interest : https://github.com/nvidia/yum-packaging-precompiled-kmod

Some comments from my side on Phil Perry's input, espcially on the raised fundamental decisions:

  1. What source code will you use for your package. The isci is an in
    kernel module which Red Hat does not build in RHEL. You could take the
    source from the Stream kernel, but it's unlikely to be maintained as RH
    don't build it - why would they backport updates / fixes etc to an
    unsupported module? Or you could backport newer maintained code from
    upstream (kernel.org). Which option you choose may depend on the driver
    in question so may change from package to package. How will you handle
    backporting of security fixes/updates in either scenario?

I prefer taking the source from the Stream kernel. As Phil mentioned, it is unlikely to be maintained by RH, but it turns out that RH sometimes applies patches to unmaintained drivers. E.g. backporting patches that apply to various drivers.

But it indeed depends on the driver in question.

For backports I suggest using patches for CentOS Stream 8. For CentOS Stream 9 I hope that we might even be able to push our changes for drivers not maintained by RH to the kernel sources (https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9).

Overall I hope that with the kernel development for CentOS Stream 9 being more open, our work will be easier. E.g. detecting required backports due to other stuff backported by RH.

  1. What packaging model will you use. Given the kABI churn in the Stream
    kernel, is kmod the best format? Have you considered akmods or dkms as
    alternatives?

In the last meeting we decided to work on kmods only for now. I think it is still the best format for CentOS Stream. Even dkms packages have to be tested for every kernel release, otherwise users might run into unexpected compile errors when installing packages.

  1. Once you have your chosen source code and packaging models, how often
    will you update and rebuild packages? Will you rebuild against every
    kernel release or will you only rebuild upon kABI breakage (assuming you
    use kmods). How will you determine/measure breakage? How will you test
    the kernel module without appropriate hardware to know if it works or
    not? How will you track source code changes if you are only rebuilding
    on breakage. How will you indicate which kernel version a package is
    intended to be used with / is built against?

I plan and suggest to rebuild packages for every kernel release.

Testing the kernel module without hardware is indeed an issue. We all probably have some hardware to test, but of course not for all drivers we might provide. However I'd argue that being a pure community driven project it is sufficient to rely on user's to report bugs for hardware we do not have access to.

  1. You will need to make and ship Driver Update Disk images for storage
    modules like isci. You may need to make new DUD images for every compose
    as I believe Stream makes new installation images for every new compose??

I have to say that I never used DUD images (I use PXE boot for almost every system I maintain). Hence I have no idea how DUD images are generated. But I can look into it.

  1. How will you handle Secure Boot signing? As I understand you will be
    unable to sign with the official RH/CentOS project key, you may want to
    generate your own SIG signing key and have users import that to their
    MOK list as elrepo does. We developed our own process for this - I have
    no idea what process Red Hat uses to sign their kmod modules.

So far we planned to use our own SIG signing key and have users import that to their MOK list. Other SIGs do not seem to be happy with this approach, hence they try to establish a different approach. See https://pagure.io/centos-infra/issue/307 for details. Not a top priority on my side.

We now have a list of various different kmod packages we can have a closer look at. This is great.

Most of these are similar in terms of the rough package structure: Using a single package that includes kernel module(s), associated configuration files, documentation and license files.
Other components (e.g. associated userspace tools, firmware, etc.) are included in seperate packages (with appropriate Requires).

Can we already agree on going this way for the kmods SIG as well?
Might be good to have this decision before we have a closer look at the different available kmod packages and agree on implementation details.

There has been some discussion in https://pagure.io/kmod-wireguard/pull-request/1 and it seems like for kmod-wireguard we are almost done. Once we are done there I'll move the repo to the centos-sig-kmods namespace here to have them all in one place.

In the meantime I had a look at kmod-isci again (as a first example for building an in-kernel module disabled by RH). Having in mind the discussions about kmod-wireguard, especially about allowing easy (i.e. automatic) rebuild for new kernel release, I changed how the source code of isci is handled. Instead of taking the source code from every Stream kernel release I took it from linux-upstream v4.18 and applied appropriate backports. This change requires to store the source code somewhere, preferrebly here on pagure as well. But it allows to actually use the same source code and compile for any EL 8 kernel released so far. Required changes to the spec file to rebuild for a new kernel release can be accomplished by a single rpmdev-bumpspec command (see [1] for some notes).

For comparison you can find the old kmod-isci (taking source from Stream) here: https://pagure.io/centos-sig-kmods/kmod-isci
The new version (taking source from linux-upstream) can be found here: https://pagure.io/kmod-isci

Note that the repository for the new version has two branches:
c8s is used to store the source code for CentOS Stream 8 (i.e. the source-git? [2])
c8s-sig-kmods is used to store exploded SRPM that shall later be pushed git git.centos.org (i.e. the dist-git? [2])

I keep the c8s-sig-kmods branch on pagure as well to allow opening PRs against it. PRs are not available on git.centos.org. This is why I general suggest to have all packages hosted here and only push to git.centos.org once we want to release a new version. But this is OT here.

Please let me know what you think about this newer version of kmod-isci. In case the new version is preferred I'll move this version to the centos-sig-kmods namespace here replacing the old one.

[1] https://pagure.io/kmod-isci/pull-request/1 needs to be merged first to allow this.

[2] I'm never sure how these are called, source-git, dist-git, ... Too many different names for me.

Metadata Update from @pjgeorg:
- Issue status updated to: Closed (was: Open)

2 months ago

Login to comment on this ticket.

Metadata