#155 NVIDIA driver and UEFI secure boot
Opened 11 months ago by ngompa. Modified 4 months ago

The BIOS/UEFI thread on devel@ reminded me that we still have a serious unsolved problem for Fedora desktops using Secure Boot UEFI: the NVIDIA driver packages we offer do not work in this configuration.

Fedora has three startup modes: classical BIOS, UEFI, and Secure Boot UEFI. In the first two modes, everything is fine and the driver works. However, when we start up in the third mode, we have a problem. Fedora's kernel is configured to block kernel modules that are not signed with a recognized certificate from being loaded. This makes the NVIDIA driver enablement we have completely ineffective in this scenario.

Ordinarily, we generally seem to be expecting people to go into their firmware settings to turn off Secure Boot for this, however I do not believe this is acceptable. Moreover, with Fedora now being preloaded on laptops that have NVIDIA GPUs on them, we need a solution for this where users will not wind up being frustrated when trying to enable this through GNOME Software.

To make matters even worse, this is entirely self-inflicted: distributions like openSUSE and Ubuntu have functioning NVIDIA binary driver packages with UEFI secure boot, and the NVIDIA blob installer script will automatically generate a local cert and key set, import it into the kernel, and build the kernel module on the machine with that key so that it will load.

Our driver packages are provided by @kwizart, who currently does not know how to solve this problem for users running Fedora on Secure Boot UEFI systems that cannot or will not turn off Secure Boot. I know it is possible to produce signed kernel module packages, but I do not know how to with the Koji build system (I know how to with the Open Build Service and manually with a local RPM build).

From my perspective, we need to solve this problem to provide a good user experience to most (if not all) Linux notebook PC users with mid-range or high-end notebook PCs, as those typically have NVIDIA GPUs.


Right now there's not a great answer for this, nor a particularly good path to one. The best suggestion I have if you really want this is to hack on making DKMS able to sign drivers with e.g. a yubikey[0], make the kernel update know not to switch to the updated kernel until that step has been done, and then put more work into getting the user to occasionally actually do that part without leaving the yubikey plugged in all the time.

[0] see https://github.com/vathpela/pesign/blob/wip/src/efikeygen.1.mdoc

I'm not part of the "boot team" but to me I would think that this needs to be handled like you would handle replacing your own distribution's version of shim with another one and therefore require adding its public key as a machine owner key (MOK).

As an example if you intend to use Fedora's shim to boot for example Arch/CentOS/Debian/OpenSuse/RHEL/Ubuntu kernels you must obtain the Secure Boot public key associated with that target OS and add that public key right.

So the same chain of trust that is required to cross-boot distribution could be established with 3rd party repositories as in the 3rd party repo would have to provide Fedora with their public key to be included, Then that 3rd party ( or consumers of that 3rd party resources ) could sign a kernel and or related modules for nvidia, virtualbox etc. right.

Javier might be able to shed some light what policy ( chain of trust ) the "boot team" ( then presumably Fedora as well ) has adopted in this regard as in only trust ourselves ( Fedora ), Trust RH ecosystem ( Fedora,RHEL/CentOS ) Trust major distributions ( Arch/CentOS/Debian/OpenSuse/RHEL/Ubuntu) etc. as well as establishing if this is an valid/viable approach and policy ( I'm no expert in this field ) but personally and if doable I think this is the correct way of keeping the chain of trust.

If we dont trust a 3rd party enough to keep their keys with the shim we ship then arguably we dont trust it to expose our end users to that/those 3rd parties repositories to begin with.

That said I also want to point out to the "boot team" that whatever policy RH has in this regard does not mean it has to be Fedora's policy as in Fedora's policy could be more relaxed ( or stricter ) than whatever RH has ( if it has any such policy ).

It goes without saying if there is an argument against including 3rd party repository keys in the MOK database due to lack of trust ( security concerns ) then the chain of trust is already broken and we should not be exposing that or those 3rd party repository to our end user base to begin with but the fact is that a consensus has been made to allow third party source inclusions from certain sources [1] and arguably if rpmfusion provides the distribution with keys those keys should be include based on that consensus and thus it's DKMS + nvidia installer process under secure boot should just work...

  1. https://fedoraproject.org/wiki/Workstation/Third_Party_Software_Repositories

Something I fundamentally do not understand is why we cannot have a TOFU-like flow similar to what we do for GPG keys for package signatures for MOK keys. There should be no reason that RPM Fusion couldn't include in their release package their SB keys to lay down on disk so that when a kmod is installed from their repository, we can handle prompting to import the keys when a kmod package includes kmods signed with it.

Yes, it's not quite as simple as package GPG checks, and we have things like akmods and DKMS as a source of complexity for this, but we could develop something so that the user can easily set up their own local key for their own kmods and stuff installed via akmods or DKMS can auto-sign with that. At least with akmods, we're already at the point where the package content has been verified at least once through the repository chain (via GPG), and as long as we have an easy flow for revocation, then I don't see why this is a problem to implement.

Presumably you would just push an update that removes the relevant trusted 3rd party source from key database should it no longer be trusted for whatever reason ( and or blacklist it ) to much rejoice of end users if and then when that happens.

What is missing here ( afaikt ) is a policy that describes the process of providing the public keys to be included in the MOK database and the directions to do so which should be a part of the policy that describes how 3rd party repository can be considered officially part of Fedora and thus reference to as an option in the installer and the DE's as an additional "Trusted" source of packages ( which may contain drivers/modules ) . ( RH must have an "Partner Program" of some sort which a proposal could be based upon ) and the trusted key(s) from rpmfusion since it has already been officially approved as a 3rd party provide.

I would be surprised if this is a much of an technical issue on our ( Fedora's behalf ) or warranty any changes in our infrastructure...

Something I fundamentally do not understand is why we cannot have a TOFU-like flow similar to what we do for GPG keys for package signatures for MOK keys. There should be no reason that RPM Fusion couldn't include in their release package their SB keys to lay down on disk so that when a kmod is installed from their repository, we can handle prompting to import the keys when a kmod package includes kmods signed with it.

Right now it's an interactive process to enroll a key with the firmware. There's the mokutil side, where the user specifies a password; and upon reboot the mokutil EFI program runs in the pre-boot environment to confirm this key's addition by interactive prompt for the same password.

There should be no reason that RPM Fusion couldn't include in their release package their SB keys to lay down on disk so that when a kmod is installed from their repository, we can handle prompting to import the keys when a kmod package includes kmods signed with it.

There is no way to do mechanical, automated key enrollment securely without a verifiable chain of trust, and the scenario you describe does not have that. Any mechanical, automated enrollment without a chain of trust to verify it means malware can enroll keys as well. Importantly, that also means malware could use our bootloader on machines installed with other distros or OSes to enroll its keys on those machines during the boot process.

So metalink/gpg signed repodata -> gpg signed package -> package signature verified -> install -> load -> verify and load MOK isn't enough of a trust chain?

That doesn't sound right at all.

You can't do this kind of analysis without starting out in the context of the threat model being discussed. Within the secure boot context, there's nothing that has verified and trusts any of the repodata, gpg signatures, or any of that. We start by trusting the system owner, who can make modifications to what we trust, either in the firmware config or with mokutil (both of which require some degree of physical presence) and we trust the keys the system vendor has enrolled in the firmware. Everything else has to chain from there.

So then what, we need to vet and trust an RPM Fusion key and load it as part of Fedora then?

And how do you vet that? What is their trust model? Who has access to use the key? What is the audit process? What are their security practices?

And how do you vet that? What is their trust model? Who has access to use the key? What is the audit process? What are their security practices?

How is that done for Fedora itself? Someone has to vet that to grant Fedora a key.

Yes, that is correct. Very few people have access to the Fedora key. Any proven packager can build a kernel, but it won't be signed.

Yes, that is correct. Very few people have access to the Fedora key. Any proven packager can build a kernel, but it won't be signed.

While I think that's broken (there should be a way to sign with a key that isn't trusted so that the process always works the same), what stops us from putting together a program to vet RPM Fusion in the same way? They run infrastructure just like ours except for the undocumented parts (which include this stuff).

There is a process to sign with a key that isn't trusted, that is the Red Hat Test Key. If you build a kernel on a local machine, or if you tried to build a kernel in koji, it would be signed with that key. But as the key is not trusted, you can't boot that kernel with secureboot turned on.

There is a process to sign with a key that isn't trusted, that is the Red Hat Test Key. If you build a kernel on a local machine, or if you tried to build a kernel in koji, it would be signed with that key. But as the key is not trusted, you can't boot that kernel with secureboot turned on.

Okay, good to know, so we do have that process.

Also worth noting, you are asking Fedora to hand over trust to a 3rd part explicitly to use that trust to sign a chunk of binary which is not signed AFAIK by the developer, and where code is not available for review. There is literally no audit path here without nvidia directly involved.

You can't do this kind of analysis without starting out in the context of the threat model being discussed. Within the secure boot context, there's nothing that has verified and trusts any of the repodata, gpg signatures, or any of that. We start by trusting the system owner, who can make modifications to what we trust, either in the firmware config or with mokutil (both of which require some degree of physical presence) and we trust the keys the system vendor has enrolled in the firmware. Everything else has to chain from there.

The threat model was presumably discussed in the discussion surrounding the proposal of allowing 3rd party inclusion in Fedora and taking into consideration when the proposal was approved. At that point the decision was made that approved 3rd party should be treated and trusted as if they were part of Fedora itself ( the end user sees no difference ). It should have been pretty apparent when allowing 3rd party inclusion that Fedora had no control over those repositories components and infrastructure right ( It should have assess those prior to their approval hence my reference to a policy which should exist )

That said once a machine has been compromised it has been compromised and the scope of the damage it can do or has done becomes irrelevant unless the intent is going down the rabbit hole of speculation and hypotheticals and the expectation of the end user having a computer forensic on a retainer to dive into the scope of the breach. The fact is the end result always remains the same for the end user regardless if he has a single OS installed or multiple OS installed, the end user has to wipe his disk(s) clean and perform a re-install otherwise he can never be assured whatever compromised his system is still not in place...

The threat model was presumably discussed in the discussion surrounding the proposal of allowing 3rd party inclusion in Fedora and taking into consideration when the proposal was approved.

Citation needed.

The fact is the end result always remains the same for the end user regardless if he has a single OS installed or multiple OS installed, the end user has to wipe his disk(s) clean and perform a re-install otherwise he can never be assured whatever compromised his system is still not in place...

You've missed the point: without Secure Boot enabled (and working) you can still be rooted after the reinstall.

I would be okay with rpm fusion doing their own signing, and then providing a document for users to review their trust model and choose whether or not they import the rpm fusion key. Then a user can go through that PITA once, and have it go, but it also means rpmfusion would need to build the modules there, and a user can't do their own module builds.

The threat model was presumably discussed in the discussion surrounding the proposal of allowing 3rd party inclusion in Fedora and taking into consideration when the proposal was approved.

Citation needed.

Back in the day we strongly opposed inclusion or reference to 3rd party repos due to legal and security related issues and now that has changed and we have change proposals workflow in place precisely to allow discussions like these to take place prior to acceptance right and I'm presuming ( read as I expect but not looked into if that actually took place ) when the change proposal to include those repositories listed here [1] the scope of that change ( including it's legal and security context and how it affected the users ) took place. If that discussion never took place, then obviously the decision to include and refer to those 3rd party repo needs to be revisited ( and removed should it come down to that ) and our process looked at to prevent such decisions be made in the future.

The fact is the end result always remains the same for the end user regardless if he has a single OS installed or multiple OS installed, the end user has to wipe his disk(s) clean and perform a re-install otherwise he can never be assured whatever compromised his system is still not in place...

You've missed the point: without Secure Boot enabled (and working) you can still be rooted after the reinstall.

Right but the same proposed action ( steps end users should take ) for the end user applies regardless if it's some approved 3rd party or it's Fedora.

JBG

  1. https://fedoraproject.org/wiki/Workstation/Third_Party_Software_Repositories

I would be okay with rpm fusion doing their own signing, and then providing a document for users to review their trust model and choose whether or not they import the rpm fusion key. Then a user can go through that PITA once, and have it go, but it also means rpmfusion would need to build the modules there, and a user can't do their own module builds.

If @kwizart can be told how to implement this, or at least if it can be documented somewhere how this works for Fedora at all, then I think this may be possible.

One potential clarification is that I'd like for it to be possible for RPM Fusion to have a trusted bot that watches for new Fedora kernels and would auto-rebuild kmod packages with the new kernel, since Koji can't do it by itself like other build systems can. Including that in the trust model would make it a lot saner to retire the akmod strategy RPM Fusion uses today for kernel modules.

I would be okay with rpm fusion doing their own signing, and then providing a document for users to review their trust model and choose whether or not they import the rpm fusion key. Then a user can go through that PITA once, and have it go, but it also means rpmfusion would need to build the modules there, and a user can't do their own module builds.

The secure boot process needs to be as transparent as it is if it's disabled so that does not help. ( the 3rd party needs to be included by default ) If it's usability suck or is filled with false positives end users will just turn it off like they have done with selinux for decades.

If we dont trust the 3rd party we should not be including/referring to it and offer application ( or drivers in this case ) that reside in it's repositories because the fact is end users aren't able to differentiate whether a component came from Fedora or from a 3rd party ( repo or application market place ) it's all just "Fedora" to him ( unless he manually has to jump through hoops to set it up ) and expecting (thus providing solutions ) that expect some kind of informed decisions about security related things from an end users is just ridiculous.

The end user places his trusts on the OS on doing the right thing for him.
Whether that turns out to be misplaced trust is entirely up to the OS, 3rd party repo or application marketplace and if anything fails Fedora as an OS will be blamed for it because that's the OS the end user is using.

I would be okay with rpm fusion doing their own signing, and then providing a document for users to review their trust model and choose whether or not they import the rpm fusion key. Then a user can go through that PITA once, and have it go, but it also means rpmfusion would need to build the modules there, and a user can't do their own module builds.

If @kwizart can be told how to implement this, or at least if it can be documented somewhere how this works for Fedora at all, then I think this may be possible.
One potential clarification is that I'd like for it to be possible for RPM Fusion to have a trusted bot that watches for new Fedora kernels and would auto-rebuild kmod packages with the new kernel, since Koji can't do it by itself like other build systems can. Including that in the trust model would make it a lot saner to retire the akmod strategy RPM Fusion uses today for kernel modules.

I don't know that I would have an issue with a bot doing the builds, but the whole value to this, is I don't have to. The document lays out what is happening, and each user can decide if they are comfortable with it or not.

We could probably swipe the license and notice metadata extension that SUSE uses for rpm-md to present that document, so that's something @dmach's team might be interested in to support this flow and bubble it up to GNOME Software.

I would be okay with rpm fusion doing their own signing, and then providing a document for users to review their trust model and choose whether or not they import the rpm fusion key. Then a user can go through that PITA once, and have it go, but it also means rpmfusion would need to build the modules there, and a user can't do their own module builds.

The secure boot process needs to be as transparent as it is if it's disabled so that does not help. ( the 3rd party needs to be included by default ) If it's usability suck or is filled with false positives end users will just turn it off like they have done with selinux for decades.

Which is likely what they are doing right now. This at least makes it a 1 time thing you do per machine, and manageable as opposed to something you must do every kernel update.

If we dont trust the 3rd party we should not be including/referring to it and offer application ( or drivers in this case ) that reside in it's repositories because the fact is end users aren't able to differentiate whether a component came from Fedora or from a 3rd party ( repo or application market place ) it's all just "Fedora" to him ( unless he manually has to jump through hoops to set it up ) and expecting (thus providing solutions ) that expect some kind of informed decisions about security related things from an end users is just ridiculous.

There are various levels of trust. Trusting an mpeg player is a bit different than trusting a kernel module. If "it's all just Fedora" to the user, then we have a problem. Third party repositories should not present as "just Fedora". The entire reason that they exist is because the things packaged there do not meet the qualifications of Fedora principles.

The end user places his trusts on the OS on doing the right thing for him.
Whether that turns out to be misplaced trust is entirely up to the OS, 3rd party repo or application marketplace and if anything fails Fedora as an OS will be blamed for it because that's the OS the end user is using.

As the person who builds signed kernels, I do not extend my trust to the 3rd party repo, and to nvidia (which is the extent of the trust chain in this scenario). This is not just "do I trust @kwizart to do the right thing", but do I trust kwizart, the maintainers of the signing key and the infrastructure that it depends upon within rpmfusion, the developers at nvidia, and whatever infrastructure they have in place for their binary since we can't see the source that is used to build it. Even if I am personally willing to extend my trust that far for my machines, I certainly couldn't say so on behalf of all Fedora users.

I'm a little lost in this discussion without a clear definitions:

  • What benefits do Fedora users gain by enabling secure boot
  • What do we need to do to avoid our secure boot infrastructure being used to attack other operating systems

[ https://mjg59.dreamwidth.org/12368.html has some background on the original thinking behind disabling unsigned modules - which is apparently the key difference between us and other distros that have less issues here ]

@pjones : can you explain:

You've missed the point: without Secure Boot enabled (and working) you can still be rooted after the reinstall.

What do we need to do to avoid our secure boot infrastructure being used to attack other operating systems

Keep the chain of trust as short as possible as in limit it only to Fedora ( Or Fedora/Centos/RHEL for cross-booting support strictly in the RH ecosystem which RH controls) but that of course comes at the prices of excluding 3rd party repositories and other OS ( cross-booting becomes unsupported, that is if it's even supported today )

Bottom line the more you add to a chain of trust, the weaker it becomes.

@pjones : can you explain:

You've missed the point: without Secure Boot enabled (and working) you can still be rooted after the reinstall.

So, the straightforward thing to do if you have a Secure Boot breakout is obviously to use it as a bootloader and load modified version of the OS - i.e. backdoor the kmod verifier while loading the kernel. This what's known as a "bootkit", and that kind of backdoor is really the simplest variety - it's not hard to find enough code to easily implement that much by googling. This is clearly pretty bad, but it's also not limited to one OS - you need different backdoor code for each target, but once you've found a bootloader or kernel that doesn't properly check signatures, or trusts a signature that it should not, or is suceptable to the right kind of exploit, you can just install that in front of other OSes as the starting point.

But that's not the worst thing attackers have done. If the attacker is at all sophisticated, they can additionally replace UEFI APIs for disk access, boot variables, and even the system firmware boot selection menus. If they do that, then they can make it appears as though you've booted install media when you've actually still got the same backdoor, and it's trivial to install the exploit again during the re-install. This does not require all that much work to do poorly as a proof of concept; how much work the attacker puts into it developing the bootkit mostly determines how hard it is to detect that they've done this and how hard it is to fix it once you've noticed.

This is what attackers will do when they don't have some other vulnerability available, like flash not being properly locked. The "hackingteam" bootkit that was leaked in 2015 includes a module that will re-link your system firmware with their backdoor in the loader - but on some classes of machines with the vulnerability that allows them to do that, they still need to be able to run pre-boot code to actually modify it.

MokManager/mokutil is a compromise between being more secure against this kind of attack and being more practical - we allow you to enroll keys as the owner of the machine, but you need some degree of physical access to do so. The result is that if there is some weakness in what you've signed and are loading with mok, a very tenacious attacker will still be able to make progress on your system, but the attack won't be very portable to other systems.

Having said all that - can we please not argue all of the decisions of Fedora 18? We discussed this to death at the time, and I've got enough going on right now without doing it again.

  1. Protection from: firmware implants, persistent attacker controlled system management modules, various kinds of OS Loader malware, a compromises kernel, including any kernel module being used as malware or as part of a chain of exploits to achieve ultimate ownership of the system. Numerous examples exist in the wild.
  2. Keep control of the Fedora (private) signing key.
  3. Without UEFI Secure Boot enabled and working, a number of locations exist that are not subject to being reset by a typical clean OS install. This includes the EFI system partition, and the chip the firmware itself is located on. Once the system is owned, it can be indistinguishable from not being owned using any familiar friendly UI, i.e. the firmware setup UI is not reliable.

Once you've disabled UEFI Secure Boot, a compromised (fake) kernel module can drop a malicious system management module onto the system, which in turn can make the flash where the firmware itself lives, writeable. Persistent malware achieved. Or even the attacker's own signing key, so they can run whatever they want. This system is now properly owned in a way that a clean install can't fix. Even if you go to the uncommon trouble of secure erasing the drive first. And this could all start with a remote attack from a web site using a browser, it doesn't require physical access ala Evil Maid attack.

And now imagine this computer happens to have possibily important private keys on it. And next wonder how you're going to get out of this hell hole without fire.

It's reasonable to consider ways to make it easier for users to install drivers that they vouch for being from a trusted source, however unqualified that is, and helping them sign those modules and enroll the key. But I'm a little skeptical of how it works for Fedora to sign 3rd party keys: it's not just a matter of authenticating current practice but verifying it on an on-going basis. If I thought that making this easier means few intentionally SB disabled systems, i'd be more supportive. I get super grouchy when I read folks on support channels and lists advocating just disable UEFI Secure Boot.

Issue #116 applies here too.

Having said all that - can we please not argue all of the decisions of Fedora 18? We discussed this to death at the time, and I've got enough going on right now without doing it again.

I don't want to argue all of that all over again. If anything, I still agree with the decisions @mjg59 did back then. I just want us to come up with something where we don't frustrate users who are trying to make their hardware work, especially now that we're moving into having Fedora preloaded on computers.

Having said all that - can we please not argue all of the decisions of Fedora 18? We discussed this to death at the time, and I've got enough going on right now without doing it again.

I don't want to argue all of that all over again. If anything, I still agree with the decisions @mjg59 did back then. I just want us to come up with something where we don't frustrate users who are trying to make their hardware work, especially now that we're moving into having Fedora preloaded on computers.

I suggested something that's better than what we've got in https://pagure.io/fedora-workstation/issue/155#comment-664899 . It doesn't seem like you've replied to it.

Having said all that - can we please not argue all of the decisions of Fedora 18? We discussed this to death at the time, and I've got enough going on right now without doing it again.
I don't want to argue all of that all over again. If anything, I still agree with the decisions @mjg59 did back then. I just want us to come up with something where we don't frustrate users who are trying to make their hardware work, especially now that we're moving into having Fedora preloaded on computers.

I suggested something that's better than what we've got in https://pagure.io/fedora-workstation/issue/155#comment-664899 . It doesn't seem like you've replied to it.

Sorry, I didn't notice that. That option is somewhat doable if we could assume people had YubiKeys. Another way that could work would be to do this with a TPM device and leverage that to essentially tie the signing to the device, but that doesn't help with the "conscious act" thing you're going for here.

If akmods was tweaked and we had a GUI frontend for it that helped with doing that process properly, it might work, though. However, I don't want to do this with DKMS because it doesn't produce a trackable result (i.e. a kmod RPM that's installed on the system).

Having said all that - can we please not argue all of the decisions of Fedora 18? We discussed this to death at the time, and I've got enough going on right now without doing it again.
I don't want to argue all of that all over again. If anything, I still agree with the decisions @mjg59 did back then. I just want us to come up with something where we don't frustrate users who are trying to make their hardware work, especially now that we're moving into having Fedora preloaded on computers.

I suggested something that's better than what we've got in https://pagure.io/fedora-workstation/issue/155#comment-664899 . It doesn't seem like you've replied to it.

Sorry, I didn't notice that. That option is somewhat doable if we could assume people had YubiKeys. Another way that could work would be to do this with a TPM device and leverage that to essentially tie the signing to the device, but that doesn't help with the "conscious act" thing you're going for here.
If akmods was tweaked and we had a GUI frontend for it that helped with doing that process properly, it might work, though. However, I don't want to do this with DKMS because it doesn't produce a trackable result (i.e. a kmod RPM that's installed on the system).

Which brings us back to, there is nothing that stops rpm fusion, or any other 3rd party from creating their own signing infrastructure, and detailing a step for users to put their single key into the trusted ring on their system. This has been the case since we first implemented secure boot. I am guessing the only thing stopping them is the assumption of responsibility there lies squarely on the repository should things be mismanaged.

Actually, from what I can tell, it's that nobody knows how to sign kernel modules for kmod packaging.

Actually, from what I can tell, it's that nobody knows how to sign kernel modules for kmod packaging.

pesign 113, which is in f33 currently, can sign kernel modules.

One way to solve this would be to move the nvidia open source kernel module building to Fedora's koji and build it together with the kernel. This would (1) make sure it exactly matches the kernel version, and (2) ensure it gets signed with Fedora key.

The main issue I see with this is that it's occasionally going to hold back kernel updates if the interface changes and the module doesn't build any more. Sadly, I don't think we have many alternative options if we want to make this work well for nvidia / Lenovo laptop users and ensure they always have access to a nvidia driver :(

I'd rather not have it bundled in the kernel tree. If we're going to do that, let's figure out how to build kmod packages in Fedora Koji instead.

One way to solve this would be to move the nvidia open source kernel module building to Fedora's koji and build it together with the kernel. This would (1) make sure it exactly matches the kernel version, and (2) ensure it gets signed with Fedora key.

You can't really build it "together" with the kernel, but it would be very easy to auto bump/build when a successful kernel build completes on a stable Fedora branch. Rawhide, you can try, there is no guarantee that it would succeed though, and Nvidia doesn't actually care. Their "policy" is no support for pre-release kernels. Typically this means they will release a fixed build shortly after Linus releases a .0

The main issue I see with this is that it's occasionally going to hold back kernel updates if the interface changes and the module doesn't build any more. Sadly, I don't think we have many alternative options if we want to make this work well for nvidia / Lenovo laptop users and ensure they always have access to a nvidia driver :(

Our policy was that we would not hold back kernel updates on the failing of nvidia, and when 3rd party was discussed there was supposed to be something that handled not booting from a new kernel until nvidia was available for users running it. In practice, we don't rebase until 3-4 weeks after Linus does a .0 update, and nvidia is working by then. In the rare occasion that a stable update has broken things, we tend to revert the patch that does so, provided it isn't security related.

I'd rather not have it bundled in the kernel tree. If we're going to do that, let's figure out how to build kmod packages in Fedora Koji instead.

No, it cannot be bundled in the kernel tree. And we really do not want to get into the business of kmod packages in Fedora proper. We could come up with a process to build select kmod packages on Fedora infrastructure, sure. But that is going to have to be much more tightly controlled than the standard "add a package to Fedora process". For instance, any proven packager can technically commit to the kernel or grub, or any of the signed packages, but if they build it, it will not be signed. Signature access is very intentionally tightly controlled.

I'd rather not have it bundled in the kernel tree. If we're going to do that, let's figure out how to build kmod packages in Fedora Koji instead.

Sure, that's really up to the kernel maintainers what way they prefer it.

The main thing is that I think we (Fedora) just need to take over the responsibility of rebuilding the nvidia kernel module. If it's in rpmfusion we have no way to ensure it gets updated/rebuilt in lock step together with kernel updates and correctly signed.

Sure, that's really up to the kernel maintainers what way they prefer it.
The main thing is that I think we (Fedora) just need to take over the responsibility of rebuilding the nvidia kernel module. If it's in rpmfusion we have no way to ensure it gets updated/rebuilt in lock step together with kernel updates and correctly signed.

It really needs to be special cased. The nvidia package goes against the Fedora principals, and can't really be a package in Fedora proper, so it would need to be built on Fedora koji, but still distributed through a 3rd party repo.

It really needs to be special cased. The nvidia package goes against the Fedora principals, and can't really be a package in Fedora proper, so it would need to be built on Fedora koji, but still distributed through a 3rd party repo.

That sounds like a good plan to me. Thanks, @jforbes! :)

If it's a separate repo, that may actually offer a functional advantage: We could have a slightly different repo structure than what regular Fedora repos have, and instead of just shipping the latest package, we could ship all the versions we've ever built in it, so that users can easily install the correct module for the exact version of kernel they have installed.

This is similar to what openSUSE does, too. The NVIDIA driver is built in their build system, and then NVIDIA hosts it themselves on download.nvidia.com for openSUSE users to consume.

Cf. https://en.opensuse.org/SDB:NVIDIA_drivers

Metadata Update from @chrismurphy:
- Issue untagged with: meeting-request
- Issue tagged with: meeting

11 months ago

Metadata Update from @chrismurphy:
- Issue untagged with: meeting

10 months ago

@ngompa can you update us on the status of this issue?

Login to comment on this ticket.

Metadata