cc: @johannbg, @javierm, @jforbes

Right now there's not a great answer for this, nor a particularly good path to one. The best suggestion I have if you really want this is to hack on making DKMS able to sign drivers with e.g. a yubikey[0], make the kernel update know not to switch to the updated kernel until that step has been done, and then put more work into getting the user to occasionally actually do that part without leaving the yubikey plugged in all the time.

[0] see https://github.com/vathpela/pesign/blob/wip/src/efikeygen.1.mdoc

I'm not part of the "boot team" but to me I would think that this needs to be handled like you would handle replacing your own distribution's version of shim with another one and therefore require adding its public key as a machine owner key (MOK).

As an example if you intend to use Fedora's shim to boot for example Arch/CentOS/Debian/OpenSuse/RHEL/Ubuntu kernels you must obtain the Secure Boot public key associated with that target OS and add that public key right.

So the same chain of trust that is required to cross-boot distribution could be established with 3rd party repositories as in the 3rd party repo would have to provide Fedora with their public key to be included, Then that 3rd party ( or consumers of that 3rd party resources ) could sign a kernel and or related modules for nvidia, virtualbox etc. right.

Javier might be able to shed some light what policy ( chain of trust ) the "boot team" ( then presumably Fedora as well ) has adopted in this regard as in only trust ourselves ( Fedora ), Trust RH ecosystem ( Fedora,RHEL/CentOS ) Trust major distributions ( Arch/CentOS/Debian/OpenSuse/RHEL/Ubuntu) etc. as well as establishing if this is an valid/viable approach and policy ( I'm no expert in this field ) but personally and if doable I think this is the correct way of keeping the chain of trust.

If we dont trust a 3rd party enough to keep their keys with the shim we ship then arguably we dont trust it to expose our end users to that/those 3rd parties repositories to begin with.

That said I also want to point out to the "boot team" that whatever policy RH has in this regard does not mean it has to be Fedora's policy as in Fedora's policy could be more relaxed ( or stricter ) than whatever RH has ( if it has any such policy ).

It goes without saying if there is an argument against including 3rd party repository keys in the MOK database due to lack of trust ( security concerns ) then the chain of trust is already broken and we should not be exposing that or those 3rd party repository to our end user base to begin with but the fact is that a consensus has been made to allow third party source inclusions from certain sources [1] and arguably if rpmfusion provides the distribution with keys those keys should be include based on that consensus and thus it's DKMS + nvidia installer process under secure boot should just work...

1. https://fedoraproject.org/wiki/Workstation/Third_Party_Software_Repositories

Something I fundamentally do not understand is why we cannot have a TOFU-like flow similar to what we do for GPG keys for package signatures for MOK keys. There should be no reason that RPM Fusion couldn't include in their release package their SB keys to lay down on disk so that when a kmod is installed from their repository, we can handle prompting to import the keys when a kmod package includes kmods signed with it.

Yes, it's not quite as simple as package GPG checks, and we have things like akmods and DKMS as a source of complexity for this, but we could develop something so that the user can easily set up their own local key for their own kmods and stuff installed via akmods or DKMS can auto-sign with that. At least with akmods, we're already at the point where the package content has been verified at least once through the repository chain (via GPG), and as long as we have an easy flow for revocation, then I don't see why this is a problem to implement.

Presumably you would just push an update that removes the relevant trusted 3rd party source from key database should it no longer be trusted for whatever reason ( and or blacklist it ) to much rejoice of end users if and then when that happens.

What is missing here ( afaikt ) is a policy that describes the process of providing the public keys to be included in the MOK database and the directions to do so which should be a part of the policy that describes how 3rd party repository can be considered officially part of Fedora and thus reference to as an option in the installer and the DE's as an additional "Trusted" source of packages ( which may contain drivers/modules ) . ( RH must have an "Partner Program" of some sort which a proposal could be based upon ) and the trusted key(s) from rpmfusion since it has already been officially approved as a 3rd party provide.

I would be surprised if this is a much of an technical issue on our ( Fedora's behalf ) or warranty any changes in our infrastructure...

Something I fundamentally do not understand is why we cannot have a TOFU-like flow similar to what we do for GPG keys for package signatures for MOK keys. There should be no reason that RPM Fusion couldn't include in their release package their SB keys to lay down on disk so that when a kmod is installed from their repository, we can handle prompting to import the keys when a kmod package includes kmods signed with it.

Right now it's an interactive process to enroll a key with the firmware. There's the mokutil side, where the user specifies a password; and upon reboot the mokutil EFI program runs in the pre-boot environment to confirm this key's addition by interactive prompt for the same password.

There should be no reason that RPM Fusion couldn't include in their release package their SB keys to lay down on disk so that when a kmod is installed from their repository, we can handle prompting to import the keys when a kmod package includes kmods signed with it.

There is no way to do mechanical, automated key enrollment securely without a verifiable chain of trust, and the scenario you describe does not have that. Any mechanical, automated enrollment without a chain of trust to verify it means malware can enroll keys as well. Importantly, that also means malware could use our bootloader on machines installed with other distros or OSes to enroll its keys on those machines during the boot process.

So metalink/gpg signed repodata -> gpg signed package -> package signature verified -> install -> load -> verify and load MOK isn't enough of a trust chain?

That doesn't sound right at all.

You can't do this kind of analysis without starting out in the context of the threat model being discussed. Within the secure boot context, there's nothing that has verified and trusts any of the repodata, gpg signatures, or any of that. We start by trusting the system owner, who can make modifications to what we trust, either in the firmware config or with mokutil (both of which require some degree of physical presence) and we trust the keys the system vendor has enrolled in the firmware. Everything else has to chain from there.

So then what, we need to vet and trust an RPM Fusion key and load it as part of Fedora then?

And how do you vet that? What is their trust model? Who has access to use the key? What is the audit process? What are their security practices?

And how do you vet that? What is their trust model? Who has access to use the key? What is the audit process? What are their security practices?

How is that done for Fedora itself? Someone has to vet that to grant Fedora a key.

Yes, that is correct. Very few people have access to the Fedora key. Any proven packager can build a kernel, but it won't be signed.

Yes, that is correct. Very few people have access to the Fedora key. Any proven packager can build a kernel, but it won't be signed.

While I think that's broken (there should be a way to sign with a key that isn't trusted so that the process always works the same), what stops us from putting together a program to vet RPM Fusion in the same way? They run infrastructure just like ours except for the undocumented parts (which include this stuff).

There is a process to sign with a key that isn't trusted, that is the Red Hat Test Key. If you build a kernel on a local machine, or if you tried to build a kernel in koji, it would be signed with that key. But as the key is not trusted, you can't boot that kernel with secureboot turned on.

There is a process to sign with a key that isn't trusted, that is the Red Hat Test Key. If you build a kernel on a local machine, or if you tried to build a kernel in koji, it would be signed with that key. But as the key is not trusted, you can't boot that kernel with secureboot turned on.

Okay, good to know, so we do have that process.

Also worth noting, you are asking Fedora to hand over trust to a 3rd part explicitly to use that trust to sign a chunk of binary which is not signed AFAIK by the developer, and where code is not available for review. There is literally no audit path here without nvidia directly involved.

You can't do this kind of analysis without starting out in the context of the threat model being discussed. Within the secure boot context, there's nothing that has verified and trusts any of the repodata, gpg signatures, or any of that. We start by trusting the system owner, who can make modifications to what we trust, either in the firmware config or with mokutil (both of which require some degree of physical presence) and we trust the keys the system vendor has enrolled in the firmware. Everything else has to chain from there.

The threat model was presumably discussed in the discussion surrounding the proposal of allowing 3rd party inclusion in Fedora and taking into consideration when the proposal was approved. At that point the decision was made that approved 3rd party should be treated and trusted as if they were part of Fedora itself ( the end user sees no difference ). It should have been pretty apparent when allowing 3rd party inclusion that Fedora had no control over those repositories components and infrastructure right ( It should have assess those prior to their approval hence my reference to a policy which should exist )

That said once a machine has been compromised it has been compromised and the scope of the damage it can do or has done becomes irrelevant unless the intent is going down the rabbit hole of speculation and hypotheticals and the expectation of the end user having a computer forensic on a retainer to dive into the scope of the breach. The fact is the end result always remains the same for the end user regardless if he has a single OS installed or multiple OS installed, the end user has to wipe his disk(s) clean and perform a re-install otherwise he can never be assured whatever compromised his system is still not in place...

The threat model was presumably discussed in the discussion surrounding the proposal of allowing 3rd party inclusion in Fedora and taking into consideration when the proposal was approved.

Citation needed.

The fact is the end result always remains the same for the end user regardless if he has a single OS installed or multiple OS installed, the end user has to wipe his disk(s) clean and perform a re-install otherwise he can never be assured whatever compromised his system is still not in place...

You've missed the point: without Secure Boot enabled (and working) you can still be rooted after the reinstall.

I would be okay with rpm fusion doing their own signing, and then providing a document for users to review their trust model and choose whether or not they import the rpm fusion key. Then a user can go through that PITA once, and have it go, but it also means rpmfusion would need to build the modules there, and a user can't do their own module builds.

The threat model was presumably discussed in the discussion surrounding the proposal of allowing 3rd party inclusion in Fedora and taking into consideration when the proposal was approved.

Citation needed.

Back in the day we strongly opposed inclusion or reference to 3rd party repos due to legal and security related issues and now that has changed and we have change proposals workflow in place precisely to allow discussions like these to take place prior to acceptance right and I'm presuming ( read as I expect but not looked into if that actually took place ) when the change proposal to include those repositories listed here [1] the scope of that change ( including it's legal and security context and how it affected the users ) took place. If that discussion never took place, then obviously the decision to include and refer to those 3rd party repo needs to be revisited ( and removed should it come down to that ) and our process looked at to prevent such decisions be made in the future.

The fact is the end result always remains the same for the end user regardless if he has a single OS installed or multiple OS installed, the end user has to wipe his disk(s) clean and perform a re-install otherwise he can never be assured whatever compromised his system is still not in place...

You've missed the point: without Secure Boot enabled (and working) you can still be rooted after the reinstall.

Right but the same proposed action ( steps end users should take ) for the end user applies regardless if it's some approved 3rd party or it's Fedora.

JBG

1. https://fedoraproject.org/wiki/Workstation/Third_Party_Software_Repositories

I would be okay with rpm fusion doing their own signing, and then providing a document for users to review their trust model and choose whether or not they import the rpm fusion key. Then a user can go through that PITA once, and have it go, but it also means rpmfusion would need to build the modules there, and a user can't do their own module builds.

If @kwizart can be told how to implement this, or at least if it can be documented somewhere how this works for Fedora at all, then I think this may be possible.

One potential clarification is that I'd like for it to be possible for RPM Fusion to have a trusted bot that watches for new Fedora kernels and would auto-rebuild kmod packages with the new kernel, since Koji can't do it by itself like other build systems can. Including that in the trust model would make it a lot saner to retire the akmod strategy RPM Fusion uses today for kernel modules.

I would be okay with rpm fusion doing their own signing, and then providing a document for users to review their trust model and choose whether or not they import the rpm fusion key. Then a user can go through that PITA once, and have it go, but it also means rpmfusion would need to build the modules there, and a user can't do their own module builds.

The secure boot process needs to be as transparent as it is if it's disabled so that does not help. ( the 3rd party needs to be included by default ) If it's usability suck or is filled with false positives end users will just turn it off like they have done with selinux for decades.

If we dont trust the 3rd party we should not be including/referring to it and offer application ( or drivers in this case ) that reside in it's repositories because the fact is end users aren't able to differentiate whether a component came from Fedora or from a 3rd party ( repo or application market place ) it's all just "Fedora" to him ( unless he manually has to jump through hoops to set it up ) and expecting (thus providing solutions ) that expect some kind of informed decisions about security related things from an end users is just ridiculous.

The end user places his trusts on the OS on doing the right thing for him.
Whether that turns out to be misplaced trust is entirely up to the OS, 3rd party repo or application marketplace and if anything fails Fedora as an OS will be blamed for it because that's the OS the end user is using.

I would be okay with rpm fusion doing their own signing, and then providing a document for users to review their trust model and choose whether or not they import the rpm fusion key. Then a user can go through that PITA once, and have it go, but it also means rpmfusion would need to build the modules there, and a user can't do their own module builds.

If @kwizart can be told how to implement this, or at least if it can be documented somewhere how this works for Fedora at all, then I think this may be possible.
One potential clarification is that I'd like for it to be possible for RPM Fusion to have a trusted bot that watches for new Fedora kernels and would auto-rebuild kmod packages with the new kernel, since Koji can't do it by itself like other build systems can. Including that in the trust model would make it a lot saner to retire the akmod strategy RPM Fusion uses today for kernel modules.

I don't know that I would have an issue with a bot doing the builds, but the whole value to this, is I don't have to. The document lays out what is happening, and each user can decide if they are comfortable with it or not.

We could probably swipe the license and notice metadata extension that SUSE uses for rpm-md to present that document, so that's something @dmach's team might be interested in to support this flow and bubble it up to GNOME Software.

I would be okay with rpm fusion doing their own signing, and then providing a document for users to review their trust model and choose whether or not they import the rpm fusion key. Then a user can go through that PITA once, and have it go, but it also means rpmfusion would need to build the modules there, and a user can't do their own module builds.

The secure boot process needs to be as transparent as it is if it's disabled so that does not help. ( the 3rd party needs to be included by default ) If it's usability suck or is filled with false positives end users will just turn it off like they have done with selinux for decades.

Which is likely what they are doing right now. This at least makes it a 1 time thing you do per machine, and manageable as opposed to something you must do every kernel update.

If we dont trust the 3rd party we should not be including/referring to it and offer application ( or drivers in this case ) that reside in it's repositories because the fact is end users aren't able to differentiate whether a component came from Fedora or from a 3rd party ( repo or application market place ) it's all just "Fedora" to him ( unless he manually has to jump through hoops to set it up ) and expecting (thus providing solutions ) that expect some kind of informed decisions about security related things from an end users is just ridiculous.

There are various levels of trust. Trusting an mpeg player is a bit different than trusting a kernel module. If "it's all just Fedora" to the user, then we have a problem. Third party repositories should not present as "just Fedora". The entire reason that they exist is because the things packaged there do not meet the qualifications of Fedora principles.

The end user places his trusts on the OS on doing the right thing for him.
Whether that turns out to be misplaced trust is entirely up to the OS, 3rd party repo or application marketplace and if anything fails Fedora as an OS will be blamed for it because that's the OS the end user is using.

As the person who builds signed kernels, I do not extend my trust to the 3rd party repo, and to nvidia (which is the extent of the trust chain in this scenario). This is not just "do I trust @kwizart to do the right thing", but do I trust kwizart, the maintainers of the signing key and the infrastructure that it depends upon within rpmfusion, the developers at nvidia, and whatever infrastructure they have in place for their binary since we can't see the source that is used to build it. Even if I am personally willing to extend my trust that far for my machines, I certainly couldn't say so on behalf of all Fedora users.

I'm a little lost in this discussion without a clear definitions:

• What benefits do Fedora users gain by enabling secure boot
• What do we need to do to avoid our secure boot infrastructure being used to attack other operating systems

[ https://mjg59.dreamwidth.org/12368.html has some background on the original thinking behind disabling unsigned modules - which is apparently the key difference between us and other distros that have less issues here ]

@pjones : can you explain:

You've missed the point: without Secure Boot enabled (and working) you can still be rooted after the reinstall.

Adding [1] for comparison and recommendation by the NSA in relation to the discussion.

1. https://www.nsa.gov/Portals/70/documents/what-we-do/cybersecurity/professional-resources/csi-boot-security-modes-and-recommendations.pdf

What do we need to do to avoid our secure boot infrastructure being used to attack other operating systems

Keep the chain of trust as short as possible as in limit it only to Fedora ( Or Fedora/Centos/RHEL for cross-booting support strictly in the RH ecosystem which RH controls) but that of course comes at the prices of excluding 3rd party repositories and other OS ( cross-booting becomes unsupported, that is if it's even supported today )

Bottom line the more you add to a chain of trust, the weaker it becomes.

@pjones : can you explain:

You've missed the point: without Secure Boot enabled (and working) you can still be rooted after the reinstall.

So, the straightforward thing to do if you have a Secure Boot breakout is obviously to use it as a bootloader and load modified version of the OS - i.e. backdoor the kmod verifier while loading the kernel. This what's known as a "bootkit", and that kind of backdoor is really the simplest variety - it's not hard to find enough code to easily implement that much by googling. This is clearly pretty bad, but it's also not limited to one OS - you need different backdoor code for each target, but once you've found a bootloader or kernel that doesn't properly check signatures, or trusts a signature that it should not, or is suceptable to the right kind of exploit, you can just install that in front of other OSes as the starting point.

But that's not the worst thing attackers have done. If the attacker is at all sophisticated, they can additionally replace UEFI APIs for disk access, boot variables, and even the system firmware boot selection menus. If they do that, then they can make it appears as though you've booted install media when you've actually still got the same backdoor, and it's trivial to install the exploit again during the re-install. This does not require all that much work to do poorly as a proof of concept; how much work the attacker puts into it developing the bootkit mostly determines how hard it is to detect that they've done this and how hard it is to fix it once you've noticed.

This is what attackers will do when they don't have some other vulnerability available, like flash not being properly locked. The "hackingteam" bootkit that was leaked in 2015 includes a module that will re-link your system firmware with their backdoor in the loader - but on some classes of machines with the vulnerability that allows them to do that, they still need to be able to run pre-boot code to actually modify it.

MokManager/mokutil is a compromise between being more secure against this kind of attack and being more practical - we allow you to enroll keys as the owner of the machine, but you need some degree of physical access to do so. The result is that if there is some weakness in what you've signed and are loading with mok, a very tenacious attacker will still be able to make progress on your system, but the attack won't be very portable to other systems.

Having said all that - can we please not argue all of the decisions of Fedora 18? We discussed this to death at the time, and I've got enough going on right now without doing it again.

1. Protection from: firmware implants, persistent attacker controlled system management modules, various kinds of OS Loader malware, a compromises kernel, including any kernel module being used as malware or as part of a chain of exploits to achieve ultimate ownership of the system. Numerous examples exist in the wild.
2. Keep control of the Fedora (private) signing key.
3. Without UEFI Secure Boot enabled and working, a number of locations exist that are not subject to being reset by a typical clean OS install. This includes the EFI system partition, and the chip the firmware itself is located on. Once the system is owned, it can be indistinguishable from not being owned using any familiar friendly UI, i.e. the firmware setup UI is not reliable.

Once you've disabled UEFI Secure Boot, a compromised (fake) kernel module can drop a malicious system management module onto the system, which in turn can make the flash where the firmware itself lives, writeable. Persistent malware achieved. Or even the attacker's own signing key, so they can run whatever they want. This system is now properly owned in a way that a clean install can't fix. Even if you go to the uncommon trouble of secure erasing the drive first. And this could all start with a remote attack from a web site using a browser, it doesn't require physical access ala Evil Maid attack.

And now imagine this computer happens to have possibily important private keys on it. And next wonder how you're going to get out of this hell hole without fire.

It's reasonable to consider ways to make it easier for users to install drivers that they vouch for being from a trusted source, however unqualified that is, and helping them sign those modules and enroll the key. But I'm a little skeptical of how it works for Fedora to sign 3rd party keys: it's not just a matter of authenticating current practice but verifying it on an on-going basis. If I thought that making this easier means few intentionally SB disabled systems, i'd be more supportive. I get super grouchy when I read folks on support channels and lists advocating just disable UEFI Secure Boot.

Issue #116 applies here too.

Having said all that - can we please not argue all of the decisions of Fedora 18? We discussed this to death at the time, and I've got enough going on right now without doing it again.

I don't want to argue all of that all over again. If anything, I still agree with the decisions @mjg59 did back then. I just want us to come up with something where we don't frustrate users who are trying to make their hardware work, especially now that we're moving into having Fedora preloaded on computers.

Having said all that - can we please not argue all of the decisions of Fedora 18? We discussed this to death at the time, and I've got enough going on right now without doing it again.

I don't want to argue all of that all over again. If anything, I still agree with the decisions @mjg59 did back then. I just want us to come up with something where we don't frustrate users who are trying to make their hardware work, especially now that we're moving into having Fedora preloaded on computers.

I suggested something that's better than what we've got in https://pagure.io/fedora-workstation/issue/155#comment-664899 . It doesn't seem like you've replied to it.

Having said all that - can we please not argue all of the decisions of Fedora 18? We discussed this to death at the time, and I've got enough going on right now without doing it again.
I don't want to argue all of that all over again. If anything, I still agree with the decisions @mjg59 did back then. I just want us to come up with something where we don't frustrate users who are trying to make their hardware work, especially now that we're moving into having Fedora preloaded on computers.

I suggested something that's better than what we've got in https://pagure.io/fedora-workstation/issue/155#comment-664899 . It doesn't seem like you've replied to it.

Sorry, I didn't notice that. That option is somewhat doable if we could assume people had YubiKeys. Another way that could work would be to do this with a TPM device and leverage that to essentially tie the signing to the device, but that doesn't help with the "conscious act" thing you're going for here.

If akmods was tweaked and we had a GUI frontend for it that helped with doing that process properly, it might work, though. However, I don't want to do this with DKMS because it doesn't produce a trackable result (i.e. a kmod RPM that's installed on the system).

Having said all that - can we please not argue all of the decisions of Fedora 18? We discussed this to death at the time, and I've got enough going on right now without doing it again.
I don't want to argue all of that all over again. If anything, I still agree with the decisions @mjg59 did back then. I just want us to come up with something where we don't frustrate users who are trying to make their hardware work, especially now that we're moving into having Fedora preloaded on computers.

I suggested something that's better than what we've got in https://pagure.io/fedora-workstation/issue/155#comment-664899 . It doesn't seem like you've replied to it.

Sorry, I didn't notice that. That option is somewhat doable if we could assume people had YubiKeys. Another way that could work would be to do this with a TPM device and leverage that to essentially tie the signing to the device, but that doesn't help with the "conscious act" thing you're going for here.
If akmods was tweaked and we had a GUI frontend for it that helped with doing that process properly, it might work, though. However, I don't want to do this with DKMS because it doesn't produce a trackable result (i.e. a kmod RPM that's installed on the system).

Which brings us back to, there is nothing that stops rpm fusion, or any other 3rd party from creating their own signing infrastructure, and detailing a step for users to put their single key into the trusted ring on their system. This has been the case since we first implemented secure boot. I am guessing the only thing stopping them is the assumption of responsibility there lies squarely on the repository should things be mismanaged.

Actually, from what I can tell, it's that nobody knows how to sign kernel modules for kmod packaging.

Actually, from what I can tell, it's that nobody knows how to sign kernel modules for kmod packaging.

pesign 113, which is in f33 currently, can sign kernel modules.

One way to solve this would be to move the nvidia open source kernel module building to Fedora's koji and build it together with the kernel. This would (1) make sure it exactly matches the kernel version, and (2) ensure it gets signed with Fedora key.

The main issue I see with this is that it's occasionally going to hold back kernel updates if the interface changes and the module doesn't build any more. Sadly, I don't think we have many alternative options if we want to make this work well for nvidia / Lenovo laptop users and ensure they always have access to a nvidia driver :(

I'd rather not have it bundled in the kernel tree. If we're going to do that, let's figure out how to build kmod packages in Fedora Koji instead.

One way to solve this would be to move the nvidia open source kernel module building to Fedora's koji and build it together with the kernel. This would (1) make sure it exactly matches the kernel version, and (2) ensure it gets signed with Fedora key.

You can't really build it "together" with the kernel, but it would be very easy to auto bump/build when a successful kernel build completes on a stable Fedora branch. Rawhide, you can try, there is no guarantee that it would succeed though, and Nvidia doesn't actually care. Their "policy" is no support for pre-release kernels. Typically this means they will release a fixed build shortly after Linus releases a .0

The main issue I see with this is that it's occasionally going to hold back kernel updates if the interface changes and the module doesn't build any more. Sadly, I don't think we have many alternative options if we want to make this work well for nvidia / Lenovo laptop users and ensure they always have access to a nvidia driver :(

Our policy was that we would not hold back kernel updates on the failing of nvidia, and when 3rd party was discussed there was supposed to be something that handled not booting from a new kernel until nvidia was available for users running it. In practice, we don't rebase until 3-4 weeks after Linus does a .0 update, and nvidia is working by then. In the rare occasion that a stable update has broken things, we tend to revert the patch that does so, provided it isn't security related.

I'd rather not have it bundled in the kernel tree. If we're going to do that, let's figure out how to build kmod packages in Fedora Koji instead.

No, it cannot be bundled in the kernel tree. And we really do not want to get into the business of kmod packages in Fedora proper. We could come up with a process to build select kmod packages on Fedora infrastructure, sure. But that is going to have to be much more tightly controlled than the standard "add a package to Fedora process". For instance, any proven packager can technically commit to the kernel or grub, or any of the signed packages, but if they build it, it will not be signed. Signature access is very intentionally tightly controlled.

I'd rather not have it bundled in the kernel tree. If we're going to do that, let's figure out how to build kmod packages in Fedora Koji instead.

Sure, that's really up to the kernel maintainers what way they prefer it.

The main thing is that I think we (Fedora) just need to take over the responsibility of rebuilding the nvidia kernel module. If it's in rpmfusion we have no way to ensure it gets updated/rebuilt in lock step together with kernel updates and correctly signed.

Sure, that's really up to the kernel maintainers what way they prefer it.
The main thing is that I think we (Fedora) just need to take over the responsibility of rebuilding the nvidia kernel module. If it's in rpmfusion we have no way to ensure it gets updated/rebuilt in lock step together with kernel updates and correctly signed.

It really needs to be special cased. The nvidia package goes against the Fedora principals, and can't really be a package in Fedora proper, so it would need to be built on Fedora koji, but still distributed through a 3rd party repo.

It really needs to be special cased. The nvidia package goes against the Fedora principals, and can't really be a package in Fedora proper, so it would need to be built on Fedora koji, but still distributed through a 3rd party repo.

That sounds like a good plan to me. Thanks, @jforbes! :)

If it's a separate repo, that may actually offer a functional advantage: We could have a slightly different repo structure than what regular Fedora repos have, and instead of just shipping the latest package, we could ship all the versions we've ever built in it, so that users can easily install the correct module for the exact version of kernel they have installed.

This is similar to what openSUSE does, too. The NVIDIA driver is built in their build system, and then NVIDIA hosts it themselves on download.nvidia.com for openSUSE users to consume.

Cf. https://en.opensuse.org/SDB:NVIDIA_drivers

@ngompa can you update us on the status of this issue?

I've been poking at this off and on for months, and today I've talked to @pjones about this more in #fedora-devel:matrix.org. There may be some progress to be made here, though @pjones can give more color in here on this topic.

I think the way to move forward with this is probably some variation on the plan below, which has components to add mechanisms for the following critical abilities: 1) to limit what certain signing certificates can be used for, 2) to simplify the inevitably scenarios where we need to issue revocations, 3) to allow admins to easily and reliably opt-out, whether to all kmods, specific kmods, or specific builds.
- Implement the critical x.509 EKU "moduleSigningOnlyKey" everywhere (might actually be done everywhere but kexec?), so we can make signing certificates that are only good for kernel modules.
- Make a way for admins to add revocations locally, both during boot and later, which isn't reversible without a reboot. I.e. some way to tell the kernel "stop trusting this now". In my mind that means adding a root-privileged keyctl method to add things to our disallowed-list, which dosn't affect anything that's already loaded. This lets us say "I've loaded all the 3rd party kmods I'm going to", as well as being the key to giving admins an opt-out mechanism.
- Add a built-in revocations list similar to shim's internal dbx to the kernel, so we can have a "all kernels going forward do not trust this {ca cert, signing cert, built module}" without getting a shim build signed.
- Add ".sbat" revocation support to the kernel's module loader, using a SBAT rule list added during the kernel build, suplemented by a similar mechanism to the keyctl above. This lets us say "all modules with the property 'fedora.kmod' with a value <= 3 are revoked", greatly simplifying revocations of /classes/ of things. We have to decide on the classes ourselves, but any given class of things more or less maps to one well known set of vulnerabilities or risks that have been eliminated. See https://github.com/rhboot/shim/blob/main/SBAT.example.md and https://github.com/rhboot/shim/blob/main/SBAT.md for details on how this works.
- Define a set of non-authenticated revocation files that go on the filesystem i.e. /etc/kernel/revocations.d/{01-kmod.esl,01-kmod.sbat} for admins to use, with a systemd service to load them and a dracut module to load them very early. This lets sysadmins set revocation policy with approximately the same threat model as against "rm vmlinuz".
- Add some sample sets of revocations that an admin can trivially copy to /etc/kernel/revocations.d/ to disable support for {all 3rd parties, all of some particular kmod, etc}.
- Issue a cert that has moduleSigningOnlyKey set for signing each kmod we decide to trust, which chains to some CA we decide.
- We may also want to add a definition for moduleSigningOnlyKey on CA certs, that specifies that cert is only allowed to issue certs with a critical moduleSigningOnlyKey EKU set. This might also be better as a second OID.

Note that what I've specified above is only a mechanism to safely load modules we've assigned trust to which aren't from the kernel package. Fundamentally we're still talking about adding third parties that can build trusted kernel modules, even if we have them do so in fedora kmod packages, and so we still need to have conversations and make formal decisions about:
- whether we're talking about kmods in fedora packages or external repos
- who we're trusting, why, and what the procedures around establishing that trust are.
- issues of key storage and authentication for building signed kmods in fedora
- what sort of topology we want for the certs, what security guarantees they provide against what threat model.

Also note that several of the things above are not my department, and we need to get @dhowells involved in this discussion.

We had a brainstorming meeting discussing this problem today with Justin and Peter. Currently we are far from a solution. Meeting minutes will be made available as usual, but we discussed so much that we aren't prepared to adequately summarize the discussion here. Instead, we're planning to post some follow-up questions for Justin and Peter here to try to understand things better.

We had a brainstorming meeting discussing this problem today with Justin and Peter. Currently we are far from a solution. Meeting minutes will be made available as usual, but we discussed so much that we aren't prepared to adequately summarize the discussion here. Instead, we're planning to post some follow-up questions for Justin and Peter here to try to understand things better.

So I think we have a solution, which may land in 5.17, as a side effect of the work that Edge/IoT is doing with IMA.

These are the follow-up questions @catanzaro mentioned two days ago upthread, and are directed at @pjones @jforbes

Allowing software to automatically import additional UEFI Secure Boot keys is not acceptable; explicit user intervention (via mokmanager ui) is required to add 3rd party keys?

Considering limited NVRAM spec, do you have any concern about adding multiple Secure Boot keys? Or the accumulation of stale keys?

Fedora private keys will not be shared with a third party, including RPM Fusion, e.g. Fedora Secure Boot CA, Fedora kernel signing key?

It looks like openSUSE locally builds the NVIDIA kernel module with locally created keys - each time the driver is updated (maybe also whenever the kernel version is bumped) there's a mokmanager "Enroll MOK" and "Delete MOK" event. This seems onerous. And maybe also impractical if we need to keep 3 keys enrolled (one per kernel).
https://en.opensuse.org/SDB:NVIDIA_drivers

Would Fedora be willing to trust (sign) a 3rd party's key for either kernel modules or UEFI Secure Boot, with a derived key intended specifically for this purpose? That way it's easier to revoke than either the Fedora kernel key or Secure Boot CA key?

These are the follow-up questions @catanzaro mentioned two days ago upthread, and are directed at @pjones @jforbes

Allowing software to automatically import additional UEFI Secure Boot keys is not acceptable; explicit user intervention (via mokmanager ui) is required to add 3rd party keys?

Yes, a user has to be aware that they are trusting something besides Fedora proper. I don't care how good RPMFusion's policies might be, they are packaging a large binary blob with a small ope n source shim with kernel permissions. This is not something to take lightly.

Considering limited NVRAM spec, do you have any concern about adding multiple Secure Boot keys? Or the accumulation of stale keys?

No, realistically I would be surprised to see anything more than 2-3 keys imported to any machine.

Fedora private keys will not be shared with a third party, including RPM Fusion, e.g. Fedora Secure Boot CA, Fedora kernel signing key?

Absolutely not. If we were to go into any role where we are doing the build and signing, it would have to be with a new key. We wouldn't even build nvidia with the Fedora keys if we were doing the build ourselves, so we certainly will not share our private keys with anyone under any circumstances.

It looks like openSUSE locally builds the NVIDIA kernel module with locally created keys - each time the driver is updated (maybe also whenever the kernel version is bumped) there's a mokmanager "Enroll MOK" and "Delete MOK" event. This seems onerous. And maybe also impractical if we need to keep 3 keys enrolled (one per kernel).
https://en.opensuse.org/SDB:NVIDIA_drivers

That seems unnecessarily problematic. You can achieve similar results by generating a per machine key and keeping it around. Patches exist for akmod for this. Nothing about the SuSE solution seems desirable at all.

Would Fedora be willing to trust (sign) a 3rd party's key for either kernel modules or UEFI Secure Boot, with a derived key intended specifically for this purpose? That way it's easier to revoke than either the Fedora kernel key or Secure Boot CA key?

While a 3rd party would need need their own key, I still do not think it wise for Fedora to trust it by default. We do not control their build environment, and even they do not know what code went into the binary blob that nvidia ships with their drivers. I believe we need a firm user acknowledgement that they are explicitly trusting a 3rd party key for an specific purpose. More importantly, if something were to happen where that 3rd party key were compromised, at least the damage would be limited in the worst case to users who chose to trust that key. If it were trusted by default, every Fedora system would be vulnerable, even if they never used that 3rd party repository. We can certainly revoke that trust, but not necessarily before the damage is done, and it would depend on user updates which are not always consistent. More importantly a user might look at a security advisory and assume it does not apply to them because they have never interacted with that 3rd party.

While a 3rd party would need need their own key, I still do not think it wise for Fedora to trust it by default.

One clarification: we really only need to trust such a key in case the user elects to install the NVIDIA driver. We certainly don't need users who don't use the NVIDIA driver to have any such trust. So with that additional limitation, does that change your answer?

(I guess it should be obvious that a driver can do whatever it wants to your computer regardless of any secure boot considerations. Users already trust the driver not to traverse their home directory and upload all their files to command and control. I don't see a particularly huge distinction between that level of trust vs. getting a key in the kernel keyring.)

On Thu, Dec 2, 2021 at 4:39 PM Michael Catanzaro pagure@pagure.io wrote:

catanzaro added a new comment to an issue you are following:
``

While a 3rd party would need need their own key, I still do not think i=
t wise for Fedora to trust it by default.

One clarification: we really only need to trust such a key in case the us=
er elects to install the NVIDIA driver. We certainly don't need users who d=
on't use the NVIDIA driver to have any such trust. So with that additional =
limitation, does that change your answer?

(I guess it should be obvious that a driver can do whatever it wants to y=
our computer regardless of any secure boot considerations. Users already tr=
ust the driver not to traverse their home directory and upload all their fi=
les to command and control. I don't see a particularly huge distinction bet=
ween that level of trust vs. getting a key in the kernel keyring.)
``

I still firmly believe that a user must acknowledge that they are
extending their trust beyond Fedora willingly, so even with that
limitation, it does not change my answer.

a user must acknowledge that they are extending their trust beyond Fedora willingly

The user must also agree to a proprietary license at some point (to be determined) but let's say the sequence looks like this:

1 GNOME Initial Setup shows this 3rd party repositories page

There is a clear statement "some proprietary software is included", these repositories are disabled by default, the user needs to take explicit action to enable them.

2 Once 3rd party repositories is enabled

$ sudo dnf repolist
repo id                                 repo name
fedora-cisco-openh264                   Fedora rawhide openh264 (From Cisco) - x86_64
google-chrome                           google-chrome
phracek-PyCharm                         Copr repo for PyCharm owned by phracek
rawhide                                 Fedora - Rawhide - Developmental packages for the next Fedora release
rawhide-modular                         Fedora - Modular Rawhide - Developmental packages for the next Fedora release
rpmfusion-nonfree-nvidia-driver         RPM Fusion for Fedora rawhide - Nonfree - NVIDIA Driver
rpmfusion-nonfree-steam                 RPM Fusion for Fedora rawhide - Nonfree - Steam

The GNOME Software equivalent listing

3 Choose to install the NVIDIA driver
4 See the EULA for the NVIDIA driver

That's at least two (possibly four) acknowledgements to extend trust beyond Fedora. Requiring a yet another acknowledgement is less about what or why, but where the acknowledgement is happening: user space vs preboot environment.

I think @pjones and @jforbes are saying user space isn't as trusted an environment for enrolling Secure Boot keys, relative to the preboot environment? Despite SELinux?

The points where I'm still stuck:

• Fedora already extends trust, by default, and without notification or opt-out, to linux-firmware which is a pile of largely proprietary binary blobs. Why is that given such a huge waiver? Practicality. Is there any other reason?
• Is it really appropriate for Fedora or RPM Fusion to sign proprietary software directly? I think NVIDIA should sign their binary blobs. Can kernel modules have multiple segments signed with different keys?
• Could the Fedora kernel contain a key to extend trust to other keys, instead of signing kernel modules? i.e. a "Fedora 3rd party RPM Fusion & NVIDIA signing key" which signs whatever NVIDIA key is used to sign their binary blob, and also RPM Fusion's key used to sign the rest of the kernel module. And none of these keys would be enrolled with mokmanager. (While this intermediate signing step results in defacto trust in proprietary software, I think it's important that FOSS projects aren't directly extending trust to the software, but rather to those company's signing keys.)
• Is such a paradigm more onerous to build and document, than one where signing keys are produced and retained locally. Which is safer? My strong suspicion is that once it is known there's a path to a local private key that can be used to sign kernel modules - it'll be recognized as a pretty big door.

I asked in F36 Change: DIGLIM (System-Wide Change proposal) if the feature could help with this issue. The response from @robertosassu is here. My take away is that it could help, but isn't part of the current design.

What's clear is we need someone who's capable of doing the work to help us understand the options, for example:

Whether it is better to trust an external repository to
adequately protect the private key used for signing, or to
keep the private key in the system that would load the
third-party kernel modules is a difficult decision, and probably
each user would like to make that decision.

Does this translate into providing both options? And if so what does that look like in terms of development: Fedora infra, 3rd party (RPM Fusion), and the UI.

I asked in F36 Change: DIGLIM (System-Wide Change proposal) if the feature could help with this issue. The response from @robertosassu is here. My take away is that it could help, but isn't part of the current design.

Unfortunately, even by whitelisting third-party kernel modules would
still not be enough due to the fact that kernel modules are verified
with an independent mechanism than the integrity subsystem.

What's clear is we need someone who's capable of doing the work to help us understand the options, for example:

Whether it is better to trust an external repository to
adequately protect the private key used for signing, or to
keep the private key in the system that would load the
third-party kernel modules is a difficult decision, and probably
each user would like to make that decision.

Does this translate into providing both options? And if so what does that look like in terms of development: Fedora infra, 3rd party (RPM Fusion), and the UI.

I would be more in favor of having Fedora providing the
infrastructure for building third-party kernel modules, although
it does not vouch for the content it is building.

This would still provide much stronger guarantees about what
is loaded by the user. The Fedora infrastructure would verify
the authenticity of the NVIDIA blob used as input for building
the kernel module, and would guarantee that the packages
used for the building process are good (they come from
Fedora itself).

I personally don’t think having the private key in the same
machine that uses the kernel modules signed with that key would
provide security. I still would hope that there is an infrastructure
(even if it is not the Fedora one) users can rely on to correctly
build and sign the kernel modules. In that case, this patch set
could help:

https://lore.kernel.org/linux-integrity/20211124044124.998170-1-eric.snowberg@oracle.com/

This would allow users to add new keys in the MOK database,
so that their third-party kernel modules can be verified with
those keys when secure boot is enabled.

Roberto

HUAWEI TECHNOLOGIES Duesseldorf GmbH, HRB 56063
Managing Director: Li Peng, Zhong Ronghua

@robertosassu @mattdm

Thanks for the responses Roberto.

I would be more in favor of having Fedora providing the infrastructure for building third-party kernel modules, although it does not vouch for the content it is building.

I'm not sure what the requirements for this are within Fedora:

• a separate Fedora key for 3rd party kernel modules, or even keys per 3rd party so that revocation is easier, should it be necessary; or
• 3rd party modules would only be available in a separate Fedora repository; or
• they really must be built and signed by a 3rd party, NVIDIA themselves or RPM Fusion.

I'm convinced we need a proper interface for the user to opt-in to the use of 3rd party kernel modules, and I think we already have that in place. But I'm not convinced it's improper for Fedora to sign 3rd party kernel modules, given a proper chain of trust verification.

https://lore.kernel.org/linux-integrity/20211124044124.998170-1-eric.snowberg@oracle.com/
This would allow users to add new keys in the MOK database, so that their third-party kernel modules can be verified with those keys when secure boot is enabled.

Would this enable a means for user-space to inform the kernel of trustworthy kernel module signing keys, without needing to deal with UEFI firmware at all?

The current UI/UX of enrolling Secure Boot keys with mokutil is terrible.

• it can't be automated e.g. GNOME Initial Setup or GNOME Software;
• the user must interact with both user-space mokutil, reboot, interact with a EFI program to again indicate they trust the key
• removal of the key is equally difficult, and may not even be possible via the firmware's setup interface;
• there's limited space in NVRAM.

If keys can only be trusted by the user indicating trust with an EFI program in the pre-boot environment, that's onerous. But if revoking trust follows the same UI/UX paradigm, that's untenable. If the idea is there's a higher risk of 3rd party kernel modules being compromised in such a way that they need to be urgently disabled, we need a way to automate key revocation.

@robertosassu @mattdm

Thanks for the responses Roberto.

I would be more in favor of having Fedora providing the infrastructure for building third-party kernel modules, although it does not vouch for the content it is building.

I'm not sure what the requirements for this are within Fedora:

• a separate Fedora key for 3rd party kernel modules, or even keys per 3rd party so that revocation is easier, should it be necessary; or
• 3rd party modules would only be available in a separate Fedora repository; or
• they really must be built and signed by a 3rd party, NVIDIA themselves or RPM Fusion.

Likely all these requirements are needed. Fedora would probably not distribute third-party kernel modules from their main repositories, but could create ad-hoc repositories. A requirement for distributing third-party kernel modules is that the sources kernel modules are built from must be signed. The key to verify those kernel modules could be embedded in the kernel, but disabled by default, and enabled upon user request.

I'm convinced we need a proper interface for the user to opt-in to the use of 3rd party kernel modules, and I think we already have that in place. But I'm not convinced it's improper for Fedora to sign 3rd party kernel modules, given a proper chain of trust verification.

https://lore.kernel.org/linux-integrity/20211124044124.998170-1-eric.snowberg@oracle.com/
This would allow users to add new keys in the MOK database, so that their third-party kernel modules can be verified with those keys when secure boot is enabled.

Would this enable a means for user-space to inform the kernel of trustworthy kernel module signing keys, without needing to deal with UEFI firmware at all?

I'm not too familiar with this patch set. It would be better to ask the author. Anyway, it is a solution if Fedora could not provide third-party kernel modules.

The current UI/UX of enrolling Secure Boot keys with mokutil is terrible.

• it can't be automated e.g. GNOME Initial Setup or GNOME Software;
• the user must interact with both user-space mokutil, reboot, interact with a EFI program to again indicate they trust the key
• removal of the key is equally difficult, and may not even be possible via the firmware's setup interface;
• there's limited space in NVRAM.

If keys can only be trusted by the user indicating trust with an EFI program in the pre-boot environment, that's onerous. But if revoking trust follows the same UI/UX paradigm, that's untenable. If the idea is there's a higher risk of 3rd party kernel modules being compromised in such a way that they need to be urgently disabled, we need a way to automate key revocation.

There is a blacklist keyring that can be used for this purpose. However, I didn't try it.

So I've been pinging relevant Red Hatters internally about this, but so far I've only heard back from Justin, who was already participating here in this issue. So this approach is stalled.

At this point, I think the Working Group should just mandate some solution and move on. Chris has proposed just signing NVIDIA's driver, and that sounds perfectly good to me. Common sense would indicate that this doesn't mean Fedora necessarily trusts the proprietary code, just that Fedora recognizes that it's mandatory for our users. Let's see how debate on devel@ plays out.

There may be an alternative to "just signing the driver" which I think a lot of people would be opposed to, that may be viable in the F-37 timeframe which is actually a side effect of the IMA work the Edge team has been doing. Will know more about the complete viability of this soon(ish).

So I've been pinging relevant Red Hatters internally about this, but so far I've only heard back from Justin, who was already participating here in this issue. So this approach is stalled.

At this point, I think the Working Group should just mandate some solution and move on. Chris has proposed just signing NVIDIA's driver, and that sounds perfectly good to me. Common sense would indicate that this doesn't mean Fedora necessarily trusts the proprietary code, just that Fedora recognizes that it's mandatory for our users. Let's see how debate on devel@ plays out.

With what key? Surely you don't propose we use the Fedora key to sign a binary blob that operates in kernel space from a company who was just hacked last month?

So I've been pinging relevant Red Hatters internally about this, but so far I've only heard back from Justin, who was already participating here in this issue. So this approach is stalled.

At this point, I think the Working Group should just mandate some solution and move on. Chris has proposed just signing NVIDIA's driver, and that sounds perfectly good to me. Common sense would indicate that this doesn't mean Fedora necessarily trusts the proprietary code, just that Fedora recognizes that it's mandatory for our users. Let's see how debate on devel@ plays out.

With what key? Surely you don't propose we use the Fedora key to sign a binary blob that operates in kernel space from a company who was just hacked last month?

Of course not, we'd make a new key and have the kernel trust that. That way we can revoke just that key in the event something has gone wrong.

Of course not, we'd make a new key and have the kernel trust that. That way we can revoke just that key in the event something has gone wrong.

Then the driver is effectively going to be trusted for everyone, even people who don’t want to use the proprietary driver. The key needs to be trusted only on systems of users who want to use the Nvidia driver, otherwise this weakens the protection against rootkits of all Fedora users.

Then the driver is effectively going to be trusted for everyone, even people who don’t want to use the proprietary driver.

The proprietary driver would only be opt-in, not installed by default, nor would it load by default.

The key needs to be trusted only on systems of users who want to use the Nvidia driver, otherwise this weakens the protection against rootkits of all Fedora users.

It seems plausible that a separate key for extending trust (to either a 3rd parties key or a binary) could be in a package that the user needs to explicitly install. That way extending trust would require the user and Fedora to opt-in.

If the malware has root, it can just install any package itself (and if it doesn’t there’s not much reason to worry about rootkits). I don’t see how it’s possible to avoid requiring user interaction with some trusted component in the boot chain to approve a key at the OS level. It’s one more step users need to go through, but if it can be done using the desktop or terminal, it doesn’t provide that much protection, and at least this gives a potential way to support all out-of-tree modules without disabling secure boot, rather than just Nvidia.

ETA: An example scenario I’m envisioning is where some malware on the system wants to use a vulnerability in the Nvidia driver to install a persistent rootkit.

(Call it far-fetched if you will, attacks keep getting more sophisticated and what is far-fetched today will be bread and butter in 5 years.)

I mean, yeah that's true, but we can't do both. We can have a good user experience OR we can distrust NVIDIA. Those are mutually-exclusive options. So what do we choose?

I'm all in favor of attacking NVIDIA, but I don't think it would do any good, and I doubt other distros would follow.

It is not that much of an attack on nvidia to ask users to trust a new key one time. And it lowers the attack surface for the majority of users who do not run any 3rd part modules. People have been following docs on how to install nvidia and get it working for over a decade now, and having a simplified thing where it can be done with a few clicks is still a user experience win. On the flip side, if something did happen and the key was compromised and used, that would be a major user experience loss of trust for all of the people not running these modules.

If it was at the kernel-level and not firmware level, then it'd be fine. Then it's also possible to automate for mass deployment without having to deal with firmware difficulties.

If it was at the kernel-level and not firmware level, then it'd be fine. Then it's also possible to automate for mass deployment without having to deal with firmware difficulties.

Any process that doesn't require physical access can be automated. If it can be automated, it can be exploited.

If it was at the kernel-level and not firmware level, then it'd be fine. Then it's also possible to automate for mass deployment without having to deal with firmware difficulties.

Any process that doesn't require physical access can be automated. If it can be automated, it can be exploited.

And? If it's not in the firmware, that's probably fine. This is something people reasonably expect now with software packaged and shipped via RPMs.

Related, it looks like there's a kernel-level machine keyring introduced in Linux 5.18. Perhaps this can be extended to be useful for our purposes?

It is not that much of an attack on nvidia to ask users to trust a new key one time.

Yes, it really is. That is way too hard for our users, not at all a reasonable ask. No matter how easy it might seem for you, please assume that 95% of our users will give up on Fedora altogether rather than attempt this.

In reality, everyone just turns off secure boot. I've never heard of anyone importing keys or signing drivers ever....

ETA: An example scenario I’m envisioning is where some malware on the system wants to use a vulnerability in the Nvidia driver to install a persistent rootkit.

So long as Secure Boot is enabled, SPI flash is write protected. There's a good chance of getting rid of the malware, if you detect it. But if SB is disabled, SPI flash can be written to. There's examples of lie in wait malware, waiting for exactly this, in order to drop a truly persistent payload into firmware. Not even replacing drives will get rid of it.

Making it so onerous for users that they resort to disabling SB, which is overwhelmingly what they are doing, under auspices of security while ignoring the insecure reality of what we've incentivized - is just folly. For all our security worry, we've in effect created a class of users who are less secure than others, and the line is drawn on hardware. For sure NVIDIA properly gets a heap of blame for their generally uncooperative stance, but they aren't the ones who've directly created the perverse incentive.

We should innovate an alternative. We can't remove all risk. We can manage the risk, by reasonably assuming at some point such a 3rd party binary becomes compromised and will have to be untrusted. And making a plan and the design to handle that accordingly.

Microsoft is already extending trust to NVIDIA for this same scenario, and with a comparatively enormous footprint. What we're doing doesn't really seem as much about security to me, than it is about punishing hapless users for their hardware choices. The position of meeting hostile with hostility, and hoping users make better choices hasn't really had an effect has it? Folks are still buying NVIDIA hardware.

I think we're better off bringing these users into the Fedora and Secure Boot and thus FOSS fold. By reducing the barriers we grow the market and improve the chances someone will work to make this even better.

Today, NVIDIA has announced that they are open sourcing the kernel-side components of their graphics driver.

@uraeus has written a blog post about it.

The initial driver code is on GitHub, so perhaps we should revisit how we solve this with this in mind...

I would say it makes this issue one completely about the proprietary driver they they may continue to ship for a while. If that is the case, I don't care about making it easier for users. The open source driver does have some obstacles. We need to file for FESCo approval since it is an out of tree module and against policy, but I expect it will not be too long lived. This isn't a ZFS like token code release, it is dual licensed MIT/GPL so it can eventually be upstreamed.

My assumption is that the driver will go in-tree eventually, once it's useful for desktops?

In the meantime, I wonder if we need to solve this anymore at all. It seems clear that the long term path forward is the new driver, even if that will take several years.

My assumption is that the driver will go in-tree eventually, once it's useful for desktops?

In the meantime, I wonder if we need to solve this anymore at all. It seems clear that the long term path forward is the new driver, even if that will take several years.

This won't help people with graphics cards older than Turing generation (GTX 16 / RTX 20), so it still needs a solution.

My assumption is that the driver will go in-tree eventually, once it's useful for desktops?

In the meantime, I wonder if we need to solve this anymore at all. It seems clear that the long term path forward is the new driver, even if that will take several years.

The nvidia open source driver they have released will not be going in-tree. Don't plan any future where the driver that supports their userspace will be in-tree, it's not a short term plan at all.

Action item: Neal to investigate this ticket again, update proposal

We are WONTFIXing this. Our plan is to focus on improving open source solutions instead (nouveau). It'

It's not ready yet but it's now looking like less work than making NVIDIA driver work with secure boot.

Looks like the new nouveau patch set is here now.

We are WONTFIXing this. Our plan is to focus on improving open source solutions instead (nouveau). It'

I get it (and I'm not going to throw a tantrum if the status on this doesn't change), but shouldn't we also provide a solution for users that have cards older than Turing, as @ngompa said? My personal machine has a 1050Ti and I personally know how to create and enroll my own keys, even on Silverblue (https://github.com/CheariX/silverblue-akmods-keys), but this shouldn't be the case for most people.

The 1060, 1050Ti and 1070 are still some of the most popular graphics cards on Steam on Linux (https://store.steampowered.com/hwsurvey/Steam-Hardware-Software-Survey-Welcome-to-Steam?platform=linux).

They are going away with time, and hopefully with the development in nouveau getting stronger we won't need it in the future, but for now we still need a solution. The way Ubuntu does it works pretty reliably and makes it way easier to get NVIDIA working with secure boot. If we do it right and do it well, it could even be useful for people that use other out-of-tree kernel modules, like virtualbox for example.

The way Ubuntu does it works pretty reliably and makes it way easier to get NVIDIA working with secure boot.

Can you explain further please? My understanding was that this is broken in Ubuntu too and that the problem is basically not solvable. @ngompa?

Last time I used Ubuntu on something with NVIDIA (16.04), it didn't enforce lockdown in the first place. Even openSUSE only just started enforcing it this year.

With lockdown enforced, our problem is that we have no tools to handle the whole secure boot key trust from the OS. If we get a kernel-level keying we can use that can be modified without screwing around with firmware, then we'd be fine. Alternatively, we need some kind of automation for importing Machine Owner Keys without having to deal with all the accessibility problems that UEFI introduces.

The way Ubuntu does it works pretty reliably and makes it way easier to get NVIDIA working with secure boot.

Can you explain further please? My understanding was that this is broken in Ubuntu too and that the problem is basically not solvable. @ngompa?

It'd probably be complicated to explain, as I'm on Fedora for quite a while on my main machine (I believe the last time I used Ubuntu was on 20.04, so I'm not sure if it is currently, broken, but here's a video showcasing what I'm talking about: https://www.youtube.com/watch?v=A0gxy3xaJlE

Basically you install the drivers, and in that process you enroll your own secure boot key. IIRC there's also a process for that in the distro's installer.

Also, found a video that actually shows the process (at least when installing the drivers via the terminal): https://youtu.be/FAknvXs4M1w?si=uNMySC11pYuhCuUF&t=119

@ngompa is there a reason we can't have a similar MOK entrollment step to what is seen in that video?

(Well, it can't be exactly the same, since package installation must be noninteractive. But similar.)

So I don't see any reason that Ubuntu's strategy would not work for us if we had developers willing to implement it. However, Fedora package installation is non-interactive and that will not change, so that part would need to be handled differently. Probably GNOME Software would need to tell the user how to use the MOK enrollment UI.

Unfortunately this issue has been open for a long time without any developers interested in fixing it. If we had developers willing to fix the experience of installing NVIDIA proprietary driver, that would be great, but I think we don't. Proposal: give up and stop offering the proprietary driver in GNOME Software, Fedora Workstation to focus on supporting only nouveau users.

-1 from me. I think having the driver available from a repo out of the box makes things easier for end users and I see no pressing issues which would warrant removing it and making driver installation harder.

The pressing issue is it doesn't actually work unless you disable secure boot, right?

If they download it from somewhere else they still need to disable secure boot, right? Why make the download step harder for them by removing the nicely integrated package?

We should only offer nice downloads for software that we are actually capable of installing properly. i.e. we should either (a) add clear instructions to help the user complete the installation process properly, or else (b) not offer the driver at all.

Proposal: give up and stop offering the proprietary driver in GNOME Software, Fedora Workstation to focus on supporting only nouveau users.

I understand where the sentiment comes from (I wouldn't be enthusiastic about working on NVIDIA stuff either), but the only ones who suffer with this would be the users. Even if nvidia-open was feature complete today, dropping the proprietary driver would just alienate a large portion of the userbase that don't have access to it (mainly Pascal users).

Fedora package installation is non-interactive and that will not change, so that part would need to be handled differently

Would there be a way for this to be done post-install? For example, there isn't a lot of value in signing your own keys if you're not using third-party kernel modules, so why not offer this as either an option or just part of the process of adding third-party repositories in the post-installation setup? That way, they'd serve even if the user doesn't use NVIDIA but ends up needing some of these modules (e.g. virtualbox or v4l2loopback).

Would there be a way for this to be done post-install?

Of course it's possible. The problem is nobody seems interested in working on this issue. If we had developers interested in making the proprietary driver work with secure boot, then presumably they would implement something similar to what Ubuntu has done and we'd move on.

Suggestions for incremental improvements out of the discussion in the meeting:
- delay blacklisting nouvau until after we're successfully use nvidia
- notify the user upon installing nvidia with a link to instructions

We discussed this again today:

• We have consensus that the status quo is worse than not offering the NVIDIA driver at all. We expect that installing this driver will break graphics for most users because it won't work if secure boot is enabled, and it will denylist the nouveau driver to block any fallback to a working configuration. Users would be better off following documentation instructing them how to install the driver manually rather than using GNOME Software.
• We do not have consensus on how to proceed. Giving up and removing the driver instead of trying to fix it would look pretty bad and is not what users want, but this issue has been open for three years now so there is not very much hope of fixing things. I'd like to remove the NVIDIA driver from fedora-workstation-repositories unless a realistic, actionable, and resourced plan to fix it can be developed, but we do not have consensus for this. Other WG members prefer to focus on fixing it.

@aday we'd like to schedule this topic again for next week's Workstation WG meeting (October 31) alongside the planned discussion of #395, when we'll have guests from graphics team at the meeting. Their opinions would be welcome.

We could start building and shipping the open-gpu-kernel-modules in a side along repo as a kmod package that we sign alongside shipping the nvidia-gsp-firmware in main Fedora for nouveau and this.

Then we can rely on NVIDIA's official Fedora repository for the userspace part. This would only cover Turing+ but it gives us coverage of everything NVIDIA has released in the past five years.

We could start building and shipping the open-gpu-kernel-modules in a side along repo as a kmod package that we sign alongside shipping the nvidia-gsp-firmware in main Fedora for nouveau and this.

Then we can rely on NVIDIA's official Fedora repository for the userspace part. This would only cover Turing+ but it gives us coverage of everything NVIDIA has released in the past five years.

cries in Pascal but I agree that this is a good approach, specially for the future, having better documentation for the manual process would be also good, since AFAIK the place you can find how to do it is only on PRMFusion's website and not in our official docs.

to say that is a lot easier than to do that.

What version of open gpu are you building for what kernel?

one set of fw and open gpu version would only support one userspace driver. This isn't really the experience people will want. You can run NVIDIA 535.113.01 only and you can't get CVE fixes isn't really a win here.

More feedback from graphics team: "FWIW, the open source driver will not support a lot of hardware that definitely needs to be able to function with Fedora and (sadly) the proprietary NVIDIA kernel driver"

one set of fw and open gpu version would only support one userspace driver. This isn't really the experience people will want. You can run NVIDIA 535.113.01 only and you can't get CVE fixes isn't really a win here.

During today's meeting, we discussed the requirement that the version of the kernel module match the userspace driver. I think Neal and Kalev had a plan for how to handle that; of course it involves packaging one kernel module version per userspace driver version. But we didn't discuss the firmware version requirement. I didn't know that we needed to ship separate firmware too.

The firmware would be matched to userspace and kernel. Not a huge deal to package, it's just an extra thing.

so that won't be the same as "alongside shipping the nvidia-gsp-firmware in main Fedora for nouveau and this." nouveau will be fixed to very specific versions of the fw in very specific paths, it is not compatible from a firmware loading pov with the nvidia open-gpu driver.

But it is a version of the gsp firmware, so that can be included in the packaging.

there isn't just one firmware, it's a mess. It's not that simple. there won't be any sharing between nouveau and this effort of the firmware files.

If this effort wants firmware files in versioned packages like nvidia-gsp-firmware-535.113.01 and generates a set of those for each nvidia release, that is fine, but it should not be sold as overlapping or useful for nouveau.

so on RHEL nvidia also do a hard kernel version requirement? would be willing to do weak kernel version deps for Fedora, it might mean less builds.

On RHEL, I believe NVIDIA follows the weak-modules thing too.

they don't.

During today's meeting, we discussed the requirement that the version of the kernel module match the userspace driver. I think Neal and Kalev had a plan for how to handle that; of course it involves packaging one kernel module version per userspace driver version. But we didn't discuss the firmware version requirement. I didn't know that we needed to ship separate firmware too.

So Red Hat's graphics team is proposing that the NVIDIA driver package should simply fail to install if secure boot is enabled. Neal thinks there is no safe way to do this.

We do all agree that the NVIDIA driver package should not blacklist nouveau.

Hi @leigh123linux , I understand you're the primary maintainer of RPM Fusion's NVIDIA driver package? We should probably have brought you into this discussion sooner.

TL;DR: Fedora Workstation currently offers easy installation of the NVIDIA proprietary driver from RPM Fusion in GNOME Software, but the installation does not actually work if secure boot is enabled and it's expected to break nouveau. We've been discussing what to do about this for some time now (this issue is 3.5 years old) but don't have consensus.

We discussed this again today:

• We have consensus that the status quo is worse than not offering the NVIDIA driver at all. We expect that installing this driver will break graphics for most users because it won't work if secure boot is enabled, and it will denylist the nouveau driver to block any fallback to a working configuration. Users would be better off following documentation instructing them how to install the driver manually rather than using GNOME Software.

The rpmfusion driver will fallback to nouveau if the user forgets to enroll the akmods signing key.
Maybe I could add some info to the nvidia metainfo about extra steps required.

mokutil --import /etc/pki/akmods/certs/public_key.der

https://src.fedoraproject.org/rpms/akmods/blob/rawhide/f/README.secureboot

• We do not have consensus on how to proceed. Giving up and removing the driver instead of trying to fix it would look pretty bad and is not what users want, but this issue has been open for three years now so there is not very much hope of fixing things. I'd like to remove the NVIDIA driver from fedora-workstation-repositories unless a realistic, actionable, and resourced plan to fix it can be developed, but we do not have consensus for this. Other WG members prefer to focus on fixing it.

@aday we'd like to schedule this topic again for next week's Workstation WG meeting (October 31) alongside the planned discussion of #395, when we'll have guests from graphics team at the meeting. Their opinions would be welcome.

We discussed this again today:

• We have consensus that the status quo is worse than not offering the NVIDIA driver at all. We expect that installing this driver will break graphics for most users because it won't work if secure boot is enabled, and it will denylist the nouveau driver to block any fallback to a working configuration. Users would be better off following documentation instructing them how to install the driver manually rather than using GNOME Software.

The rpmfusion driver will fallback to nouveau if the user forgets to enroll the akmods signing key.
Maybe I could add some info to the nvidia metainfo about extra steps required.

mokutil --import /etc/pki/akmods/certs/public_key.der

https://src.fedoraproject.org/rpms/akmods/blob/rawhide/f/README.secureboot

That would be really helpful! At least then users have a chance of figuring out what to do next.

We discussed this again today:

• We have consensus that the status quo is worse than not offering the NVIDIA driver at all. We expect that installing this driver will break graphics for most users because it won't work if secure boot is enabled, and it will denylist the nouveau driver to block any fallback to a working configuration. Users would be better off following documentation instructing them how to install the driver manually rather than using GNOME Software.

The rpmfusion driver will fallback to nouveau if the user forgets to enroll the akmods signing key.
Maybe I could add some info to the nvidia metainfo about extra steps required.

mokutil --import /etc/pki/akmods/certs/public_key.der

https://src.fedoraproject.org/rpms/akmods/blob/rawhide/f/README.secureboot

That would be really helpful! At least then users have a chance of figuring out what to do next.

I'm not that good at wording stuff, a pr would be gratefully accepted.

$ cat /usr/share/doc/akmods/README.secureboot |fpaste
Uploading (2.6KiB)...
https://paste.centos.org/view/6ccfe07d

https://github.com/rpmfusion/xorg-x11-drv-nvidia

We could start building and shipping the open-gpu-kernel-modules in a side along repo as a kmod package that we sign alongside shipping the nvidia-gsp-firmware in main Fedora for nouveau and this.

The working group discussed this proposal during this week's call. We agreed three actions to move it along:

• @catanzaro to ask the graphics team about their thoughts on this plan
• @otaylor to ask kernel team if they will oppose packaging out of tree driver (we think this is likely)
• @ngompa to build driver package for testing

• @catanzaro to ask the graphics team about their thoughts on this plan
• @otaylor to ask kernel team if they will oppose packaging out of tree driver (we think this is likely)
• @ngompa to build driver package for testing

Right, but the feedback from the graphics team was quite negative. Since we're expecting negative feedback from kernel team too, I suggest we consider that plan rejected and abandon the other action items.

At this point we are looking at adding instructions for how to manually enroll the signing key to the appstream metainfo provided by RPM Fusion, so users who scroll down past the big images on the GNOME Software page have some chance.

I think most users will not scroll enough, though, or have sufficient technical expertise to follow the instructions if they do. We should consider design changes to the driver installation experience.

• @catanzaro to ask the graphics team about their thoughts on this plan
• @otaylor to ask kernel team if they will oppose packaging out of tree driver (we think this is likely)
• @ngompa to build driver package for testing

Right, but the feedback from the graphics team was quite negative. Since we're expecting negative feedback from kernel team too, I suggest we consider that plan rejected and abandon the other action items.

At this point we are looking at adding instructions for how to manually enroll the signing key to the appstream metainfo provided by RPM Fusion, so users who scroll down past the big images on the GNOME Software page have some chance.

I think most users will not scroll enough, though, or have sufficient technical expertise to follow the instructions if they do. We should consider design changes to the driver installation experience.

In the Linuxmint installer offers to do mok enrollment, could anaconda do the same?

It might be worth having it considered with the UI redesign going on for Anaconda for Fedora 40. @jkonecny could provide some guidance here.

But Anaconda can't install the driver, what key is it going to enroll?

It can just set up a machine owner key that akmods can then use later.

It can just set up a machine owner key that akmods can then use later.

but having a private key in the filesystem that can be used to sign the module locally kind of defeats the purpose of Secure Boot...

Because it means that an attacker that could escalate privileges (or get physical access to the machine) can access that key and sign a rootkit that could be part of the boot path.

The goal of Secure Boot is that only software from a trusted party would be booted but there is no guarantee that this will be the case if there's a cert enrolled in MOK with the private key always stored in the filesystem.

If this is the solution chosen, at the very least has to make it very clear to users what are the risks of doing that.

After reading the issue from the beginning I see that pjones already mentioned the same (e.g: https://pagure.io/fedora-workstation/issue/155#comment-664951) but in much more detail.

I'm thinking we might as well just tell users to turn off secure boot. Seems more honest.

In case it's relevant to the outcome of this issue: for a long time my view has been that the current NVIDIA install experience through Software isn't good. A good solution would:

• Suggest installing additional drivers if they are needed and available
• Show details about the relevant hardware
• Show which drivers are installed
• Allow switching between which driver is used

Software has none of this. Additionally, NVIDIA driver installation isn't very discoverable there.

but having a private key in the filesystem that can be used to sign the module locally kind of defeats the purpose of Secure Boot...

It's kind of the same issue as using TPM2 for LUKS unlocking, it's a trade-off between the most security and the most convenience. I'm usually not the type of person to say "JuSt GiVE tHe UsErS MoRE OpTiOnS" as if more options are always necessarily a good thing, but having an option for both of these in the installation process would likely be ideal. It seems to be working really well for Ubuntu and Mint.

Because it means that an attacker that could escalate privileges (or get physical access to the machine) can access that key and sign a rootkit that could be part of the boot path.

If an attacker can escalate privileges to this degree, at that point we have already failed our users at many other points in our security stack. This doesn't seem like something that can open a hole in the system's security by itself, at least.

A good solution would:

• Suggest installing additional drivers if they are needed and available
• Show details about the relevant hardware
• Show which drivers are installed
• Allow switching between which driver is used

Software has none of this. Additionally, NVIDIA driver installation isn't very discoverable there.

Could we just use Ubuntu's driver installation tool?

If an attacker can escalate privileges to this degree, at that point we have already failed our users at many other points in our security stack. This doesn't seem like something that can open a hole in the system's security by itself, at least.

The point is that Secure Boot prevents the attacker with root access from installing a rootkit, so you can get back to a safe state by reinstalling your OS. i.e. it limits badness in a situation in which you've already lost. Is there any other value in Secure Boot?

Could we just use Ubuntu's driver installation tool?

It'd be worth considering, if it's viable. From a quick search online, it looks like Ubuntu's driver installation UI is part of a more general "Software & Updates" tool.

There have been forks of it ported to Fedora before. I believe Korora did it once, I think? I guess we'd have to look to see if there's something to reuse.

The point is that Secure Boot prevents the attacker with root access from installing a rootkit, so you can get back to a safe state by reinstalling your OS. i.e. it limits badness in a situation in which you've already lost. Is there any other value in Secure Boot?

But the thing is, what is stopping an attacker with root access from doing the key signing process manually? I managed to create and enroll my own MOK keys really easily after I knew what I was doing, and I have zero technical knowledge on any of this. And you could always go back to a safe state by reinstalling your OS, since you'd create and enroll a different key every time with this process, right?

As stated before, I have zero technical knowledge on this, so I might be wrong, but isn't this how it would work?

But the thing is, what is stopping an attacker with root access from doing the key signing process manually? I managed to create and enroll my own MOK keys really easily after I knew what I was doing, and I have zero technical knowledge on any of this. And you could always go back to a safe state by reinstalling your OS, since you'd create and enroll a different key every time with this process, right?

Reinstalling your OS won't blacklist an existing key, so the old is still trusted unless you kill it. It wouldn't get rid of any malware installed into the EFI. I suppose there are people who would blindly accept a new key even though they never added one, but...

But the thing is, what is stopping an attacker with root access from doing the key signing process manually? I managed to create and enroll my own MOK keys really easily after I knew what I was doing, and I have zero technical knowledge on any of this. And you could always go back to a safe state by reinstalling your OS, since you'd create and enroll a different key every time with this process, right?

Reinstalling your OS won't blacklist an existing key, so the old is still trusted unless you kill it. It wouldn't get rid of any malware installed into the EFI.

Oh... yeah, that can open a whole can of worms then. I get it now.

Hi @mcrha, Allan mentioned you were doing some work on the NVIDIA driver installation workflow in GNOME Software. Could you link us to more information about this please?

Hi @nielsdg, Neal has mentioned that nouveau should work adequately for RTX 40+ graphics cards since this commit. Any thoughts on this?

I'm not sure how much more I want to care about this for the NVIDIA context.

At this point, Fedora Linux 40 is queued up with the initial revamped nouveau+gsp kernel driver and NVK in Mesa. At least for RTX 40x0 (Ada) GPUs, this will be active by default.

For GTX 16x0/RTX 20x0 (Turing) and RTX 30x0 (Ampere) GPUs, can be currently turned on via a boot flag nouveau.config=NvGspRm=1. At least for one of my alternative kernel builds, I've added
. This may turn into a distro kernel Kconfig thing in a future kernel release, based on what @airlied and I talked about at Linux Plumbers Conference last year and other places.

So then what's left for the proprietary driver? CUDA mainly. @karolherbst has been making progress on Rusticl though, and getting PyTorch to work through that would obviate most of the use-cases for CUDA.

Outside of CUDA, there's still the uncomfortable zone of GPUs from Maxwell 2 (GTX 900 series) and Pascal (GTX 10x0 series). These are still supported by the NVIDIA proprietary driver today and have poor support in Nouveau right now due to poor firmware provided by NVIDIA. But these GPUs are very old. Turing GPUs launched in 2018 (nearly six years ago!) and continue to be sold today.

I'm personally affected as an owner of a GTX 960 and I can't find it in myself to care anymore.

So, the question is: what should we do now? I still think our approach around kernel modules needs to change, but the urgency is quite low if we increasingly don't need to deal with the NVIDIA driver anymore. The only other out-of-tree drivers in common use are the VirtualBox host drivers (which we can avoid if we package the VirtualBox-KVM project) and the Broadcom Wi-Fi drivers for Macs (which is hopefully and finally getting fixed upstream now that the Asahi Linux folks (such as @marcan) are being allowed to be part of upstream driver maintenance).

If VirtualBox-KVM is packaged in Fedora, then I just don't know if we have anything else to worry about for the short-to-medium term.

I'm not sure how much more I want to care about this for the NVIDIA context.

As long as it's provided by fedora-workstation-repositories, it is our responsibility and we need to care for it.

I personally support removing the NVIDIA driver on the basis that nobody is interested in making it work properly in the 3.5 years this issue has been open. (Well, you were interested in making it work, but your proposals have all failed.) I'm not sure the WG will agree to this, though. Last time I proposed this, it failed. Still, the fact that the latest 40* series GPUs should no longer need the proprietary driver is very good news and materially changes things. I just hope users who try the new nouveau drive will agree....

Allan mentioned you were doing some work on the NVIDIA driver installation workflow in GNOME Software. Could you link us to more information about this please?

There is nothing public to link to, I only made some tests and it turned out there is nothing gnome-software can help here with, when the secure boot is on. After installing the drivers, users need to run:

    sudo mokutil --import /etc/pki/akmods/certs/public_key.der

choose a password, then reboot the machine, choose to enroll the key and re-type the password, and then reboot again, and that's all, the drivers work also with the secure boot on.

There are some tricky parts about setting the driver when on Wayland (I made it truly used only with X11), but that's out of this setup stage.

Once this Mesa PR is merged and built, Fedora 40 will have things working for the full range of GPUs supported by the nouveau GSP firmware.

At that point, my recommendation is that we should consider dropping the preloaded NVIDIA driver repository in fedora-workstation-repositories.

Agreed: we will remove the NVIDIA proprietary driver from fedora-workstation-repositories because we don't see any path to fixing things such that it will actually work after being installed by GNOME Software. We expect users will need to follow a series of command line instructions like this to install the driver successfully.

We will also recommend to the RPM Fusion maintainers that they not provide appstream metadata, to ensure it does not appear in GNOME Software.

I am here once again to ask: is this really necessary to do now of all times? The new drivers in Mesa are coming but have yet to be battle-tested as really working well on stable, and all this does is make installing the proprietary drivers harder for users that don't have their latest GPUs (specially not providing the appstream metadata). I don't see how this would improve anything for both users and maintainers.

Maybe it's just something personal for me (as I have a 1050Ti on my laptop with no upgrade in sight, as it's been serving me well with the rpmfusion-nonfree-nvidia-driver repo, it keeps me from needing to layer all of RPMFusion on my Silverblue install for just a driver).

What we still need to figure out: upgrade path. Ideally we don't want to break users who already have the NVIDIA driver installed from the rpmfusion-nonfree-nvidia-driver repo.

I am here once again to ask: is this really necessary to do now of all times? The new drivers in Mesa are coming but have yet to be battle-tested as really working well on stable, and all this does is make installing the proprietary drivers harder for users that don't have their latest GPUs (specially not providing the appstream metadata). I don't see how this would improve anything for both users and maintainers.

I don't think this makes it harder to install the drivers, because the current installation mechanism does not work (unless you just do not have secure boot enabled, but that would be unusual nowadays). It just doesn't make sense to provide an installation mechanism that doesn't work. And we clearly cannot fix it (this issue is 3.5 years old, and we've gotten nowhere). That is, this decision is orthogonal to whether the new GSP firmware works well or not. We do not know whether users will be happy with nouveau and the new firmware. Hopefully so, but I won't believe it until I see it. It just makes sense to do regardless.

To avoid any confusion, we don't suggest removing the rpmfusion-nonfree-nvidia-driver repo. That would be up to RPM Fusion maintainers to decide. We are just going to remove it from fedora-workstation-repositories. Actually, I suppose we could alternatively just remove the appstream metadata and keep the repo. That would accomplish the same objective. Hadn't considered that. The real goal is only to stop it from appearing in GNOME Software.

We could also subpackage it out so it's not available by default.

Let's wait for Kalev (he's sick currently) and see what implementation he prefers.

It isn't worth the effort to keep steam repo if you intend to drop nvidia.
I wont waste anymore time with this and will be dropping nvidia and steam support for all releases.
Nvidia appdate wont be removed from rpmfusion-appdata package as I refuse to hide them from users that require them.

Having nouveau GSP firmware is a very good news.

But for now the nvidia driver remains relevant for some workstation users specially for cuda use-case even if alternatives might exists.

Either way I'm fine with workstation removing the rpmfusion nvidia-driver repository for f40.

I also think end-users do not install using gnome-software given most documentations online do not advertise this install methods. Instead most end-users are directed to follow https://rpmfusion.org/Howto/NVIDIA or less backed documentations.
So I don't think having an exception for appstream metadata worth it.

For clarity, please remind that the current kmod packaging scheme do not require to disable secureboot as soon a a end-users own secureboot keys is created/installed. akmods will sign the produced kmod automatically. Improvements welcomed.

If any change where to be made, wrt having a secureboot signed pre-built kmod for either RHEL/Fedora or both, the process would be perfectly known. We just need to find some sponsor for new infra so this can be fully automated.

Also should the installation be done graphically and the nvidia drivers would be missing the secureboot key, the fallback to nouveau allows graphical boot anyway, so nothing to worry.

As for keeping nvidia-driver, rpmfusion might keep it to provide latest nvidia driver whereas main repository will keep the production branch. That's because some end-users experienced regressions with the latest serie.

But at the same time, most wayland enabled desktop end-users request the latest serie.

as soon a a end-users own secureboot keys is created/installed. akmods will sign the produced kmod automatically. Improvements welcomed.

We are missing a guided graphical interface for creating and managing user's own keys. If we had that and injected it somewhere early in the setup process, then we'd be "comparable" to other distributions.

Nvidia appdate wont be removed from rpmfusion-appdata package as I refuse to hide them from users that require them.

Well we've pretty clearly established that graphical installation doesn't work and is harmful to users, but... OK. We can remove both of the RPM Fusion repos from fedora-workstation-repositories. Shame to lose Steam as well, but users can install that from Flathub.

Unfortunately it's not really possible to create any upgrade path for users, so users who already have the NVIDIA driver or Steam installed from RPM Fusion will have no upgrade path. And it's not possible for us to fix that, because we cannot depend on the main RPM Fusion repos. :(

Agreed: we will remove the NVIDIA proprietary driver from fedora-workstation-repositories because we don't see any path to fixing things such that it will actually work after being installed by GNOME Software. ...

Personally I'm hesitant about this plan. The optics are bad and we need to be very careful about how it comes across. I think that there are some steps we can consider, like documenting and publicising the challenges and design options and roadblocks. We could also potentially consider half-way steps like removing the Software integration but leaving the other arrangements in place.

At the end of the day, the only real challenge is that we can't have a user-modifiable kernel keyring for signatures used to verify kernel modules.

Because we're required to push everything down into firmware, our options are extremely limited unless we compile an alternative kernel with another certificate (similar to what RHEL is doing in gl#cki-project/kernel-ark!2936) or we make a modified version of MokManager that lets us have a workflow that is non-interactive outside of the OS (like using some kind of TPM based encrypt/decrypt flow to make it "trustable" to auto-decrypt and auto-import at early boot).

Since we cannot do either of these things because the maintainers of those packages are uninterested in supporting that, we're stuck and therefore we have nothing we can do here.

Personally I'm hesitant about this plan. The optics are bad and we need to be very careful about how it comes across. I think that there are some steps we can consider, like documenting and publicising the challenges and design options and roadblocks.

Documenting things is useful to do (but of course won't fix the problems). We've done that for hibernation already.

We could also potentially consider half-way steps like removing the Software integration but leaving the other arrangements in place.

Well I just suggested this above (removing the appstream metadata), but it has already been rejected by Leigh, so I don't think we can do it. This actually seems like the best option to me, because then there are no changes for users who already have it installed: the upgrade path is preserved. But it's also outside our control. If Leigh and Nicolas don't want to do it, then it's not going to happen.

Sorry, I missed the meeting where things were decided and am not fully up to date because of that. I am a bit hesitant about this plan as well because it seems like we are throwing out the baby with the bathwater here.

As for the actual plan, instead of completely removing the repository, maybe temporarily removing the appstream data makes most sense, so that gnome-software wouldn't see that the driver is available to install? And add it back in the future if/when the gnome-software integration improves? This way, we don't break the installs for people upgrading to a new version of the distro, and also hide the brokenness from the GUI software installer.

@kwizart @leigh123linux Do you guys think it would be possible to compromise on this a bit? Maybe it would be possible to remove appstream data from the nvidia repo, and keep it for the main rpmfusion one, or something like that?

I also think this is something where we should have invited the rpmfusion people to our meeting instead of just announcing that this is how it's going to be. Dialog and communication is important.

Maybe it would be possible to remove appstream data from the nvidia repo, and keep it for the main rpmfusion one, or something like that?

I thought that's what we were doing. I agree with @kalev, we should discuss this with rpmfusion folks.

I thought that's what we were doing. I agree with @kalev, we should discuss this with rpmfusion folks.

What we agreed to do was to remove the repo outright.

But we didn't consider that all we really want to do is stop the NVIDIA driver from appearing in GNOME Software. For that, removing the appstream metadata would suffice. The repo itself is harmless.

The RPM Fusion folks are already here fortunately (@leigh123linux and @kwizart) but Leigh clearly does not want to remove the metadata. (I'm not sure why, since we know it does not work for most users. Is it really that useful to keep around for the few users who don't use secure boot?)

I thought that's what we were doing. I agree with @kalev, we should discuss this with rpmfusion folks.

What we agreed to do was to remove the repo outright.

But we didn't consider that all we really want to do is stop the NVIDIA driver from appearing in GNOME Software. For that, removing the appstream metadata would suffice. The repo itself is harmless.

The RPM Fusion folks are already here fortunately (@leigh123linux and @kwizart) but Leigh clearly does not want to remove the metadata. (I'm not sure why, since we know it does not work for most users. Is it really that useful to keep around for the few users who don't use secure boot?)

I was referring to this appdata packages.

rpmfusion-free-appstream-data-39-1.fc39.noarch
rpmfusion-nonfree-appstream-data-39-2.fc39.noarch

The repo appdata has been removed

deleting nvidia-driver/38/x86_64/repodata/appstream.xml
deleting nvidia-driver/38/x86_64/repodata/appstream-ignore.xml
deleting nvidia-driver/38/x86_64/repodata/appstream-icons.tar
deleting nvidia-driver/39/x86_64/repodata/appstream.xml
deleting nvidia-driver/39/x86_64/repodata/appstream-ignore.xml
deleting nvidia-driver/39/x86_64/repodata/appstream-icons.tar
deleting nvidia-driver/40/x86_64/repodata/appstream.xml
deleting nvidia-driver/40/x86_64/repodata/appstream-ignore.xml
deleting nvidia-driver/40/x86_64/repodata/appstream-icons.tar
deleting nvidia-driver/41/x86_64/repodata/appstream.xml
deleting nvidia-driver/41/x86_64/repodata/appstream-ignore.xml
deleting nvidia-driver/41/x86_64/repodata/appstream-icons.tar

sent 796,799 bytes received 10,577 bytes 322,950.40 bytes/sec
total size is 50,909,726,631 speedup is 63,055.78

Do you want me to do the same for the steam repo?

No, please. The steam repo is still good and useful.

What we agreed to do was to remove the repo outright.

Err, well what I wrote in the minutes is:

Consensus to remove it as an option in GNOME Software by removing appstream data

Thanks, Leigh!

Consensus to remove it as an option in GNOME Software by removing appstream data

OK, well one of us got confused. I'm not sure which. But I think we have consensus that we're happier with this outcome than with removing the repos outright. Thanks Leigh!

I'm finishing a change for the gnome-software to help the users and guide then what to do to enable the installed drivers in the system with the Secure Boot on. It's to be proposed to the upstream in some not so distant future. Missing appstream data makes it harder to test, but not a big deal either. Is there anything I should do when/if such change is approved upstream?

If the installation workflow is going to be fixed in Fedora 41, then we can ask Leigh to restore the appstream metadata for Fedora 41 and newer.