#155 NVIDIA driver and UEFI secure boot
Closed: Won't fix 14 days ago by catanzaro. Opened 2 years ago by ngompa.

The BIOS/UEFI thread on devel@ reminded me that we still have a serious unsolved problem for Fedora desktops using Secure Boot UEFI: the NVIDIA driver packages we offer do not work in this configuration.

Fedora has three startup modes: classical BIOS, UEFI, and Secure Boot UEFI. In the first two modes, everything is fine and the driver works. However, when we start up in the third mode, we have a problem. Fedora's kernel is configured to block kernel modules that are not signed with a recognized certificate from being loaded. This makes the NVIDIA driver enablement we have completely ineffective in this scenario.

Ordinarily, we generally seem to be expecting people to go into their firmware settings to turn off Secure Boot for this, however I do not believe this is acceptable. Moreover, with Fedora now being preloaded on laptops that have NVIDIA GPUs on them, we need a solution for this where users will not wind up being frustrated when trying to enable this through GNOME Software.

To make matters even worse, this is entirely self-inflicted: distributions like openSUSE and Ubuntu have functioning NVIDIA binary driver packages with UEFI secure boot, and the NVIDIA blob installer script will automatically generate a local cert and key set, import it into the kernel, and build the kernel module on the machine with that key so that it will load.

Our driver packages are provided by @kwizart, who currently does not know how to solve this problem for users running Fedora on Secure Boot UEFI systems that cannot or will not turn off Secure Boot. I know it is possible to produce signed kernel module packages, but I do not know how to with the Koji build system (I know how to with the Open Build Service and manually with a local RPM build).

From my perspective, we need to solve this problem to provide a good user experience to most (if not all) Linux notebook PC users with mid-range or high-end notebook PCs, as those typically have NVIDIA GPUs.


Right now there's not a great answer for this, nor a particularly good path to one. The best suggestion I have if you really want this is to hack on making DKMS able to sign drivers with e.g. a yubikey[0], make the kernel update know not to switch to the updated kernel until that step has been done, and then put more work into getting the user to occasionally actually do that part without leaving the yubikey plugged in all the time.

[0] see https://github.com/vathpela/pesign/blob/wip/src/efikeygen.1.mdoc

I'm not part of the "boot team" but to me I would think that this needs to be handled like you would handle replacing your own distribution's version of shim with another one and therefore require adding its public key as a machine owner key (MOK).

As an example if you intend to use Fedora's shim to boot for example Arch/CentOS/Debian/OpenSuse/RHEL/Ubuntu kernels you must obtain the Secure Boot public key associated with that target OS and add that public key right.

So the same chain of trust that is required to cross-boot distribution could be established with 3rd party repositories as in the 3rd party repo would have to provide Fedora with their public key to be included, Then that 3rd party ( or consumers of that 3rd party resources ) could sign a kernel and or related modules for nvidia, virtualbox etc. right.

Javier might be able to shed some light what policy ( chain of trust ) the "boot team" ( then presumably Fedora as well ) has adopted in this regard as in only trust ourselves ( Fedora ), Trust RH ecosystem ( Fedora,RHEL/CentOS ) Trust major distributions ( Arch/CentOS/Debian/OpenSuse/RHEL/Ubuntu) etc. as well as establishing if this is an valid/viable approach and policy ( I'm no expert in this field ) but personally and if doable I think this is the correct way of keeping the chain of trust.

If we dont trust a 3rd party enough to keep their keys with the shim we ship then arguably we dont trust it to expose our end users to that/those 3rd parties repositories to begin with.

That said I also want to point out to the "boot team" that whatever policy RH has in this regard does not mean it has to be Fedora's policy as in Fedora's policy could be more relaxed ( or stricter ) than whatever RH has ( if it has any such policy ).

It goes without saying if there is an argument against including 3rd party repository keys in the MOK database due to lack of trust ( security concerns ) then the chain of trust is already broken and we should not be exposing that or those 3rd party repository to our end user base to begin with but the fact is that a consensus has been made to allow third party source inclusions from certain sources [1] and arguably if rpmfusion provides the distribution with keys those keys should be include based on that consensus and thus it's DKMS + nvidia installer process under secure boot should just work...

  1. https://fedoraproject.org/wiki/Workstation/Third_Party_Software_Repositories

Something I fundamentally do not understand is why we cannot have a TOFU-like flow similar to what we do for GPG keys for package signatures for MOK keys. There should be no reason that RPM Fusion couldn't include in their release package their SB keys to lay down on disk so that when a kmod is installed from their repository, we can handle prompting to import the keys when a kmod package includes kmods signed with it.

Yes, it's not quite as simple as package GPG checks, and we have things like akmods and DKMS as a source of complexity for this, but we could develop something so that the user can easily set up their own local key for their own kmods and stuff installed via akmods or DKMS can auto-sign with that. At least with akmods, we're already at the point where the package content has been verified at least once through the repository chain (via GPG), and as long as we have an easy flow for revocation, then I don't see why this is a problem to implement.

Presumably you would just push an update that removes the relevant trusted 3rd party source from key database should it no longer be trusted for whatever reason ( and or blacklist it ) to much rejoice of end users if and then when that happens.

What is missing here ( afaikt ) is a policy that describes the process of providing the public keys to be included in the MOK database and the directions to do so which should be a part of the policy that describes how 3rd party repository can be considered officially part of Fedora and thus reference to as an option in the installer and the DE's as an additional "Trusted" source of packages ( which may contain drivers/modules ) . ( RH must have an "Partner Program" of some sort which a proposal could be based upon ) and the trusted key(s) from rpmfusion since it has already been officially approved as a 3rd party provide.

I would be surprised if this is a much of an technical issue on our ( Fedora's behalf ) or warranty any changes in our infrastructure...

Something I fundamentally do not understand is why we cannot have a TOFU-like flow similar to what we do for GPG keys for package signatures for MOK keys. There should be no reason that RPM Fusion couldn't include in their release package their SB keys to lay down on disk so that when a kmod is installed from their repository, we can handle prompting to import the keys when a kmod package includes kmods signed with it.

Right now it's an interactive process to enroll a key with the firmware. There's the mokutil side, where the user specifies a password; and upon reboot the mokutil EFI program runs in the pre-boot environment to confirm this key's addition by interactive prompt for the same password.

There should be no reason that RPM Fusion couldn't include in their release package their SB keys to lay down on disk so that when a kmod is installed from their repository, we can handle prompting to import the keys when a kmod package includes kmods signed with it.

There is no way to do mechanical, automated key enrollment securely without a verifiable chain of trust, and the scenario you describe does not have that. Any mechanical, automated enrollment without a chain of trust to verify it means malware can enroll keys as well. Importantly, that also means malware could use our bootloader on machines installed with other distros or OSes to enroll its keys on those machines during the boot process.

So metalink/gpg signed repodata -> gpg signed package -> package signature verified -> install -> load -> verify and load MOK isn't enough of a trust chain?

That doesn't sound right at all.

You can't do this kind of analysis without starting out in the context of the threat model being discussed. Within the secure boot context, there's nothing that has verified and trusts any of the repodata, gpg signatures, or any of that. We start by trusting the system owner, who can make modifications to what we trust, either in the firmware config or with mokutil (both of which require some degree of physical presence) and we trust the keys the system vendor has enrolled in the firmware. Everything else has to chain from there.

So then what, we need to vet and trust an RPM Fusion key and load it as part of Fedora then?

And how do you vet that? What is their trust model? Who has access to use the key? What is the audit process? What are their security practices?

And how do you vet that? What is their trust model? Who has access to use the key? What is the audit process? What are their security practices?

How is that done for Fedora itself? Someone has to vet that to grant Fedora a key.

Yes, that is correct. Very few people have access to the Fedora key. Any proven packager can build a kernel, but it won't be signed.

Yes, that is correct. Very few people have access to the Fedora key. Any proven packager can build a kernel, but it won't be signed.

While I think that's broken (there should be a way to sign with a key that isn't trusted so that the process always works the same), what stops us from putting together a program to vet RPM Fusion in the same way? They run infrastructure just like ours except for the undocumented parts (which include this stuff).

There is a process to sign with a key that isn't trusted, that is the Red Hat Test Key. If you build a kernel on a local machine, or if you tried to build a kernel in koji, it would be signed with that key. But as the key is not trusted, you can't boot that kernel with secureboot turned on.

There is a process to sign with a key that isn't trusted, that is the Red Hat Test Key. If you build a kernel on a local machine, or if you tried to build a kernel in koji, it would be signed with that key. But as the key is not trusted, you can't boot that kernel with secureboot turned on.

Okay, good to know, so we do have that process.

Also worth noting, you are asking Fedora to hand over trust to a 3rd part explicitly to use that trust to sign a chunk of binary which is not signed AFAIK by the developer, and where code is not available for review. There is literally no audit path here without nvidia directly involved.

You can't do this kind of analysis without starting out in the context of the threat model being discussed. Within the secure boot context, there's nothing that has verified and trusts any of the repodata, gpg signatures, or any of that. We start by trusting the system owner, who can make modifications to what we trust, either in the firmware config or with mokutil (both of which require some degree of physical presence) and we trust the keys the system vendor has enrolled in the firmware. Everything else has to chain from there.

The threat model was presumably discussed in the discussion surrounding the proposal of allowing 3rd party inclusion in Fedora and taking into consideration when the proposal was approved. At that point the decision was made that approved 3rd party should be treated and trusted as if they were part of Fedora itself ( the end user sees no difference ). It should have been pretty apparent when allowing 3rd party inclusion that Fedora had no control over those repositories components and infrastructure right ( It should have assess those prior to their approval hence my reference to a policy which should exist )

That said once a machine has been compromised it has been compromised and the scope of the damage it can do or has done becomes irrelevant unless the intent is going down the rabbit hole of speculation and hypotheticals and the expectation of the end user having a computer forensic on a retainer to dive into the scope of the breach. The fact is the end result always remains the same for the end user regardless if he has a single OS installed or multiple OS installed, the end user has to wipe his disk(s) clean and perform a re-install otherwise he can never be assured whatever compromised his system is still not in place...

The threat model was presumably discussed in the discussion surrounding the proposal of allowing 3rd party inclusion in Fedora and taking into consideration when the proposal was approved.

Citation needed.

The fact is the end result always remains the same for the end user regardless if he has a single OS installed or multiple OS installed, the end user has to wipe his disk(s) clean and perform a re-install otherwise he can never be assured whatever compromised his system is still not in place...

You've missed the point: without Secure Boot enabled (and working) you can still be rooted after the reinstall.

I would be okay with rpm fusion doing their own signing, and then providing a document for users to review their trust model and choose whether or not they import the rpm fusion key. Then a user can go through that PITA once, and have it go, but it also means rpmfusion would need to build the modules there, and a user can't do their own module builds.

The threat model was presumably discussed in the discussion surrounding the proposal of allowing 3rd party inclusion in Fedora and taking into consideration when the proposal was approved.

Citation needed.

Back in the day we strongly opposed inclusion or reference to 3rd party repos due to legal and security related issues and now that has changed and we have change proposals workflow in place precisely to allow discussions like these to take place prior to acceptance right and I'm presuming ( read as I expect but not looked into if that actually took place ) when the change proposal to include those repositories listed here [1] the scope of that change ( including it's legal and security context and how it affected the users ) took place. If that discussion never took place, then obviously the decision to include and refer to those 3rd party repo needs to be revisited ( and removed should it come down to that ) and our process looked at to prevent such decisions be made in the future.

The fact is the end result always remains the same for the end user regardless if he has a single OS installed or multiple OS installed, the end user has to wipe his disk(s) clean and perform a re-install otherwise he can never be assured whatever compromised his system is still not in place...

You've missed the point: without Secure Boot enabled (and working) you can still be rooted after the reinstall.

Right but the same proposed action ( steps end users should take ) for the end user applies regardless if it's some approved 3rd party or it's Fedora.

JBG

  1. https://fedoraproject.org/wiki/Workstation/Third_Party_Software_Repositories

I would be okay with rpm fusion doing their own signing, and then providing a document for users to review their trust model and choose whether or not they import the rpm fusion key. Then a user can go through that PITA once, and have it go, but it also means rpmfusion would need to build the modules there, and a user can't do their own module builds.

If @kwizart can be told how to implement this, or at least if it can be documented somewhere how this works for Fedora at all, then I think this may be possible.

One potential clarification is that I'd like for it to be possible for RPM Fusion to have a trusted bot that watches for new Fedora kernels and would auto-rebuild kmod packages with the new kernel, since Koji can't do it by itself like other build systems can. Including that in the trust model would make it a lot saner to retire the akmod strategy RPM Fusion uses today for kernel modules.

I would be okay with rpm fusion doing their own signing, and then providing a document for users to review their trust model and choose whether or not they import the rpm fusion key. Then a user can go through that PITA once, and have it go, but it also means rpmfusion would need to build the modules there, and a user can't do their own module builds.

The secure boot process needs to be as transparent as it is if it's disabled so that does not help. ( the 3rd party needs to be included by default ) If it's usability suck or is filled with false positives end users will just turn it off like they have done with selinux for decades.

If we dont trust the 3rd party we should not be including/referring to it and offer application ( or drivers in this case ) that reside in it's repositories because the fact is end users aren't able to differentiate whether a component came from Fedora or from a 3rd party ( repo or application market place ) it's all just "Fedora" to him ( unless he manually has to jump through hoops to set it up ) and expecting (thus providing solutions ) that expect some kind of informed decisions about security related things from an end users is just ridiculous.

The end user places his trusts on the OS on doing the right thing for him.
Whether that turns out to be misplaced trust is entirely up to the OS, 3rd party repo or application marketplace and if anything fails Fedora as an OS will be blamed for it because that's the OS the end user is using.

I would be okay with rpm fusion doing their own signing, and then providing a document for users to review their trust model and choose whether or not they import the rpm fusion key. Then a user can go through that PITA once, and have it go, but it also means rpmfusion would need to build the modules there, and a user can't do their own module builds.

If @kwizart can be told how to implement this, or at least if it can be documented somewhere how this works for Fedora at all, then I think this may be possible.
One potential clarification is that I'd like for it to be possible for RPM Fusion to have a trusted bot that watches for new Fedora kernels and would auto-rebuild kmod packages with the new kernel, since Koji can't do it by itself like other build systems can. Including that in the trust model would make it a lot saner to retire the akmod strategy RPM Fusion uses today for kernel modules.

I don't know that I would have an issue with a bot doing the builds, but the whole value to this, is I don't have to. The document lays out what is happening, and each user can decide if they are comfortable with it or not.

We could probably swipe the license and notice metadata extension that SUSE uses for rpm-md to present that document, so that's something @dmach's team might be interested in to support this flow and bubble it up to GNOME Software.

I would be okay with rpm fusion doing their own signing, and then providing a document for users to review their trust model and choose whether or not they import the rpm fusion key. Then a user can go through that PITA once, and have it go, but it also means rpmfusion would need to build the modules there, and a user can't do their own module builds.

The secure boot process needs to be as transparent as it is if it's disabled so that does not help. ( the 3rd party needs to be included by default ) If it's usability suck or is filled with false positives end users will just turn it off like they have done with selinux for decades.

Which is likely what they are doing right now. This at least makes it a 1 time thing you do per machine, and manageable as opposed to something you must do every kernel update.

If we dont trust the 3rd party we should not be including/referring to it and offer application ( or drivers in this case ) that reside in it's repositories because the fact is end users aren't able to differentiate whether a component came from Fedora or from a 3rd party ( repo or application market place ) it's all just "Fedora" to him ( unless he manually has to jump through hoops to set it up ) and expecting (thus providing solutions ) that expect some kind of informed decisions about security related things from an end users is just ridiculous.

There are various levels of trust. Trusting an mpeg player is a bit different than trusting a kernel module. If "it's all just Fedora" to the user, then we have a problem. Third party repositories should not present as "just Fedora". The entire reason that they exist is because the things packaged there do not meet the qualifications of Fedora principles.

The end user places his trusts on the OS on doing the right thing for him.
Whether that turns out to be misplaced trust is entirely up to the OS, 3rd party repo or application marketplace and if anything fails Fedora as an OS will be blamed for it because that's the OS the end user is using.

As the person who builds signed kernels, I do not extend my trust to the 3rd party repo, and to nvidia (which is the extent of the trust chain in this scenario). This is not just "do I trust @kwizart to do the right thing", but do I trust kwizart, the maintainers of the signing key and the infrastructure that it depends upon within rpmfusion, the developers at nvidia, and whatever infrastructure they have in place for their binary since we can't see the source that is used to build it. Even if I am personally willing to extend my trust that far for my machines, I certainly couldn't say so on behalf of all Fedora users.

I'm a little lost in this discussion without a clear definitions:

  • What benefits do Fedora users gain by enabling secure boot
  • What do we need to do to avoid our secure boot infrastructure being used to attack other operating systems

[ https://mjg59.dreamwidth.org/12368.html has some background on the original thinking behind disabling unsigned modules - which is apparently the key difference between us and other distros that have less issues here ]

@pjones : can you explain:

You've missed the point: without Secure Boot enabled (and working) you can still be rooted after the reinstall.

What do we need to do to avoid our secure boot infrastructure being used to attack other operating systems

Keep the chain of trust as short as possible as in limit it only to Fedora ( Or Fedora/Centos/RHEL for cross-booting support strictly in the RH ecosystem which RH controls) but that of course comes at the prices of excluding 3rd party repositories and other OS ( cross-booting becomes unsupported, that is if it's even supported today )

Bottom line the more you add to a chain of trust, the weaker it becomes.

@pjones : can you explain:

You've missed the point: without Secure Boot enabled (and working) you can still be rooted after the reinstall.

So, the straightforward thing to do if you have a Secure Boot breakout is obviously to use it as a bootloader and load modified version of the OS - i.e. backdoor the kmod verifier while loading the kernel. This what's known as a "bootkit", and that kind of backdoor is really the simplest variety - it's not hard to find enough code to easily implement that much by googling. This is clearly pretty bad, but it's also not limited to one OS - you need different backdoor code for each target, but once you've found a bootloader or kernel that doesn't properly check signatures, or trusts a signature that it should not, or is suceptable to the right kind of exploit, you can just install that in front of other OSes as the starting point.

But that's not the worst thing attackers have done. If the attacker is at all sophisticated, they can additionally replace UEFI APIs for disk access, boot variables, and even the system firmware boot selection menus. If they do that, then they can make it appears as though you've booted install media when you've actually still got the same backdoor, and it's trivial to install the exploit again during the re-install. This does not require all that much work to do poorly as a proof of concept; how much work the attacker puts into it developing the bootkit mostly determines how hard it is to detect that they've done this and how hard it is to fix it once you've noticed.

This is what attackers will do when they don't have some other vulnerability available, like flash not being properly locked. The "hackingteam" bootkit that was leaked in 2015 includes a module that will re-link your system firmware with their backdoor in the loader - but on some classes of machines with the vulnerability that allows them to do that, they still need to be able to run pre-boot code to actually modify it.

MokManager/mokutil is a compromise between being more secure against this kind of attack and being more practical - we allow you to enroll keys as the owner of the machine, but you need some degree of physical access to do so. The result is that if there is some weakness in what you've signed and are loading with mok, a very tenacious attacker will still be able to make progress on your system, but the attack won't be very portable to other systems.

Having said all that - can we please not argue all of the decisions of Fedora 18? We discussed this to death at the time, and I've got enough going on right now without doing it again.

  1. Protection from: firmware implants, persistent attacker controlled system management modules, various kinds of OS Loader malware, a compromises kernel, including any kernel module being used as malware or as part of a chain of exploits to achieve ultimate ownership of the system. Numerous examples exist in the wild.
  2. Keep control of the Fedora (private) signing key.
  3. Without UEFI Secure Boot enabled and working, a number of locations exist that are not subject to being reset by a typical clean OS install. This includes the EFI system partition, and the chip the firmware itself is located on. Once the system is owned, it can be indistinguishable from not being owned using any familiar friendly UI, i.e. the firmware setup UI is not reliable.

Once you've disabled UEFI Secure Boot, a compromised (fake) kernel module can drop a malicious system management module onto the system, which in turn can make the flash where the firmware itself lives, writeable. Persistent malware achieved. Or even the attacker's own signing key, so they can run whatever they want. This system is now properly owned in a way that a clean install can't fix. Even if you go to the uncommon trouble of secure erasing the drive first. And this could all start with a remote attack from a web site using a browser, it doesn't require physical access ala Evil Maid attack.

And now imagine this computer happens to have possibily important private keys on it. And next wonder how you're going to get out of this hell hole without fire.

It's reasonable to consider ways to make it easier for users to install drivers that they vouch for being from a trusted source, however unqualified that is, and helping them sign those modules and enroll the key. But I'm a little skeptical of how it works for Fedora to sign 3rd party keys: it's not just a matter of authenticating current practice but verifying it on an on-going basis. If I thought that making this easier means few intentionally SB disabled systems, i'd be more supportive. I get super grouchy when I read folks on support channels and lists advocating just disable UEFI Secure Boot.

Issue #116 applies here too.

Having said all that - can we please not argue all of the decisions of Fedora 18? We discussed this to death at the time, and I've got enough going on right now without doing it again.

I don't want to argue all of that all over again. If anything, I still agree with the decisions @mjg59 did back then. I just want us to come up with something where we don't frustrate users who are trying to make their hardware work, especially now that we're moving into having Fedora preloaded on computers.

Having said all that - can we please not argue all of the decisions of Fedora 18? We discussed this to death at the time, and I've got enough going on right now without doing it again.

I don't want to argue all of that all over again. If anything, I still agree with the decisions @mjg59 did back then. I just want us to come up with something where we don't frustrate users who are trying to make their hardware work, especially now that we're moving into having Fedora preloaded on computers.

I suggested something that's better than what we've got in https://pagure.io/fedora-workstation/issue/155#comment-664899 . It doesn't seem like you've replied to it.

Having said all that - can we please not argue all of the decisions of Fedora 18? We discussed this to death at the time, and I've got enough going on right now without doing it again.
I don't want to argue all of that all over again. If anything, I still agree with the decisions @mjg59 did back then. I just want us to come up with something where we don't frustrate users who are trying to make their hardware work, especially now that we're moving into having Fedora preloaded on computers.

I suggested something that's better than what we've got in https://pagure.io/fedora-workstation/issue/155#comment-664899 . It doesn't seem like you've replied to it.

Sorry, I didn't notice that. That option is somewhat doable if we could assume people had YubiKeys. Another way that could work would be to do this with a TPM device and leverage that to essentially tie the signing to the device, but that doesn't help with the "conscious act" thing you're going for here.

If akmods was tweaked and we had a GUI frontend for it that helped with doing that process properly, it might work, though. However, I don't want to do this with DKMS because it doesn't produce a trackable result (i.e. a kmod RPM that's installed on the system).

Having said all that - can we please not argue all of the decisions of Fedora 18? We discussed this to death at the time, and I've got enough going on right now without doing it again.
I don't want to argue all of that all over again. If anything, I still agree with the decisions @mjg59 did back then. I just want us to come up with something where we don't frustrate users who are trying to make their hardware work, especially now that we're moving into having Fedora preloaded on computers.

I suggested something that's better than what we've got in https://pagure.io/fedora-workstation/issue/155#comment-664899 . It doesn't seem like you've replied to it.

Sorry, I didn't notice that. That option is somewhat doable if we could assume people had YubiKeys. Another way that could work would be to do this with a TPM device and leverage that to essentially tie the signing to the device, but that doesn't help with the "conscious act" thing you're going for here.
If akmods was tweaked and we had a GUI frontend for it that helped with doing that process properly, it might work, though. However, I don't want to do this with DKMS because it doesn't produce a trackable result (i.e. a kmod RPM that's installed on the system).

Which brings us back to, there is nothing that stops rpm fusion, or any other 3rd party from creating their own signing infrastructure, and detailing a step for users to put their single key into the trusted ring on their system. This has been the case since we first implemented secure boot. I am guessing the only thing stopping them is the assumption of responsibility there lies squarely on the repository should things be mismanaged.

Actually, from what I can tell, it's that nobody knows how to sign kernel modules for kmod packaging.

Actually, from what I can tell, it's that nobody knows how to sign kernel modules for kmod packaging.

pesign 113, which is in f33 currently, can sign kernel modules.

One way to solve this would be to move the nvidia open source kernel module building to Fedora's koji and build it together with the kernel. This would (1) make sure it exactly matches the kernel version, and (2) ensure it gets signed with Fedora key.

The main issue I see with this is that it's occasionally going to hold back kernel updates if the interface changes and the module doesn't build any more. Sadly, I don't think we have many alternative options if we want to make this work well for nvidia / Lenovo laptop users and ensure they always have access to a nvidia driver :(

I'd rather not have it bundled in the kernel tree. If we're going to do that, let's figure out how to build kmod packages in Fedora Koji instead.

One way to solve this would be to move the nvidia open source kernel module building to Fedora's koji and build it together with the kernel. This would (1) make sure it exactly matches the kernel version, and (2) ensure it gets signed with Fedora key.

You can't really build it "together" with the kernel, but it would be very easy to auto bump/build when a successful kernel build completes on a stable Fedora branch. Rawhide, you can try, there is no guarantee that it would succeed though, and Nvidia doesn't actually care. Their "policy" is no support for pre-release kernels. Typically this means they will release a fixed build shortly after Linus releases a .0

The main issue I see with this is that it's occasionally going to hold back kernel updates if the interface changes and the module doesn't build any more. Sadly, I don't think we have many alternative options if we want to make this work well for nvidia / Lenovo laptop users and ensure they always have access to a nvidia driver :(

Our policy was that we would not hold back kernel updates on the failing of nvidia, and when 3rd party was discussed there was supposed to be something that handled not booting from a new kernel until nvidia was available for users running it. In practice, we don't rebase until 3-4 weeks after Linus does a .0 update, and nvidia is working by then. In the rare occasion that a stable update has broken things, we tend to revert the patch that does so, provided it isn't security related.

I'd rather not have it bundled in the kernel tree. If we're going to do that, let's figure out how to build kmod packages in Fedora Koji instead.

No, it cannot be bundled in the kernel tree. And we really do not want to get into the business of kmod packages in Fedora proper. We could come up with a process to build select kmod packages on Fedora infrastructure, sure. But that is going to have to be much more tightly controlled than the standard "add a package to Fedora process". For instance, any proven packager can technically commit to the kernel or grub, or any of the signed packages, but if they build it, it will not be signed. Signature access is very intentionally tightly controlled.

I'd rather not have it bundled in the kernel tree. If we're going to do that, let's figure out how to build kmod packages in Fedora Koji instead.

Sure, that's really up to the kernel maintainers what way they prefer it.

The main thing is that I think we (Fedora) just need to take over the responsibility of rebuilding the nvidia kernel module. If it's in rpmfusion we have no way to ensure it gets updated/rebuilt in lock step together with kernel updates and correctly signed.

Sure, that's really up to the kernel maintainers what way they prefer it.
The main thing is that I think we (Fedora) just need to take over the responsibility of rebuilding the nvidia kernel module. If it's in rpmfusion we have no way to ensure it gets updated/rebuilt in lock step together with kernel updates and correctly signed.

It really needs to be special cased. The nvidia package goes against the Fedora principals, and can't really be a package in Fedora proper, so it would need to be built on Fedora koji, but still distributed through a 3rd party repo.

It really needs to be special cased. The nvidia package goes against the Fedora principals, and can't really be a package in Fedora proper, so it would need to be built on Fedora koji, but still distributed through a 3rd party repo.

That sounds like a good plan to me. Thanks, @jforbes! :)

If it's a separate repo, that may actually offer a functional advantage: We could have a slightly different repo structure than what regular Fedora repos have, and instead of just shipping the latest package, we could ship all the versions we've ever built in it, so that users can easily install the correct module for the exact version of kernel they have installed.

This is similar to what openSUSE does, too. The NVIDIA driver is built in their build system, and then NVIDIA hosts it themselves on download.nvidia.com for openSUSE users to consume.

Cf. https://en.opensuse.org/SDB:NVIDIA_drivers

Metadata Update from @chrismurphy:
- Issue untagged with: meeting-request
- Issue tagged with: meeting

2 years ago

Metadata Update from @chrismurphy:
- Issue untagged with: meeting

2 years ago

@ngompa can you update us on the status of this issue?

I've been poking at this off and on for months, and today I've talked to @pjones about this more in #fedora-devel:matrix.org. There may be some progress to be made here, though @pjones can give more color in here on this topic.

I think the way to move forward with this is probably some variation on the plan below, which has components to add mechanisms for the following critical abilities: 1) to limit what certain signing certificates can be used for, 2) to simplify the inevitably scenarios where we need to issue revocations, 3) to allow admins to easily and reliably opt-out, whether to all kmods, specific kmods, or specific builds.
- Implement the critical x.509 EKU "moduleSigningOnlyKey" everywhere (might actually be done everywhere but kexec?), so we can make signing certificates that are only good for kernel modules.
- Make a way for admins to add revocations locally, both during boot and later, which isn't reversible without a reboot. I.e. some way to tell the kernel "stop trusting this now". In my mind that means adding a root-privileged keyctl method to add things to our disallowed-list, which dosn't affect anything that's already loaded. This lets us say "I've loaded all the 3rd party kmods I'm going to", as well as being the key to giving admins an opt-out mechanism.
- Add a built-in revocations list similar to shim's internal dbx to the kernel, so we can have a "all kernels going forward do not trust this {ca cert, signing cert, built module}" without getting a shim build signed.
- Add ".sbat" revocation support to the kernel's module loader, using a SBAT rule list added during the kernel build, suplemented by a similar mechanism to the keyctl above. This lets us say "all modules with the property 'fedora.kmod' with a value <= 3 are revoked", greatly simplifying revocations of /classes/ of things. We have to decide on the classes ourselves, but any given class of things more or less maps to one well known set of vulnerabilities or risks that have been eliminated. See https://github.com/rhboot/shim/blob/main/SBAT.example.md and https://github.com/rhboot/shim/blob/main/SBAT.md for details on how this works.
- Define a set of non-authenticated revocation files that go on the filesystem i.e. /etc/kernel/revocations.d/{01-kmod.esl,01-kmod.sbat} for admins to use, with a systemd service to load them and a dracut module to load them very early. This lets sysadmins set revocation policy with approximately the same threat model as against "rm vmlinuz".
- Add some sample sets of revocations that an admin can trivially copy to /etc/kernel/revocations.d/ to disable support for {all 3rd parties, all of some particular kmod, etc}.
- Issue a cert that has moduleSigningOnlyKey set for signing each kmod we decide to trust, which chains to some CA we decide.
- We may also want to add a definition for moduleSigningOnlyKey on CA certs, that specifies that cert is only allowed to issue certs with a critical moduleSigningOnlyKey EKU set. This might also be better as a second OID.

Note that what I've specified above is only a mechanism to safely load modules we've assigned trust to which aren't from the kernel package. Fundamentally we're still talking about adding third parties that can build trusted kernel modules, even if we have them do so in fedora kmod packages, and so we still need to have conversations and make formal decisions about:
- whether we're talking about kmods in fedora packages or external repos
- who we're trusting, why, and what the procedures around establishing that trust are.
- issues of key storage and authentication for building signed kmods in fedora
- what sort of topology we want for the certs, what security guarantees they provide against what threat model.

Also note that several of the things above are not my department, and we need to get @dhowells involved in this discussion.

Metadata Update from @ngompa:
- Issue tagged with: experience

2 years ago

Metadata Update from @ngompa:
- Issue set to the milestone: Fedora 36 (was: Future Release)
- Issue tagged with: nvidia

2 years ago

We had a brainstorming meeting discussing this problem today with Justin and Peter. Currently we are far from a solution. Meeting minutes will be made available as usual, but we discussed so much that we aren't prepared to adequately summarize the discussion here. Instead, we're planning to post some follow-up questions for Justin and Peter here to try to understand things better.

We had a brainstorming meeting discussing this problem today with Justin and Peter. Currently we are far from a solution. Meeting minutes will be made available as usual, but we discussed so much that we aren't prepared to adequately summarize the discussion here. Instead, we're planning to post some follow-up questions for Justin and Peter here to try to understand things better.

So I think we have a solution, which may land in 5.17, as a side effect of the work that Edge/IoT is doing with IMA.

These are the follow-up questions @catanzaro mentioned two days ago upthread, and are directed at @pjones @jforbes

Allowing software to automatically import additional UEFI Secure Boot keys is not acceptable; explicit user intervention (via mokmanager ui) is required to add 3rd party keys?

Considering limited NVRAM spec, do you have any concern about adding multiple Secure Boot keys? Or the accumulation of stale keys?

Fedora private keys will not be shared with a third party, including RPM Fusion, e.g. Fedora Secure Boot CA, Fedora kernel signing key?

It looks like openSUSE locally builds the NVIDIA kernel module with locally created keys - each time the driver is updated (maybe also whenever the kernel version is bumped) there's a mokmanager "Enroll MOK" and "Delete MOK" event. This seems onerous. And maybe also impractical if we need to keep 3 keys enrolled (one per kernel).
https://en.opensuse.org/SDB:NVIDIA_drivers

Would Fedora be willing to trust (sign) a 3rd party's key for either kernel modules or UEFI Secure Boot, with a derived key intended specifically for this purpose? That way it's easier to revoke than either the Fedora kernel key or Secure Boot CA key?

These are the follow-up questions @catanzaro mentioned two days ago upthread, and are directed at @pjones @jforbes

Allowing software to automatically import additional UEFI Secure Boot keys is not acceptable; explicit user intervention (via mokmanager ui) is required to add 3rd party keys?

Yes, a user has to be aware that they are trusting something besides Fedora proper. I don't care how good RPMFusion's policies might be, they are packaging a large binary blob with a small ope n source shim with kernel permissions. This is not something to take lightly.

Considering limited NVRAM spec, do you have any concern about adding multiple Secure Boot keys? Or the accumulation of stale keys?

No, realistically I would be surprised to see anything more than 2-3 keys imported to any machine.

Fedora private keys will not be shared with a third party, including RPM Fusion, e.g. Fedora Secure Boot CA, Fedora kernel signing key?

Absolutely not. If we were to go into any role where we are doing the build and signing, it would have to be with a new key. We wouldn't even build nvidia with the Fedora keys if we were doing the build ourselves, so we certainly will not share our private keys with anyone under any circumstances.

It looks like openSUSE locally builds the NVIDIA kernel module with locally created keys - each time the driver is updated (maybe also whenever the kernel version is bumped) there's a mokmanager "Enroll MOK" and "Delete MOK" event. This seems onerous. And maybe also impractical if we need to keep 3 keys enrolled (one per kernel).
https://en.opensuse.org/SDB:NVIDIA_drivers

That seems unnecessarily problematic. You can achieve similar results by generating a per machine key and keeping it around. Patches exist for akmod for this. Nothing about the SuSE solution seems desirable at all.

Would Fedora be willing to trust (sign) a 3rd party's key for either kernel modules or UEFI Secure Boot, with a derived key intended specifically for this purpose? That way it's easier to revoke than either the Fedora kernel key or Secure Boot CA key?

While a 3rd party would need need their own key, I still do not think it wise for Fedora to trust it by default. We do not control their build environment, and even they do not know what code went into the binary blob that nvidia ships with their drivers. I believe we need a firm user acknowledgement that they are explicitly trusting a 3rd party key for an specific purpose. More importantly, if something were to happen where that 3rd party key were compromised, at least the damage would be limited in the worst case to users who chose to trust that key. If it were trusted by default, every Fedora system would be vulnerable, even if they never used that 3rd party repository. We can certainly revoke that trust, but not necessarily before the damage is done, and it would depend on user updates which are not always consistent. More importantly a user might look at a security advisory and assume it does not apply to them because they have never interacted with that 3rd party.

While a 3rd party would need need their own key, I still do not think it wise for Fedora to trust it by default.

One clarification: we really only need to trust such a key in case the user elects to install the NVIDIA driver. We certainly don't need users who don't use the NVIDIA driver to have any such trust. So with that additional limitation, does that change your answer?

(I guess it should be obvious that a driver can do whatever it wants to your computer regardless of any secure boot considerations. Users already trust the driver not to traverse their home directory and upload all their files to command and control. I don't see a particularly huge distinction between that level of trust vs. getting a key in the kernel keyring.)

On Thu, Dec 2, 2021 at 4:39 PM Michael Catanzaro pagure@pagure.io wrote:

catanzaro added a new comment to an issue you are following:
``

While a 3rd party would need need their own key, I still do not think i=
t wise for Fedora to trust it by default.

One clarification: we really only need to trust such a key in case the us=
er elects to install the NVIDIA driver. We certainly don't need users who d=
on't use the NVIDIA driver to have any such trust. So with that additional =
limitation, does that change your answer?

(I guess it should be obvious that a driver can do whatever it wants to y=
our computer regardless of any secure boot considerations. Users already tr=
ust the driver not to traverse their home directory and upload all their fi=
les to command and control. I don't see a particularly huge distinction bet=
ween that level of trust vs. getting a key in the kernel keyring.)
``

I still firmly believe that a user must acknowledge that they are
extending their trust beyond Fedora willingly, so even with that
limitation, it does not change my answer.

a user must acknowledge that they are extending their trust beyond Fedora willingly

The user must also agree to a proprietary license at some point (to be determined) but let's say the sequence looks like this:

1 GNOME Initial Setup shows this 3rd party repositories page
Screenshot_from_2021-12-03_10-31-40.png
There is a clear statement "some proprietary software is included", these repositories are disabled by default, the user needs to take explicit action to enable them.

2 Once 3rd party repositories is enabled

$ sudo dnf repolist
repo id                                 repo name
fedora-cisco-openh264                   Fedora rawhide openh264 (From Cisco) - x86_64
google-chrome                           google-chrome
phracek-PyCharm                         Copr repo for PyCharm owned by phracek
rawhide                                 Fedora - Rawhide - Developmental packages for the next Fedora release
rawhide-modular                         Fedora - Modular Rawhide - Developmental packages for the next Fedora release
rpmfusion-nonfree-nvidia-driver         RPM Fusion for Fedora rawhide - Nonfree - NVIDIA Driver
rpmfusion-nonfree-steam                 RPM Fusion for Fedora rawhide - Nonfree - Steam

The GNOME Software equivalent listing
Screenshot_from_2021-12-03_10-49-57.png

3 Choose to install the NVIDIA driver
4 See the EULA for the NVIDIA driver

That's at least two (possibly four) acknowledgements to extend trust beyond Fedora. Requiring a yet another acknowledgement is less about what or why, but where the acknowledgement is happening: user space vs preboot environment.

I think @pjones and @jforbes are saying user space isn't as trusted an environment for enrolling Secure Boot keys, relative to the preboot environment? Despite SELinux?

The points where I'm still stuck:

  • Fedora already extends trust, by default, and without notification or opt-out, to linux-firmware which is a pile of largely proprietary binary blobs. Why is that given such a huge waiver? Practicality. Is there any other reason?
  • Is it really appropriate for Fedora or RPM Fusion to sign proprietary software directly? I think NVIDIA should sign their binary blobs. Can kernel modules have multiple segments signed with different keys?
  • Could the Fedora kernel contain a key to extend trust to other keys, instead of signing kernel modules? i.e. a "Fedora 3rd party RPM Fusion & NVIDIA signing key" which signs whatever NVIDIA key is used to sign their binary blob, and also RPM Fusion's key used to sign the rest of the kernel module. And none of these keys would be enrolled with mokmanager. (While this intermediate signing step results in defacto trust in proprietary software, I think it's important that FOSS projects aren't directly extending trust to the software, but rather to those company's signing keys.)
  • Is such a paradigm more onerous to build and document, than one where signing keys are produced and retained locally. Which is safer? My strong suspicion is that once it is known there's a path to a local private key that can be used to sign kernel modules - it'll be recognized as a pretty big door.

I asked in F36 Change: DIGLIM (System-Wide Change proposal) if the feature could help with this issue. The response from @robertosassu is here. My take away is that it could help, but isn't part of the current design.

What's clear is we need someone who's capable of doing the work to help us understand the options, for example:

Whether it is better to trust an external repository to
adequately protect the private key used for signing, or to
keep the private key in the system that would load the
third-party kernel modules is a difficult decision, and probably
each user would like to make that decision.

Does this translate into providing both options? And if so what does that look like in terms of development: Fedora infra, 3rd party (RPM Fusion), and the UI.

I asked in F36 Change: DIGLIM (System-Wide Change proposal) if the feature could help with this issue. The response from @robertosassu is here. My take away is that it could help, but isn't part of the current design.

Unfortunately, even by whitelisting third-party kernel modules would
still not be enough due to the fact that kernel modules are verified
with an independent mechanism than the integrity subsystem.

What's clear is we need someone who's capable of doing the work to help us understand the options, for example:

Whether it is better to trust an external repository to
adequately protect the private key used for signing, or to
keep the private key in the system that would load the
third-party kernel modules is a difficult decision, and probably
each user would like to make that decision.

Does this translate into providing both options? And if so what does that look like in terms of development: Fedora infra, 3rd party (RPM Fusion), and the UI.

I would be more in favor of having Fedora providing the
infrastructure for building third-party kernel modules, although
it does not vouch for the content it is building.

This would still provide much stronger guarantees about what
is loaded by the user. The Fedora infrastructure would verify
the authenticity of the NVIDIA blob used as input for building
the kernel module, and would guarantee that the packages
used for the building process are good (they come from
Fedora itself).

I personally don’t think having the private key in the same
machine that uses the kernel modules signed with that key would
provide security. I still would hope that there is an infrastructure
(even if it is not the Fedora one) users can rely on to correctly
build and sign the kernel modules. In that case, this patch set
could help:

https://lore.kernel.org/linux-integrity/20211124044124.998170-1-eric.snowberg@oracle.com/

This would allow users to add new keys in the MOK database,
so that their third-party kernel modules can be verified with
those keys when secure boot is enabled.

Roberto

HUAWEI TECHNOLOGIES Duesseldorf GmbH, HRB 56063
Managing Director: Li Peng, Zhong Ronghua

@robertosassu @mattdm

Thanks for the responses Roberto.

I would be more in favor of having Fedora providing the infrastructure for building third-party kernel modules, although it does not vouch for the content it is building.

I'm not sure what the requirements for this are within Fedora:

  • a separate Fedora key for 3rd party kernel modules, or even keys per 3rd party so that revocation is easier, should it be necessary; or
  • 3rd party modules would only be available in a separate Fedora repository; or
  • they really must be built and signed by a 3rd party, NVIDIA themselves or RPM Fusion.

I'm convinced we need a proper interface for the user to opt-in to the use of 3rd party kernel modules, and I think we already have that in place. But I'm not convinced it's improper for Fedora to sign 3rd party kernel modules, given a proper chain of trust verification.

https://lore.kernel.org/linux-integrity/20211124044124.998170-1-eric.snowberg@oracle.com/
This would allow users to add new keys in the MOK database, so that their third-party kernel modules can be verified with those keys when secure boot is enabled.

Would this enable a means for user-space to inform the kernel of trustworthy kernel module signing keys, without needing to deal with UEFI firmware at all?

The current UI/UX of enrolling Secure Boot keys with mokutil is terrible.

  • it can't be automated e.g. GNOME Initial Setup or GNOME Software;
  • the user must interact with both user-space mokutil, reboot, interact with a EFI program to again indicate they trust the key
  • removal of the key is equally difficult, and may not even be possible via the firmware's setup interface;
  • there's limited space in NVRAM.

If keys can only be trusted by the user indicating trust with an EFI program in the pre-boot environment, that's onerous. But if revoking trust follows the same UI/UX paradigm, that's untenable. If the idea is there's a higher risk of 3rd party kernel modules being compromised in such a way that they need to be urgently disabled, we need a way to automate key revocation.

@robertosassu @mattdm

Thanks for the responses Roberto.

I would be more in favor of having Fedora providing the infrastructure for building third-party kernel modules, although it does not vouch for the content it is building.

I'm not sure what the requirements for this are within Fedora:

  • a separate Fedora key for 3rd party kernel modules, or even keys per 3rd party so that revocation is easier, should it be necessary; or
  • 3rd party modules would only be available in a separate Fedora repository; or
  • they really must be built and signed by a 3rd party, NVIDIA themselves or RPM Fusion.

Likely all these requirements are needed. Fedora would probably not distribute third-party kernel modules from their main repositories, but could create ad-hoc repositories. A requirement for distributing third-party kernel modules is that the sources kernel modules are built from must be signed. The key to verify those kernel modules could be embedded in the kernel, but disabled by default, and enabled upon user request.

I'm convinced we need a proper interface for the user to opt-in to the use of 3rd party kernel modules, and I think we already have that in place. But I'm not convinced it's improper for Fedora to sign 3rd party kernel modules, given a proper chain of trust verification.

https://lore.kernel.org/linux-integrity/20211124044124.998170-1-eric.snowberg@oracle.com/
This would allow users to add new keys in the MOK database, so that their third-party kernel modules can be verified with those keys when secure boot is enabled.

Would this enable a means for user-space to inform the kernel of trustworthy kernel module signing keys, without needing to deal with UEFI firmware at all?

I'm not too familiar with this patch set. It would be better to ask the author. Anyway, it is a solution if Fedora could not provide third-party kernel modules.

The current UI/UX of enrolling Secure Boot keys with mokutil is terrible.

  • it can't be automated e.g. GNOME Initial Setup or GNOME Software;
  • the user must interact with both user-space mokutil, reboot, interact with a EFI program to again indicate they trust the key
  • removal of the key is equally difficult, and may not even be possible via the firmware's setup interface;
  • there's limited space in NVRAM.

If keys can only be trusted by the user indicating trust with an EFI program in the pre-boot environment, that's onerous. But if revoking trust follows the same UI/UX paradigm, that's untenable. If the idea is there's a higher risk of 3rd party kernel modules being compromised in such a way that they need to be urgently disabled, we need a way to automate key revocation.

There is a blacklist keyring that can be used for this purpose. However, I didn't try it.

So I've been pinging relevant Red Hatters internally about this, but so far I've only heard back from Justin, who was already participating here in this issue. So this approach is stalled.

At this point, I think the Working Group should just mandate some solution and move on. Chris has proposed just signing NVIDIA's driver, and that sounds perfectly good to me. Common sense would indicate that this doesn't mean Fedora necessarily trusts the proprietary code, just that Fedora recognizes that it's mandatory for our users. Let's see how debate on devel@ plays out.

There may be an alternative to "just signing the driver" which I think a lot of people would be opposed to, that may be viable in the F-37 timeframe which is actually a side effect of the IMA work the Edge team has been doing. Will know more about the complete viability of this soon(ish).

So I've been pinging relevant Red Hatters internally about this, but so far I've only heard back from Justin, who was already participating here in this issue. So this approach is stalled.

At this point, I think the Working Group should just mandate some solution and move on. Chris has proposed just signing NVIDIA's driver, and that sounds perfectly good to me. Common sense would indicate that this doesn't mean Fedora necessarily trusts the proprietary code, just that Fedora recognizes that it's mandatory for our users. Let's see how debate on devel@ plays out.

With what key? Surely you don't propose we use the Fedora key to sign a binary blob that operates in kernel space from a company who was just hacked last month?

So I've been pinging relevant Red Hatters internally about this, but so far I've only heard back from Justin, who was already participating here in this issue. So this approach is stalled.

At this point, I think the Working Group should just mandate some solution and move on. Chris has proposed just signing NVIDIA's driver, and that sounds perfectly good to me. Common sense would indicate that this doesn't mean Fedora necessarily trusts the proprietary code, just that Fedora recognizes that it's mandatory for our users. Let's see how debate on devel@ plays out.

With what key? Surely you don't propose we use the Fedora key to sign a binary blob that operates in kernel space from a company who was just hacked last month?

Of course not, we'd make a new key and have the kernel trust that. That way we can revoke just that key in the event something has gone wrong.

Of course not, we'd make a new key and have the kernel trust that. That way we can revoke just that key in the event something has gone wrong.

Then the driver is effectively going to be trusted for everyone, even people who don’t want to use the proprietary driver. The key needs to be trusted only on systems of users who want to use the Nvidia driver, otherwise this weakens the protection against rootkits of all Fedora users.

Then the driver is effectively going to be trusted for everyone, even people who don’t want to use the proprietary driver.

The proprietary driver would only be opt-in, not installed by default, nor would it load by default.

The key needs to be trusted only on systems of users who want to use the Nvidia driver, otherwise this weakens the protection against rootkits of all Fedora users.

It seems plausible that a separate key for extending trust (to either a 3rd parties key or a binary) could be in a package that the user needs to explicitly install. That way extending trust would require the user and Fedora to opt-in.

If the malware has root, it can just install any package itself (and if it doesn’t there’s not much reason to worry about rootkits). I don’t see how it’s possible to avoid requiring user interaction with some trusted component in the boot chain to approve a key at the OS level. It’s one more step users need to go through, but if it can be done using the desktop or terminal, it doesn’t provide that much protection, and at least this gives a potential way to support all out-of-tree modules without disabling secure boot, rather than just Nvidia.

ETA: An example scenario I’m envisioning is where some malware on the system wants to use a vulnerability in the Nvidia driver to install a persistent rootkit.

(Call it far-fetched if you will, attacks keep getting more sophisticated and what is far-fetched today will be bread and butter in 5 years.)

I mean, yeah that's true, but we can't do both. We can have a good user experience OR we can distrust NVIDIA. Those are mutually-exclusive options. So what do we choose?

I'm all in favor of attacking NVIDIA, but I don't think it would do any good, and I doubt other distros would follow.

It is not that much of an attack on nvidia to ask users to trust a new key one time. And it lowers the attack surface for the majority of users who do not run any 3rd part modules. People have been following docs on how to install nvidia and get it working for over a decade now, and having a simplified thing where it can be done with a few clicks is still a user experience win. On the flip side, if something did happen and the key was compromised and used, that would be a major user experience loss of trust for all of the people not running these modules.

If it was at the kernel-level and not firmware level, then it'd be fine. Then it's also possible to automate for mass deployment without having to deal with firmware difficulties.

If it was at the kernel-level and not firmware level, then it'd be fine. Then it's also possible to automate for mass deployment without having to deal with firmware difficulties.

Any process that doesn't require physical access can be automated. If it can be automated, it can be exploited.

If it was at the kernel-level and not firmware level, then it'd be fine. Then it's also possible to automate for mass deployment without having to deal with firmware difficulties.

Any process that doesn't require physical access can be automated. If it can be automated, it can be exploited.

And? If it's not in the firmware, that's probably fine. This is something people reasonably expect now with software packaged and shipped via RPMs.

Related, it looks like there's a kernel-level machine keyring introduced in Linux 5.18. Perhaps this can be extended to be useful for our purposes?

It is not that much of an attack on nvidia to ask users to trust a new key one time.

Yes, it really is. That is way too hard for our users, not at all a reasonable ask. No matter how easy it might seem for you, please assume that 95% of our users will give up on Fedora altogether rather than attempt this.

In reality, everyone just turns off secure boot. I've never heard of anyone importing keys or signing drivers ever....

ETA: An example scenario I’m envisioning is where some malware on the system wants to use a vulnerability in the Nvidia driver to install a persistent rootkit.

So long as Secure Boot is enabled, SPI flash is write protected. There's a good chance of getting rid of the malware, if you detect it. But if SB is disabled, SPI flash can be written to. There's examples of lie in wait malware, waiting for exactly this, in order to drop a truly persistent payload into firmware. Not even replacing drives will get rid of it.

Making it so onerous for users that they resort to disabling SB, which is overwhelmingly what they are doing, under auspices of security while ignoring the insecure reality of what we've incentivized - is just folly. For all our security worry, we've in effect created a class of users who are less secure than others, and the line is drawn on hardware. For sure NVIDIA properly gets a heap of blame for their generally uncooperative stance, but they aren't the ones who've directly created the perverse incentive.

We should innovate an alternative. We can't remove all risk. We can manage the risk, by reasonably assuming at some point such a 3rd party binary becomes compromised and will have to be untrusted. And making a plan and the design to handle that accordingly.

Microsoft is already extending trust to NVIDIA for this same scenario, and with a comparatively enormous footprint. What we're doing doesn't really seem as much about security to me, than it is about punishing hapless users for their hardware choices. The position of meeting hostile with hostility, and hoping users make better choices hasn't really had an effect has it? Folks are still buying NVIDIA hardware.

I think we're better off bringing these users into the Fedora and Secure Boot and thus FOSS fold. By reducing the barriers we grow the market and improve the chances someone will work to make this even better.

I would say it makes this issue one completely about the proprietary driver they they may continue to ship for a while. If that is the case, I don't care about making it easier for users. The open source driver does have some obstacles. We need to file for FESCo approval since it is an out of tree module and against policy, but I expect it will not be too long lived. This isn't a ZFS like token code release, it is dual licensed MIT/GPL so it can eventually be upstreamed.

My assumption is that the driver will go in-tree eventually, once it's useful for desktops?

In the meantime, I wonder if we need to solve this anymore at all. It seems clear that the long term path forward is the new driver, even if that will take several years.

My assumption is that the driver will go in-tree eventually, once it's useful for desktops?

In the meantime, I wonder if we need to solve this anymore at all. It seems clear that the long term path forward is the new driver, even if that will take several years.

This won't help people with graphics cards older than Turing generation (GTX 16 / RTX 20), so it still needs a solution.

Metadata Update from @chrismurphy:
- Issue tagged with: meeting

a year ago

My assumption is that the driver will go in-tree eventually, once it's useful for desktops?

In the meantime, I wonder if we need to solve this anymore at all. It seems clear that the long term path forward is the new driver, even if that will take several years.

The nvidia open source driver they have released will not be going in-tree. Don't plan any future where the driver that supports their userspace will be in-tree, it's not a short term plan at all.

Metadata Update from @chrismurphy:
- Issue untagged with: meeting
- Issue set to the milestone: Future Release (was: Fedora 36)

a year ago

Action item: Neal to investigate this ticket again, update proposal

Metadata Update from @catanzaro:
- Issue assigned to ngompa
- Issue tagged with: pending-action

3 months ago

We are WONTFIXing this. Our plan is to focus on improving open source solutions instead (nouveau). It'

It's not ready yet but it's now looking like less work than making NVIDIA driver work with secure boot.

Metadata Update from @catanzaro:
- Issue close_status updated to: Won't fix
- Issue status updated to: Closed (was: Open)

14 days ago

Metadata Update from @catanzaro:
- Issue untagged with: pending-action

14 days ago

Login to comment on this ticket.

Metadata