Note that this means that going forward, all CI, pull requests, and auto-rebuilds will fail.

cc: @dcavalca, @daandemeyer, @salimma, @davdunc, @dbrandonjohnson

Can we split it out from systemd so it's a self contained unit? This will mitigate the security implications of signing small discrete units of code rather than the wide systemd stack that includes network and DNS.

This would be useful for when keys need to be revoked at the shim/secure boot level.

I would also prefer this to be split because closing down the ability to fix the systemd package by the broader community would be extremely painful. I hate how those packages work, but there's no way to fix that.

Can we split it out from systemd so it's a self contained unit? This will mitigate the security implications of signing small discrete units of code rather than the wide systemd stack that includes network and DNS.

The only part that is signed is systemd-bootx64.efi. If the build was moved to a separate package, the exact same code would be signed.

This would be useful for when keys need to be revoked at the shim/secure boot level.

sd-boot is built with SBAT information. My understanding is that if a revocation would happen, we'd just specify the appropriate versions in the revocation data. (Something like <251.0, <252.3, <253.2, i.e. effectively killing all but the latest stable releases for a few branches we care about.)

Note that this means that going forward, all CI, pull requests, and auto-rebuilds will fail.

Are you sure? I just did a scratch build (https://koji.fedoraproject.org/koji/taskinfo?taskID=86306226), and it seems to work fine:

+ /usr/libexec/pesign/pesign-rpmbuild-helper x86_64 /usr/bin/pesign /usr/bin/pesign-client --client-token 'OpenSC Card (Fedora Signer)' --client-cert '/CN=Fedora Secure Boot Signer' --cert 'Red Hat Test Certificate' --hostname buildvm-x86-11.iad2.fedoraproject.org --vendor 'Fedora Project' --rhelcert 'Red Hat Test Certificate' --in systemd-bootx64.efi --out systemd-bootx64.efi.signed --sign
+ main x86_64 /usr/bin/pesign /usr/bin/pesign-client --client-token 'OpenSC Card (Fedora Signer)' --client-cert '/CN=Fedora Secure Boot Signer' --cert 'Red Hat Test Certificate' --hostname buildvm-x86-11.iad2.fedoraproject.org --vendor 'Fedora Project' --rhelcert 'Red Hat Test Certificate' --in systemd-bootx64.efi --out systemd-bootx64.efi.signed --sign

I assume that the real thing will just use different texts in those fields…

People keep requesting a separate package. That is certainly possible, but so far I haven't seen any convincing justification.

Note that this means that going forward, all CI, pull requests, and auto-rebuilds will fail.

Are you sure? I just did a scratch build (https://koji.fedoraproject.org/koji/taskinfo?taskID=86306226), and it seems to work fine:
+ /usr/libexec/pesign/pesign-rpmbuild-helper x86_64 /usr/bin/pesign /usr/bin/pesign-client --client-token 'OpenSC Card (Fedora Signer)' --client-cert '/CN=Fedora Secure Boot Signer' --cert 'Red Hat Test Certificate' --hostname buildvm-x86-11.iad2.fedoraproject.org --vendor 'Fedora Project' --rhelcert 'Red Hat Test Certificate' --in systemd-bootx64.efi --out systemd-bootx64.efi.signed --sign + main x86_64 /usr/bin/pesign /usr/bin/pesign-client --client-token 'OpenSC Card (Fedora Signer)' --client-cert '/CN=Fedora Secure Boot Signer' --cert 'Red Hat Test Certificate' --hostname buildvm-x86-11.iad2.fedoraproject.org --vendor 'Fedora Project' --rhelcert 'Red Hat Test Certificate' --in systemd-bootx64.efi --out systemd-bootx64.efi.signed --sign

I assume that the real thing will just use different texts in those fields…

Hmm, actually that might be worse. I didn't realize it succeeds now with dummy certs. That means that we can have repeats of what happened with GRUB recently in Rawhide where everything broke for UEFI+SB people running Rawhide.

I assume that the real thing will just use different texts in those fields…

Hmm, actually that might be worse. I didn't realize it succeeds now with dummy certs. That means that we can have repeats of what happened with GRUB recently in Rawhide where everything broke for UEFI+SB people running Rawhide.

Oh wait, your package isn't assigned to the secureboot channel yet and locked down. That's probably why this still works.

The only part that is signed is systemd-bootx64.efi. If the build was moved to a separate package, the exact same code would be signed.

The only thing signed currently is systemd-bootx64.efi and presumably aarch64 equiv. But what's to say that won't change. As a boot loader it's statically linked and self contained. There's no reason for it to be not independent and hence an independent package.

This would be useful for when keys need to be revoked at the shim/secure boot level.

sd-boot is built with SBAT information. My understanding is that if a revocation would happen, we'd just specify the appropriate versions in the revocation data. (Something like <251.0, <252.3, <253.2, i.e. effectively killing all but the latest stable releases for a few branches we care about.)

But if it was purely something like a key revocation updating all of systemd, journalctl, resolved, networkd and all other periphery is a lot of churn for a SBAT update. For something like IoT it could be a lot of data unnecessarily.

This would be useful for when keys need to be revoked at the shim/secure boot level.

sd-boot is built with SBAT information. My understanding is that if a revocation would happen, we'd just specify the appropriate versions in the revocation data. (Something like <251.0, <252.3, <253.2, i.e. effectively killing all but the latest stable releases for a few branches we care about.)

Almost but not quite - the value has to be a single positive integer. Basically, it tells us which generation of known vulnerabilities are resolved in the current version.

Can we split it out from systemd so it's a self contained unit? This will mitigate the security implications of signing small discrete units of code rather than the wide systemd stack that includes network and DNS.

The only part that is signed is systemd-bootx64.efi. If the build was moved to a separate package, the exact same code would be signed.

This would be useful for when keys need to be revoked at the shim/secure boot level.

sd-boot is built with SBAT information. My understanding is that if a revocation would happen, we'd just specify the appropriate versions in the revocation data. (Something like <251.0, <252.3, <253.2, i.e. effectively killing all but the latest stable releases for a few branches we care about.)

Note that this means that going forward, all CI, pull requests, and auto-rebuilds will fail.

Are you sure? I just did a scratch build (https://koji.fedoraproject.org/koji/taskinfo?taskID=86306226), and it seems to work fine:
+ /usr/libexec/pesign/pesign-rpmbuild-helper x86_64 /usr/bin/pesign /usr/bin/pesign-client --client-token 'OpenSC Card (Fedora Signer)' --client-cert '/CN=Fedora Secure Boot Signer' --cert 'Red Hat Test Certificate' --hostname buildvm-x86-11.iad2.fedoraproject.org --vendor 'Fedora Project' --rhelcert 'Red Hat Test Certificate' --in systemd-bootx64.efi --out systemd-bootx64.efi.signed --sign + main x86_64 /usr/bin/pesign /usr/bin/pesign-client --client-token 'OpenSC Card (Fedora Signer)' --client-cert '/CN=Fedora Secure Boot Signer' --cert 'Red Hat Test Certificate' --hostname buildvm-x86-11.iad2.fedoraproject.org --vendor 'Fedora Project' --rhelcert 'Red Hat Test Certificate' --in systemd-bootx64.efi --out systemd-bootx64.efi.signed --sign

I assume that the real thing will just use different texts in those fields…

People keep requesting a separate package. That is certainly possible, but so far I haven't seen any convincing justification.

This is a "success" that's really a failure. It's signed with the "Red Hat Test Certificate". Microsoft doesn't trust that one, so anything signed with it will not boot on a system with SB enabled. AIUI, anyhow.

I agree with the request to make it a separate package. As you noted, I do build systemd sometimes. I'm not on the SB signers list, I probably shouldn't be, and I don't think you can be added to it just for one package (you're either on it or you're not). I'd like to still be able to build the main systemd package without needing to be added to the SB signers list.

OK, since y'all say that this should be a separate package, I split it out similarly to the Debian setup.
Reviews welcome:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2079784

The package currently uses systemd-udev as the source. Once we split out systemd-boot-unsigned in the main systemd rpm and that is available, it can switch to that. This only difference that this will make is that the unsigned files don't need to be installed on user systems.

I'll edit the ticket description with s/systemd/systemd-boot/.

I'd like to restart this request, but with slightly changed scope.

The idea is to sign sd-boot by a certificate that is not part of the SecureBoot trust path, i.e. is not signed by Fedora Secure Boot CA and is not connected to any root of trust. This will allow users to implement a "trust on first use" policy — they will need to add the certificate to their SecureBoot database (either manually or through some enrolment helper), but afterwards they can enable SecureBoot and rely on having a legitimate Fedora build of the boot loader. Signing in the build infra removes the need for people to maintain a local certificate and keys and resign packages after each update. Having a real (even if untrusted) signature makes development work which includes the boot loader much easier to test.

Signing of systemd-boot by a certificate trusted by Fedora Secure Boot CA is something that I'm leaving for later. I expect that we'll want to do this at some point, but that needs more discussion, and we actually don't need this right now.

Updated request:
Please set up a separate certificate Fedora systemd-boot (or something like that).
Please add systemd-boot [1] builds to the build channel for signing with this signature.
My FAS is zbyszek. We can add other users later on.

[1] https://src.fedoraproject.org/rpms/systemd-boot

/cc @kraxel

So, I have no idea how to create a cert there. I will need @pjones to help me with a howto.

Not that currently that is only x86_64 (no other arches), so you will have to have it sign only on x86_64 for now.

Ok, jumping in here.

Is there any documentation how secure boot signing in Fedora is supposed to work? The current state of affairs is rather confusing.

First, there are two CA certificates. Number one was created on 2012-12-07. That one is broken btw, it is not properly tagged as CA certificate. Number two was created on 2020-07-09, looks like this is supposed to replace the older one.

grub has two signatures. One signed by a apparently shared signer, which in turn is signed by the old CA. The other is signed by a per-package signer, which in turn is signed by the new CA:

# sbverify --list /boot/efi/EFI/fedora/grubx64.efi 
signature 1
image signature issuers:
 - /CN=Fedora Secure Boot CA
image signature certificates:
 - subject: /CN=Fedora Secure Boot Signer
   issuer:  /CN=Fedora Secure Boot CA
signature 2
image signature issuers:
 - /C=US/ST=Massachusetts/L=Cambridge/O=Red Hat, Inc./OU=Fedora Secure Boot CA 20200709/CN=fedoraca
image signature certificates:
 - subject: /C=US/ST=Massachusetts/L=Cambridge/O=Red Hat, Inc./OU=Fedora Secure Boot Signer/OU=bkernel01 grub2/CN=grub2-signer
   issuer:  /C=US/ST=Massachusetts/L=Cambridge/O=Red Hat, Inc./OU=Fedora Secure Boot CA 20200709/CN=fedoraca

kernel looks simliar, except that the second signature is signed by "CN=kernel-signer".

So creating a 'systemd-boot-signer' (signed or not by the CA) should not be much of a problem I think ...

BTW: shim.efi uses the old (and broken) CA certificate for verification. Seems this is not intentional. The file in distgit is named 'fedora-ca-20200709.cer', which suggests it is supposed to contain the new CA certificate. It actually contains the old one though. Reported that a few months back:
https://bugzilla.redhat.com/show_bug.cgi?id=2107982
Seems nobody cares ...

So, I have no idea how to create a cert there. I will need @pjones to help me with a howto.

I have no idea either. @ngompa said that he made some notes that could be used a start for docs.

Not that currently that is only x86_64 (no other arches), so you will have to have it sign only on x86_64 for now.

What is "that"? I certainly want to build and sign all possible arches, i.e. ia32, x64, aa64, and riscv64 once that becomes a thing in Fedora.

First, there are two CA certificates. Number one was created on 2012-12-07. That one is broken btw, it is not properly tagged as CA certificate. Number two was created on 2020-07-09, looks like this is supposed to replace the older one.

I noticed that too, but IIUC, the expiration date and other metadata doesn't matter: once some cert is enrolled, SecureBoot doesn't care about the details of the certificate. (Though of course it'd be nice to have this clean.)

So creating a 'systemd-boot-signer' (signed or not by the CA) should not be much of a problem I think

I'd love to see some chart that shows how the trust is done, starting from Microsoft Corporation UEFI CA 2011 and Microsoft 3rd Party UEFI CA down to our certs here…

The following may be out of date, but 2 years ago secure boot signing required a dedicated hardware system that can talk to the 'key' device which is a dedicated hardware piece. Basically we would need to have a dedicated 2U-6U system for each architecture which only does signing. We only have limited rack space so that system means less other services (aka builders). It also requires special handling as whatever device must be able to be 'locked' inside the case of the dedicated system to meet security controls (ie if it is a USB dongle, the dongle must be physically controlled versus hanging out external to the system in the rack. [This is the reason for 2U need.. we tried a 1U system but found that there was not enough room inside for any of the signing devices we have done. Different architectures have dedicated sizes. ]

We had plans to try and get aarch64 signed in 2019, but various problems came up to get someone to be able to do the needed work. Currently we would probably need a budget for dedicated hardware, purchases of required cryptokey boards, working to see if the hardware can be made to work together, then shipping of said hardware via a secure system to the datacenter. At which point it would require power, rack space (probably removal of existing hardware to do so) and setup.

Not that currently that is only x86_64 (no other arches), so you will have to have it sign only on x86_64 for now.

What is "that"? I certainly want to build and sign all possible arches, i.e. ia32, x64, aa64, and riscv64 once that becomes a thing in Fedora.

kernel + grub are likewise signed on x64 only (in Fedora, RHEL might be different).

First, there are two CA certificates. Number one was created on 2012-12-07. That one is broken btw, it is not properly tagged as CA certificate. Number two was created on 2020-07-09, looks like this is supposed to replace the older one.

I noticed that too, but IIUC, the expiration date and other metadata doesn't matter: once some cert is enrolled, SecureBoot doesn't care about the details of the certificate. (Though of course it'd be nice to have this clean.)

Well, shim.efi doesn't care, but edk2 / ovmf does and ignores the broken cert.
Yes, the expiration date doesn't matter for secure boot.

I'd love to see some chart that shows how the trust is done, starting from Microsoft Corporation UEFI CA 2011 and Microsoft 3rd Party UEFI CA down to our certs here…

The bare firmware trusts the certificates enrolled in the 'db' database (excludding stuff blacklisted in 'dbx'). On physical hardware you usally find the microsoft windows key and the microsoft 3rd party key there. Possibly additional vendor keys. Default configuration for virtual machines is simliar.

shim.efi is signed by microsoft so the firmware loads it.

shim.efi has the distro certificate compiled in. Additionaly it maintains its own databases for allowed (MokList) and excluded (MokListX) certs. These can be updated with 'mokutil'. shim considers the compiled-in cert and certs listed in 'db' or 'MokList' valid for secure boot signing. On a typical fedora install the shim databases are empty and only the compiled-in certificate is used.

shim.efi goes load the bootloader (or other efi binaies when you pass a path on the command line), checks the signature and runs it when it passes. Additionally shim.efi installs a efi protocol for secure boot verification, so bootloaders can just call shim instead of implementing all of that on their own. So shim.efi verifies the fedora signatures on both grub and kernel.

Booting self-compiled and self-signed kernels requires enrolling the cert to MokList using mokutil.
Same goes for systemd-boot cert in case that doesn't get the fedora-ca blessing.

I'm now not entirely sure what to answer here...

As smooge mentioned, we only have x86_64 builders/hardware. They have smart cards in them. We do not have that for other arches (yet). So, it's currently simple not possible to sign them, as the hardware that would do that signing does not exist.

I'm unclear on the cert issue, I suggest you file a shim bug in bugzilla on it?

Adding rharwood here to see if he can help move this along...

CC: @rharwood

Adding rharwood here to see if he can help move this along...

(My pronouns are they/them.)

Just so we all have the same context here:

Our shim is signed by Microsoft on the condition that it can only load grub2, fwupd, and Linux, as is the norm. Providing the capability to use another bootloader in Fedora using that trust chain would violate said agreement.

(My pronouns are they/them.)

My apologies. ;(

Just so we all have the same context here:

Our shim is signed by Microsoft on the condition that it can only load grub2, fwupd, and Linux, as is the norm. Providing the capability to use another bootloader in Fedora using that trust chain would violate said agreement.

ok, thats good to know. I'm open to ideas on how to implement the desired self signed / non ms cert signing. I was thinking we could just enroll it next to the official one and reuse the same infrastructure. I'm open to other ideas how to implement this if thats not acceptable.

The bare firmware trusts the certificates enrolled [...]
shim.efi is signed by microsoft [...]
shim.efi goes load the bootloader (or other efi binaies [...]
So shim.efi verifies the fedora signatures on both grub and kernel.

Thanks. This is a very useful summary. I'm still missing some details near the bottom: which certificate authority is used, e.g. they certs available for inspection somewhere online, or only through extraction from the binaries. What is the expected lifetime and use of the cert authority.

we only have x86_64 builders/hardware.

Mind blown! I wanted to check this for myself, so I downloaded the packages described as "This package contains the version signed by the UEFI signing service.", and indeed, only BOOTX64.EFI is signed at all, but BOOTIA32.EFI and BOOTAA64.EFI are not signed even by the Fedora cert.

I get the hardware/budget limitations, but I'm surprised that this is not prominently documented. This is not something that users should discover for themselves.
Also, why is BOOTIA32.EFI not signed? It is built on the x86_64 builder.

For systemd-boot, the signed package build doesn't do any architecture-specific operations. It just copies files from an earlier archful build of systemd. Thus, it would make no difference if this was done done on a builder with different arch. I think this could be done even without any changes to the infra, by introducing one additional step in the flow: systemd.src.rpm → systemd-boot-unsigned.<arch>.rpm used for systemd-boot.src.rpm → systemd-boot-<arch>.noarch which is where was supposed to happen. This could be changed to systemd.src.rpm → systemd-boot-unsigned.<arch>.rpm → systemd-boot-unsigned-merged.noarch.rpm → systemd-boot-<arch>.noarch, i.e. first gather the archful builds into an unsigned noarch package, and then do the signing in a final step with ExclusiveArch:x86_64. When scheduled via a chain-build, it wouldn't even be inconvenient. (An alternative would be to somehow cause koji to schedule the aarch64 build on the x86_64 builder, but that seems much more complicated and not worth the effort if the issue can be sidestepped by packaging shenanigans.)

Our shim is signed by Microsoft on the condition that it can only load grub2, fwupd, and Linux, as is the norm. Providing the capability to use another bootloader in Fedora using that trust chain would violate said agreement.

Quotation (desperately) needed! I have heard this statement before, but without any substantiation. Please provide the document that specifies the terms of that agreement so that we can figure out how to satisfy the requirements.

Anyway, right now I'm specifically asking for a signature that is NOT trusted by shim, but would still allow users to enroll some authority (into Mok?) and rely on booting authentic Fedora sd-boot builds. See https://pagure.io/releng/issue/10765#comment-832493 above.

Quotation (desperately) needed! I have heard this statement before, but without any substantiation. Please provide the document that specifies the terms of that agreement so that we can figure out how to satisfy the requirements.

This is disingenuous - you have been present when we've talked about how this works before. But I can can state it again since it hasn't changed.

There's a review process that happens at https://github.com/rhboot/shim-review . Submitters declare capabilities of their shim, and reviewers (including, eventually, Microsoft) will check them. There is no requirements list, and you're not being excluded by some hidden process.

For what it's worth, this is why I haven't bothered to request rEFInd-signed, because I know it'll be flat-out rejected.

Quotation (desperately) needed! I have heard this statement before, but without any substantiation. Please provide the document that specifies the terms of that agreement so that we can figure out how to satisfy the requirements.

There's a review process that happens at https://github.com/rhboot/shim-review

I'm aware of this link. It asks a bunch of questions, e.g. all the ones about grub2 are not relevant to sd-boot, and the rest would apply to sd-boot distributed by Fedora exactly the same way as to grub2 (contact info, kernel version and patches).

Looking at https://github.com/rhboot/shim-review/issues/160, the only part that would change would be

- We also have fwupd, which will have similar .sbat provisions to grub2.
+ We also have fwupd and sd-boot, which have similar .sbat provisions to grub2.

Is this all?

[deleted]

I'll just leave the link to signing SUSEs application where the following was mentioned:

We should not be approving this shim until the systemd-boot questions have been merged which will not be happening this year, as so far we do not accept shims that trust systemd-boot.

The questions referenced here can be seen in the following PR: https://github.com/rhboot/shim-review/pull/357

https://github.com/rhboot/shim-review/pull/357 has been merged and we have an official shim procedure for signing shim with support for sd-boot:
https://github.com/rhboot/shim-review/blob/main/docs/reviewer-guidelines.md#systemd-boot

What can we do to move this forward?

Following the videoconf converstation, I filed a bug against shim to include systemd-boot in the review protocol:
https://bugzilla.redhat.com/show_bug.cgi?id=2268695

I know, this conversation has been had in other channels, but maybe its time to actually just sign systemd-boot with a new fedora-systemd-boot cert that is created, rather than trying to wedge this into a shim->systemd-boot path.

Doing that doesn't preclude some future, where its signed with the main fedora keys as well. But in the meantime, systemd-boot users can then enroll the resulting keychains/etc in their firmware and boot without shim. It does create a bit of additional complexity, because the firmware needs to have both the fedora and fedora-systemd-boot key enrolled but that should be fairly trivial vs where we are today.

Which frankly IMHO is a far better solution than this MS->shim thing we have going on now anyway. I always though shim was one of those stopgaps until all the OEM's could be convinced to add a few distro keys anyway, but MS has successfully created a two tier (windows vs everyone else) signing system in the meantime and we are still dependent on their graces.

maybe its time to actually just sign systemd-boot with a new fedora-systemd-boot cert that is created

Yes, that is the plan right now. (We've been back and forth on this, and that's the latest intent.)
For this to happen, we need to deploy a new version of sigul in the Fedora infrastructure. This is really close to happening, it might happen before F40 Final Freeze. If not, then probably soon after. I think that's a really nice solution for sd-boot, because of all the reasons you mention. After this is done and we get some feedback from users, we can consider the question of trusting it from shim again.

Yeah, I was hoping to do it yesterday and then today, but a big security thing came up... ;(

I'm going to try on monday. Thats not much time before freeze, but if it doesn't go smoothly I can just back it out.