Note that this means that going forward, all CI, pull requests, and auto-rebuilds will fail.

cc: @dcavalca, @daandemeyer, @salimma, @davdunc, @dbrandonjohnson

Can we split it out from systemd so it's a self contained unit? This will mitigate the security implications of signing small discrete units of code rather than the wide systemd stack that includes network and DNS.

This would be useful for when keys need to be revoked at the shim/secure boot level.

I would also prefer this to be split because closing down the ability to fix the systemd package by the broader community would be extremely painful. I hate how those packages work, but there's no way to fix that.

Can we split it out from systemd so it's a self contained unit? This will mitigate the security implications of signing small discrete units of code rather than the wide systemd stack that includes network and DNS.

The only part that is signed is systemd-bootx64.efi. If the build was moved to a separate package, the exact same code would be signed.

This would be useful for when keys need to be revoked at the shim/secure boot level.

sd-boot is built with SBAT information. My understanding is that if a revocation would happen, we'd just specify the appropriate versions in the revocation data. (Something like <251.0, <252.3, <253.2, i.e. effectively killing all but the latest stable releases for a few branches we care about.)

Note that this means that going forward, all CI, pull requests, and auto-rebuilds will fail.

Are you sure? I just did a scratch build (https://koji.fedoraproject.org/koji/taskinfo?taskID=86306226), and it seems to work fine:

+ /usr/libexec/pesign/pesign-rpmbuild-helper x86_64 /usr/bin/pesign /usr/bin/pesign-client --client-token 'OpenSC Card (Fedora Signer)' --client-cert '/CN=Fedora Secure Boot Signer' --cert 'Red Hat Test Certificate' --hostname buildvm-x86-11.iad2.fedoraproject.org --vendor 'Fedora Project' --rhelcert 'Red Hat Test Certificate' --in systemd-bootx64.efi --out systemd-bootx64.efi.signed --sign
+ main x86_64 /usr/bin/pesign /usr/bin/pesign-client --client-token 'OpenSC Card (Fedora Signer)' --client-cert '/CN=Fedora Secure Boot Signer' --cert 'Red Hat Test Certificate' --hostname buildvm-x86-11.iad2.fedoraproject.org --vendor 'Fedora Project' --rhelcert 'Red Hat Test Certificate' --in systemd-bootx64.efi --out systemd-bootx64.efi.signed --sign

I assume that the real thing will just use different texts in those fields…

People keep requesting a separate package. That is certainly possible, but so far I haven't seen any convincing justification.

Note that this means that going forward, all CI, pull requests, and auto-rebuilds will fail.

Are you sure? I just did a scratch build (https://koji.fedoraproject.org/koji/taskinfo?taskID=86306226), and it seems to work fine:
+ /usr/libexec/pesign/pesign-rpmbuild-helper x86_64 /usr/bin/pesign /usr/bin/pesign-client --client-token 'OpenSC Card (Fedora Signer)' --client-cert '/CN=Fedora Secure Boot Signer' --cert 'Red Hat Test Certificate' --hostname buildvm-x86-11.iad2.fedoraproject.org --vendor 'Fedora Project' --rhelcert 'Red Hat Test Certificate' --in systemd-bootx64.efi --out systemd-bootx64.efi.signed --sign + main x86_64 /usr/bin/pesign /usr/bin/pesign-client --client-token 'OpenSC Card (Fedora Signer)' --client-cert '/CN=Fedora Secure Boot Signer' --cert 'Red Hat Test Certificate' --hostname buildvm-x86-11.iad2.fedoraproject.org --vendor 'Fedora Project' --rhelcert 'Red Hat Test Certificate' --in systemd-bootx64.efi --out systemd-bootx64.efi.signed --sign

I assume that the real thing will just use different texts in those fields…

Hmm, actually that might be worse. I didn't realize it succeeds now with dummy certs. That means that we can have repeats of what happened with GRUB recently in Rawhide where everything broke for UEFI+SB people running Rawhide.

I assume that the real thing will just use different texts in those fields…

Hmm, actually that might be worse. I didn't realize it succeeds now with dummy certs. That means that we can have repeats of what happened with GRUB recently in Rawhide where everything broke for UEFI+SB people running Rawhide.

Oh wait, your package isn't assigned to the secureboot channel yet and locked down. That's probably why this still works.

The only part that is signed is systemd-bootx64.efi. If the build was moved to a separate package, the exact same code would be signed.

The only thing signed currently is systemd-bootx64.efi and presumably aarch64 equiv. But what's to say that won't change. As a boot loader it's statically linked and self contained. There's no reason for it to be not independent and hence an independent package.

This would be useful for when keys need to be revoked at the shim/secure boot level.

sd-boot is built with SBAT information. My understanding is that if a revocation would happen, we'd just specify the appropriate versions in the revocation data. (Something like <251.0, <252.3, <253.2, i.e. effectively killing all but the latest stable releases for a few branches we care about.)

But if it was purely something like a key revocation updating all of systemd, journalctl, resolved, networkd and all other periphery is a lot of churn for a SBAT update. For something like IoT it could be a lot of data unnecessarily.

This would be useful for when keys need to be revoked at the shim/secure boot level.

sd-boot is built with SBAT information. My understanding is that if a revocation would happen, we'd just specify the appropriate versions in the revocation data. (Something like <251.0, <252.3, <253.2, i.e. effectively killing all but the latest stable releases for a few branches we care about.)

Almost but not quite - the value has to be a single positive integer. Basically, it tells us which generation of known vulnerabilities are resolved in the current version.

Can we split it out from systemd so it's a self contained unit? This will mitigate the security implications of signing small discrete units of code rather than the wide systemd stack that includes network and DNS.

The only part that is signed is systemd-bootx64.efi. If the build was moved to a separate package, the exact same code would be signed.

This would be useful for when keys need to be revoked at the shim/secure boot level.

sd-boot is built with SBAT information. My understanding is that if a revocation would happen, we'd just specify the appropriate versions in the revocation data. (Something like <251.0, <252.3, <253.2, i.e. effectively killing all but the latest stable releases for a few branches we care about.)

Note that this means that going forward, all CI, pull requests, and auto-rebuilds will fail.

Are you sure? I just did a scratch build (https://koji.fedoraproject.org/koji/taskinfo?taskID=86306226), and it seems to work fine:
+ /usr/libexec/pesign/pesign-rpmbuild-helper x86_64 /usr/bin/pesign /usr/bin/pesign-client --client-token 'OpenSC Card (Fedora Signer)' --client-cert '/CN=Fedora Secure Boot Signer' --cert 'Red Hat Test Certificate' --hostname buildvm-x86-11.iad2.fedoraproject.org --vendor 'Fedora Project' --rhelcert 'Red Hat Test Certificate' --in systemd-bootx64.efi --out systemd-bootx64.efi.signed --sign + main x86_64 /usr/bin/pesign /usr/bin/pesign-client --client-token 'OpenSC Card (Fedora Signer)' --client-cert '/CN=Fedora Secure Boot Signer' --cert 'Red Hat Test Certificate' --hostname buildvm-x86-11.iad2.fedoraproject.org --vendor 'Fedora Project' --rhelcert 'Red Hat Test Certificate' --in systemd-bootx64.efi --out systemd-bootx64.efi.signed --sign

I assume that the real thing will just use different texts in those fields…

People keep requesting a separate package. That is certainly possible, but so far I haven't seen any convincing justification.

This is a "success" that's really a failure. It's signed with the "Red Hat Test Certificate". Microsoft doesn't trust that one, so anything signed with it will not boot on a system with SB enabled. AIUI, anyhow.

I agree with the request to make it a separate package. As you noted, I do build systemd sometimes. I'm not on the SB signers list, I probably shouldn't be, and I don't think you can be added to it just for one package (you're either on it or you're not). I'd like to still be able to build the main systemd package without needing to be added to the SB signers list.

OK, since y'all say that this should be a separate package, I split it out similarly to the Debian setup.
Reviews welcome:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=2079784

The package currently uses systemd-udev as the source. Once we split out systemd-boot-unsigned in the main systemd rpm and that is available, it can switch to that. This only difference that this will make is that the unsigned files don't need to be installed on user systems.

I'll edit the ticket description with s/systemd/systemd-boot/.

I'd like to restart this request, but with slightly changed scope.

The idea is to sign sd-boot by a certificate that is not part of the SecureBoot trust path, i.e. is not signed by Fedora Secure Boot CA and is not connected to any root of trust. This will allow users to implement a "trust on first use" policy — they will need to add the certificate to their SecureBoot database (either manually or through some enrolment helper), but afterwards they can enable SecureBoot and rely on having a legitimate Fedora build of the boot loader. Signing in the build infra removes the need for people to maintain a local certificate and keys and resign packages after each update. Having a real (even if untrusted) signature makes development work which includes the boot loader much easier to test.

Signing of systemd-boot by a certificate trusted by Fedora Secure Boot CA is something that I'm leaving for later. I expect that we'll want to do this at some point, but that needs more discussion, and we actually don't need this right now.

Updated request:
Please set up a separate certificate Fedora systemd-boot (or something like that).
Please add systemd-boot [1] builds to the build channel for signing with this signature.
My FAS is zbyszek. We can add other users later on.

[1] https://src.fedoraproject.org/rpms/systemd-boot

/cc @kraxel

So, I have no idea how to create a cert there. I will need @pjones to help me with a howto.

Not that currently that is only x86_64 (no other arches), so you will have to have it sign only on x86_64 for now.

Ok, jumping in here.

Is there any documentation how secure boot signing in Fedora is supposed to work? The current state of affairs is rather confusing.

First, there are two CA certificates. Number one was created on 2012-12-07. That one is broken btw, it is not properly tagged as CA certificate. Number two was created on 2020-07-09, looks like this is supposed to replace the older one.

grub has two signatures. One signed by a apparently shared signer, which in turn is signed by the old CA. The other is signed by a per-package signer, which in turn is signed by the new CA:

# sbverify --list /boot/efi/EFI/fedora/grubx64.efi 
signature 1
image signature issuers:
 - /CN=Fedora Secure Boot CA
image signature certificates:
 - subject: /CN=Fedora Secure Boot Signer
   issuer:  /CN=Fedora Secure Boot CA
signature 2
image signature issuers:
 - /C=US/ST=Massachusetts/L=Cambridge/O=Red Hat, Inc./OU=Fedora Secure Boot CA 20200709/CN=fedoraca
image signature certificates:
 - subject: /C=US/ST=Massachusetts/L=Cambridge/O=Red Hat, Inc./OU=Fedora Secure Boot Signer/OU=bkernel01 grub2/CN=grub2-signer
   issuer:  /C=US/ST=Massachusetts/L=Cambridge/O=Red Hat, Inc./OU=Fedora Secure Boot CA 20200709/CN=fedoraca

kernel looks simliar, except that the second signature is signed by "CN=kernel-signer".

So creating a 'systemd-boot-signer' (signed or not by the CA) should not be much of a problem I think ...

BTW: shim.efi uses the old (and broken) CA certificate for verification. Seems this is not intentional. The file in distgit is named 'fedora-ca-20200709.cer', which suggests it is supposed to contain the new CA certificate. It actually contains the old one though. Reported that a few months back:
https://bugzilla.redhat.com/show_bug.cgi?id=2107982
Seems nobody cares ...

So, I have no idea how to create a cert there. I will need @pjones to help me with a howto.

I have no idea either. @ngompa said that he made some notes that could be used a start for docs.

Not that currently that is only x86_64 (no other arches), so you will have to have it sign only on x86_64 for now.

What is "that"? I certainly want to build and sign all possible arches, i.e. ia32, x64, aa64, and riscv64 once that becomes a thing in Fedora.

First, there are two CA certificates. Number one was created on 2012-12-07. That one is broken btw, it is not properly tagged as CA certificate. Number two was created on 2020-07-09, looks like this is supposed to replace the older one.

I noticed that too, but IIUC, the expiration date and other metadata doesn't matter: once some cert is enrolled, SecureBoot doesn't care about the details of the certificate. (Though of course it'd be nice to have this clean.)

So creating a 'systemd-boot-signer' (signed or not by the CA) should not be much of a problem I think

I'd love to see some chart that shows how the trust is done, starting from Microsoft Corporation UEFI CA 2011 and Microsoft 3rd Party UEFI CA down to our certs here…

The following may be out of date, but 2 years ago secure boot signing required a dedicated hardware system that can talk to the 'key' device which is a dedicated hardware piece. Basically we would need to have a dedicated 2U-6U system for each architecture which only does signing. We only have limited rack space so that system means less other services (aka builders). It also requires special handling as whatever device must be able to be 'locked' inside the case of the dedicated system to meet security controls (ie if it is a USB dongle, the dongle must be physically controlled versus hanging out external to the system in the rack. [This is the reason for 2U need.. we tried a 1U system but found that there was not enough room inside for any of the signing devices we have done. Different architectures have dedicated sizes. ]

We had plans to try and get aarch64 signed in 2019, but various problems came up to get someone to be able to do the needed work. Currently we would probably need a budget for dedicated hardware, purchases of required cryptokey boards, working to see if the hardware can be made to work together, then shipping of said hardware via a secure system to the datacenter. At which point it would require power, rack space (probably removal of existing hardware to do so) and setup.

Not that currently that is only x86_64 (no other arches), so you will have to have it sign only on x86_64 for now.

What is "that"? I certainly want to build and sign all possible arches, i.e. ia32, x64, aa64, and riscv64 once that becomes a thing in Fedora.

kernel + grub are likewise signed on x64 only (in Fedora, RHEL might be different).

First, there are two CA certificates. Number one was created on 2012-12-07. That one is broken btw, it is not properly tagged as CA certificate. Number two was created on 2020-07-09, looks like this is supposed to replace the older one.

I noticed that too, but IIUC, the expiration date and other metadata doesn't matter: once some cert is enrolled, SecureBoot doesn't care about the details of the certificate. (Though of course it'd be nice to have this clean.)

Well, shim.efi doesn't care, but edk2 / ovmf does and ignores the broken cert.
Yes, the expiration date doesn't matter for secure boot.

I'd love to see some chart that shows how the trust is done, starting from Microsoft Corporation UEFI CA 2011 and Microsoft 3rd Party UEFI CA down to our certs here…

The bare firmware trusts the certificates enrolled in the 'db' database (excludding stuff blacklisted in 'dbx'). On physical hardware you usally find the microsoft windows key and the microsoft 3rd party key there. Possibly additional vendor keys. Default configuration for virtual machines is simliar.

shim.efi is signed by microsoft so the firmware loads it.

shim.efi has the distro certificate compiled in. Additionaly it maintains its own databases for allowed (MokList) and excluded (MokListX) certs. These can be updated with 'mokutil'. shim considers the compiled-in cert and certs listed in 'db' or 'MokList' valid for secure boot signing. On a typical fedora install the shim databases are empty and only the compiled-in certificate is used.

shim.efi goes load the bootloader (or other efi binaies when you pass a path on the command line), checks the signature and runs it when it passes. Additionally shim.efi installs a efi protocol for secure boot verification, so bootloaders can just call shim instead of implementing all of that on their own. So shim.efi verifies the fedora signatures on both grub and kernel.

Booting self-compiled and self-signed kernels requires enrolling the cert to MokList using mokutil.
Same goes for systemd-boot cert in case that doesn't get the fedora-ca blessing.

I'm now not entirely sure what to answer here...

As smooge mentioned, we only have x86_64 builders/hardware. They have smart cards in them. We do not have that for other arches (yet). So, it's currently simple not possible to sign them, as the hardware that would do that signing does not exist.

I'm unclear on the cert issue, I suggest you file a shim bug in bugzilla on it?

Adding rharwood here to see if he can help move this along...

CC: @rharwood

Adding rharwood here to see if he can help move this along...

(My pronouns are they/them.)

Just so we all have the same context here:

Our shim is signed by Microsoft on the condition that it can only load grub2, fwupd, and Linux, as is the norm. Providing the capability to use another bootloader in Fedora using that trust chain would violate said agreement.

(My pronouns are they/them.)

My apologies. ;(

Just so we all have the same context here:

Our shim is signed by Microsoft on the condition that it can only load grub2, fwupd, and Linux, as is the norm. Providing the capability to use another bootloader in Fedora using that trust chain would violate said agreement.

ok, thats good to know. I'm open to ideas on how to implement the desired self signed / non ms cert signing. I was thinking we could just enroll it next to the official one and reuse the same infrastructure. I'm open to other ideas how to implement this if thats not acceptable.

The bare firmware trusts the certificates enrolled [...]
shim.efi is signed by microsoft [...]
shim.efi goes load the bootloader (or other efi binaies [...]
So shim.efi verifies the fedora signatures on both grub and kernel.

Thanks. This is a very useful summary. I'm still missing some details near the bottom: which certificate authority is used, e.g. they certs available for inspection somewhere online, or only through extraction from the binaries. What is the expected lifetime and use of the cert authority.

we only have x86_64 builders/hardware.

Mind blown! I wanted to check this for myself, so I downloaded the packages described as "This package contains the version signed by the UEFI signing service.", and indeed, only BOOTX64.EFI is signed at all, but BOOTIA32.EFI and BOOTAA64.EFI are not signed even by the Fedora cert.

I get the hardware/budget limitations, but I'm surprised that this is not prominently documented. This is not something that users should discover for themselves.
Also, why is BOOTIA32.EFI not signed? It is built on the x86_64 builder.

For systemd-boot, the signed package build doesn't do any architecture-specific operations. It just copies files from an earlier archful build of systemd. Thus, it would make no difference if this was done done on a builder with different arch. I think this could be done even without any changes to the infra, by introducing one additional step in the flow: systemd.src.rpm → systemd-boot-unsigned.<arch>.rpm used for systemd-boot.src.rpm → systemd-boot-<arch>.noarch which is where was supposed to happen. This could be changed to systemd.src.rpm → systemd-boot-unsigned.<arch>.rpm → systemd-boot-unsigned-merged.noarch.rpm → systemd-boot-<arch>.noarch, i.e. first gather the archful builds into an unsigned noarch package, and then do the signing in a final step with ExclusiveArch:x86_64. When scheduled via a chain-build, it wouldn't even be inconvenient. (An alternative would be to somehow cause koji to schedule the aarch64 build on the x86_64 builder, but that seems much more complicated and not worth the effort if the issue can be sidestepped by packaging shenanigans.)

Our shim is signed by Microsoft on the condition that it can only load grub2, fwupd, and Linux, as is the norm. Providing the capability to use another bootloader in Fedora using that trust chain would violate said agreement.

Quotation (desperately) needed! I have heard this statement before, but without any substantiation. Please provide the document that specifies the terms of that agreement so that we can figure out how to satisfy the requirements.

Anyway, right now I'm specifically asking for a signature that is NOT trusted by shim, but would still allow users to enroll some authority (into Mok?) and rely on booting authentic Fedora sd-boot builds. See https://pagure.io/releng/issue/10765#comment-832493 above.

Quotation (desperately) needed! I have heard this statement before, but without any substantiation. Please provide the document that specifies the terms of that agreement so that we can figure out how to satisfy the requirements.

This is disingenuous - you have been present when we've talked about how this works before. But I can can state it again since it hasn't changed.

There's a review process that happens at https://github.com/rhboot/shim-review . Submitters declare capabilities of their shim, and reviewers (including, eventually, Microsoft) will check them. There is no requirements list, and you're not being excluded by some hidden process.

For what it's worth, this is why I haven't bothered to request rEFInd-signed, because I know it'll be flat-out rejected.

Quotation (desperately) needed! I have heard this statement before, but without any substantiation. Please provide the document that specifies the terms of that agreement so that we can figure out how to satisfy the requirements.

There's a review process that happens at https://github.com/rhboot/shim-review

I'm aware of this link. It asks a bunch of questions, e.g. all the ones about grub2 are not relevant to sd-boot, and the rest would apply to sd-boot distributed by Fedora exactly the same way as to grub2 (contact info, kernel version and patches).

Looking at https://github.com/rhboot/shim-review/issues/160, the only part that would change would be

- We also have fwupd, which will have similar .sbat provisions to grub2.
+ We also have fwupd and sd-boot, which have similar .sbat provisions to grub2.

Is this all?

[deleted]

I'll just leave the link to signing SUSEs application where the following was mentioned:

We should not be approving this shim until the systemd-boot questions have been merged which will not be happening this year, as so far we do not accept shims that trust systemd-boot.

The questions referenced here can be seen in the following PR: https://github.com/rhboot/shim-review/pull/357

https://github.com/rhboot/shim-review/pull/357 has been merged and we have an official shim procedure for signing shim with support for sd-boot:
https://github.com/rhboot/shim-review/blob/main/docs/reviewer-guidelines.md#systemd-boot

What can we do to move this forward?

Following the videoconf converstation, I filed a bug against shim to include systemd-boot in the review protocol:
https://bugzilla.redhat.com/show_bug.cgi?id=2268695

I know, this conversation has been had in other channels, but maybe its time to actually just sign systemd-boot with a new fedora-systemd-boot cert that is created, rather than trying to wedge this into a shim->systemd-boot path.

Doing that doesn't preclude some future, where its signed with the main fedora keys as well. But in the meantime, systemd-boot users can then enroll the resulting keychains/etc in their firmware and boot without shim. It does create a bit of additional complexity, because the firmware needs to have both the fedora and fedora-systemd-boot key enrolled but that should be fairly trivial vs where we are today.

Which frankly IMHO is a far better solution than this MS->shim thing we have going on now anyway. I always though shim was one of those stopgaps until all the OEM's could be convinced to add a few distro keys anyway, but MS has successfully created a two tier (windows vs everyone else) signing system in the meantime and we are still dependent on their graces.

maybe its time to actually just sign systemd-boot with a new fedora-systemd-boot cert that is created

Yes, that is the plan right now. (We've been back and forth on this, and that's the latest intent.)
For this to happen, we need to deploy a new version of sigul in the Fedora infrastructure. This is really close to happening, it might happen before F40 Final Freeze. If not, then probably soon after. I think that's a really nice solution for sd-boot, because of all the reasons you mention. After this is done and we get some feedback from users, we can consider the question of trusting it from shim again.

Yeah, I was hoping to do it yesterday and then today, but a big security thing came up... ;(

I'm going to try on monday. Thats not much time before freeze, but if it doesn't go smoothly I can just back it out.

So, xz hit then other stuff, then final freeze... etc.

But... today, I finally managed to roll new sigul out. :)

https://pagure.io/fedora-infrastructure/issue/11505

So, now I need to look at how to connect up the new sigul pesign stuff with our buildsystem, but hopefully thats not going to be too crazy, so I hope the end of this saga is getting near.

Any update?

FYI:
systemd-boot-signed in Azure: https://github.com/microsoft/azurelinux/pull/9521/files
Debian: https://salsa.debian.org/systemd-team/systemd/-/commit/3fa8b4b6bafe399a03d30299bc1c1e2227f0592f

Sadly no. I have pinged again. Will look at alternate ways to move forward.

So, this has been a long road, but hopefully some end is near.

We recently upgraded to sigul-12, which supports pesigning. We want to move signing over to that from the existing x86_64 only smartcards.

The last bit we need to make this work is to figure out how we want to hook sigul into the process and how to migrate to it.

Currently:

packages needing secure boot signing are set in the koji hub policy to go to specific builders. Those builders only do these builds and have smart card hardware in them and pesignd running on them to access those cards. The socket to access pesignd is then exported into the builds mock chroot and it knows if it can see that socket it should call pesign to sign things.

I guess my first thought would be that we could just add code to pesign to call sigul instead of using the smart card, but I am not sure if they want to take this sort of change.

If not, we would need something that looks like pesignd to do this.

The sigul setup just uses sigul command line client and on the server
calls pesign to do the actual signing. It's not got any smart card
interface that I can see.

After that part is figured out, we need to figure out how to migrate over to the new setup. Do we need to export/import from the smart cards? Do we need new certs? Do we sign systemd-boot differently from kernel?

I'm going to also mail this to folks involved (pesign maintainers, etc).

Okay, I've done some digging on how best to wire things up to sigul.

As you say, there are two broad approaches we could take. Either we modify pesign to include a mode that doesn't do any signing itself and simply acts as a sigul client, or we build something that implements the pesign daemon interface and acts as a sigul client.

In both cases I think we'd need to run the sigul client as a subprocess rather than re-implementing the client. However, I lean towards the second approach (leave pesign as-is) for a couple reasons.

Firstly, the interface it offers over the unix sock is quite straightforward and small and I don't think it will be particularly difficult to implement ourselves. Secondly, it seems weird to me to put a bunch of sigul-specific stuff in pesign.

The implementation could be any one of the following (or evolve from one to another):

• A small shim that exposes the pesign daemon interface and ferries requests and responses between the sigul server and the pesign-client via the sigul-client CLI.

• An alternate mode to the sigul client to offer the pesign daemon interface.

• Don't try to emulate the pesign daemon and do our own (which would look similar). I only bring this up since some of the commands offered in the pesign API don't necessarily make sense when bridged to sigul. This would require teaching the packages using the pesign client about a new client.

I'm hesitant to add this directly to the sigul client as I know it's in need of a re-write (dead dependencies and so on) which Patrick had started on. I'm going to play around with some proof of concepts next week, but likely will start with the "little shim that shells out" approach first.

Thoughts/comments/objections welcome, of course.

The implementation could be any one of the following (or evolve from one to another):

A small shim that exposes the pesign daemon interface and ferries requests and responses between the sigul server and the pesign-client via the sigul-client CLI.

An alternate mode to the sigul client to offer the pesign daemon interface.

Don't try to emulate the pesign daemon and do our own (which would look similar). I only bring this up since some of the commands offered in the pesign API don't necessarily make sense when bridged to sigul. This would require teaching the packages using the pesign client about a new client.

So, one thing we need is to keep a way to restrict this to specific builders. So, currently only those specific builders have a pesignd running and expose the socket to koji/mock.
If we move to something thats not a socket it might be more difficult to do this (I mean, we could firewall off things, but then non official builds might have to timeout or error, not sure they could detect that they were supposed to sign with the test key with a timeout?

I'm hesitant to add this directly to the sigul client as I know it's in need of a re-write (dead dependencies and so on) which Patrick had started on. I'm going to play around with some proof of concepts next week, but likely will start with the "little shim that shells out" approach first.

Yeah, it's also worth noting that sigul in the end on the vault... calls pesign to actually do the signing. So conceptually it makes more sense to do this either seperately or in sigul, not in pesign.

Thanks again for working on this!

The implementation could be any one of the following (or evolve from one to another):

A small shim that exposes the pesign daemon interface and ferries requests and responses between the sigul server and the pesign-client via the sigul-client CLI.

An alternate mode to the sigul client to offer the pesign daemon interface.

Don't try to emulate the pesign daemon and do our own (which would look similar). I only bring this up since some of the commands offered in the pesign API don't necessarily make sense when bridged to sigul. This would require teaching the packages using the pesign client about a new client.

So, one thing we need is to keep a way to restrict this to specific builders. So, currently only those specific builders have a pesignd running and expose the socket to koji/mock.
If we move to something thats not a socket it might be more difficult to do this (I mean, we could firewall off things, but then non official builds might have to timeout or error, not sure they could detect that they were supposed to sign with the test key with a timeout?

I had imagined something that was also a unix socket, but didn't support all the same commands and pesignd when I suggested we do our own thing (having a command to kill the daemon from the client doesn't necessarily seem important to our use-case, for example). If managing permissions via a socket is the easiest thing for you that's great because that's easy for me to implement, but if you wish it worked some other way I'm happy to consider that as well!

Yeah, unix socket I think would be easiest. Then we can just use the existing setup on the builder side.

Status update here:

I've been working on a proof-of-concept, I feel good about it and my understanding of pesignd and the client. No real ETA on it, but maybe a couple weeks depending on other things I've got going on.

One thing I thought of this morning which might be a bit of a problem in the not-too-distant future: does this run in a Fedora environment or RHEL?

The sigul client currently depends on python-nss, which got orphaned and is dead upstream. It's available in Fedora 41, but not 42. Since this tool just shells out to the sigul client it'll need to be installed which is going to be a problem soon if the build hosts are Fedora (which they are IIRC).

I was planning on working on sigul after this, but I want to make sure I know what to prioritize and how long I have.

This runs on a Fedora environment, so it is an issue. It'll also be a problem in RHEL 10.

The implementation could be any one of the following (or evolve from one to another):

A small shim that exposes the pesign daemon interface and ferries requests and responses between the sigul server and the pesign-client via the sigul-client CLI.

An alternate mode to the sigul client to offer the pesign daemon interface.

Don't try to emulate the pesign daemon and do our own (which would look similar). I only bring this up since some of the commands offered in the pesign API don't necessarily make sense when bridged to sigul. This would require teaching the packages using the pesign client about a new client.

So, one thing we need is to keep a way to restrict this to specific builders. So, currently only those specific builders have a pesignd running and expose the socket to koji/mock.

I think we should stop restricting this to specific builders, and instead only have specific builders use the "real" certs. That would make builds less goofy.

Status update here:

I've been working on a proof-of-concept, I feel good about it and my understanding of pesignd and the client. No real ETA on it, but maybe a couple weeks depending on other things I've got going on.

One thing I thought of this morning which might be a bit of a problem in the not-too-distant future: does this run in a Fedora environment or RHEL?

Fedora.

The sigul client currently depends on python-nss, which got orphaned and is dead upstream. It's available in Fedora 41, but not 42. Since this tool just shells out to the sigul client it'll need to be installed which is going to be a problem soon if the build hosts are Fedora (which they are IIRC).

I was planning on working on sigul after this, but I want to make sure I know what to prioritize and how long I have.

Yeah, this is indeed the case. So, it should be ok for now, but once f42 is out and we want to move the builders to it, we will be in trouble. ;(
We can of course work around this in various ways, but yeah...

I think we should stop restricting this to specific builders, and instead only have specific builders use the "real" certs. That would make builds less goofy.

I don't think the build has any way to know if it should use a 'real' cert or not aside from if the socket it looks for is there.
Or at least thats how it used to work...

The build should be blind to what cert is being used. That should only be builder-side configuration.

The build should be blind to what cert is being used. That should only be builder-side configuration.

I'm open to designing and implementing a replacement for pesign-client, but I'm starting with a re-implementation of the pesign daemon interface. This is primarily so I understand how things work, but also because it makes it very easy to drop in (and roll back if necessary). I've already got a few ideas on what a new interface would look like, but that's not my focus at the moment.

If you could expand on the experience you'd like, and in what specific instances the current approach is causing problems, either here or (preferably) as an issue at https://github.com/jeremycline/sigul-pesign-bridge/issues I'll certainly try to make things better.