Can you elaborate on this request ? as it can be solved (or not) through different mechanisms.

Either you just want to have a dedicate keypair that will be used by koji builders (if you want to build kernel through cbs.centos.org) but not the centos distro key, that's eventually possible.
But that means enrolling through mokutil on each machine (like Elrepo is doing with http://elrepo.org/tiki/SecureBootKey)

If you want the kernel to be signed by CentOS Secure BootDistro key, cbs.centos.org infra isn't allowed to reach the signing infra for this, so that would be a different discussion.

So can you just give more info/background ?

Based on previous discussions with @ngompa and @bstinson I think we need to be able to sign with the actual CentOS key for these packages to be useful. This doesn't necessarily need to be automated (at least initially), and we could take a step by step approach:

• come up with a process that allows us to say "please sign this kernel", and a human manually vets the package and signs it
• eventually trust (a subset of) SIG members to vet the package
• eventually automate the whole process

So that will be a long discussion as (reminder) cbs.centos.org is hosted in Community Cage, and without any link with where official signing HSM is (also limited for @redhat.com specific employees).
So yes, maybe a "manual" process is for now the only way to go, but worth discussing also with internal about if that's considered a good/valid use case.

@bex : as RH liaison role, can you start the thread eventually ? waiting on feedback on this one before promising anything that would go against prodsec/infosec rules :)

A manual process for releasing new signed kernels is fine with us, we just need a procedure to have you folks do it.

@bex (or @jwboyer eventually) : any progress at the Red Hat side for this ? (nothing we want to promise through infra and releng that goes against what was told to us initially so waiting on more feedback

I don't have any official updates. I can discuss this with Bex tomorrow.

Note: the following is an opinion, not official communication.

Having just now read this ticket for the first time, I find it unlikely that we would sign SIG kernels with the official CentOS keys, manually or otherwise. It may be eventually possible to enable the SIGs to have their own key to sign things with, but that may require some infrastructure work.

Then we need key delegation, because it's unreasonable that there's no way to establish trust from the base CentOS Project key.

Why is that unreasonable? Trust is exactly the concern.

You're asking the project to sign something they did not produce as part of the official project in a manner that automatically makes it trusted on infrastructure that otherwise trusts content from CentOS. That would at best be very surprising to end users, at worst lead to a compromised system and revocation of CentOS keys as a result.

I agree that we can't expect the project to sign arbitrary artifacts it did not produce. However, CentOS SIGs are part of the CentOS project. We already allow SIGs to sign packages with a per-SIG key, and ship centos-release-* packages for them in Extras to install those keys.

The problem with Secure Boot, is that an equivalent approach (have per-SIG signing keys) will not work in practice, because it's unlikely we'll be able to get all these keys signed by Microsoft. OTOH, expecting users to manually enroll keys will also not work, as the process is cumbersome and hardware-specific (plus, not all platforms support custom keys).

Let me propose a compromise solution here. We make a new "CentOS Secure SIG key", and we work with Microsoft to get it trusted. We then make subkeys for each SIG that inherit from that key. The end result is that SIG images will work for users out of the box, but they will be clearly attributed to a separate key, so in case of any issues will not impact the standing of the main CentOS distribution key.

I agree that we can't expect the project to sign arbitrary artifacts it did not produce. However, CentOS SIGs are part of the CentOS project. We already allow SIGs to sign packages with a per-SIG key, and ship centos-release-* packages for them in Extras to install those keys.

Users still have to accept the keys during dnf transactions to actually enroll the key, correct?

The problem with Secure Boot, is that an equivalent approach (have per-SIG signing keys) will not work in practice, because it's unlikely we'll be able to get all these keys signed by Microsoft. OTOH, expecting users to manually enroll keys will also not work, as the process is cumbersome and hardware-specific (plus, not all platforms support custom keys).

Let me propose a compromise solution here. We make a new "CentOS Secure SIG key", and we work with Microsoft to get it trusted. We then make subkeys for each SIG that inherit from that key. The end result is that SIG images will work for users out of the box, but they will be clearly attributed to a separate key, so in case of any issues will not impact the standing of the main CentOS distribution key.

This does seem like a potential compromise that is worth pursuing.

I agree that we can't expect the project to sign arbitrary artifacts it did not produce. However, CentOS SIGs are part of the CentOS project. We already allow SIGs to sign packages with a per-SIG key, and ship centos-release-* packages for them in Extras to install those keys.

Users still have to accept the keys during dnf transactions to actually enroll the key, correct?

If they run dnf install centos-release-hyperscale interactively, yes. If they install this via config management / embed it into an image / use a container / etc., no.

Let me propose a compromise solution here. We make a new "CentOS Secure SIG key", and we work with Microsoft to get it trusted. We then make subkeys for each SIG that inherit from that key. The end result is that SIG images will work for users out of the box, but they will be clearly attributed to a separate key, so in case of any issues will not impact the standing of the main CentOS distribution key.

This does seem like a potential compromise that is worth pursuing.

Alright, what are the next steps to move forward here? If we go down this path, we'd need to:

• create a new key in a secure way (I assume by storing it in a similar / the same HSM that's used for the current one)
• submit it to Microsoft to get it trusted
• setup delegation so that we can create per-SIG keys
• setup a manual process so that SIGs can request their artifacts to be signed
• even better, hook these keys up with koji so that SIGs can sign stuff automatically

But before we do any of that, we should ensure this is something CentOS and Red Hat are actually ok with. On the CentOS side, I suppose we could run this by the board next month. On the Red Hat side, can you and @bex discuss this and come back here with an update?

I agree that we can't expect the project to sign arbitrary artifacts it did not produce. However, CentOS SIGs are part of the CentOS project. We already allow SIGs to sign packages with a per-SIG key, and ship centos-release-* packages for them in Extras to install those keys.

Users still have to accept the keys during dnf transactions to actually enroll the key, correct?

If they run dnf install centos-release-hyperscale interactively, yes. If they install this via config management / embed it into an image / use a container / etc., no.

Let me propose a compromise solution here. We make a new "CentOS Secure SIG key", and we work with Microsoft to get it trusted. We then make subkeys for each SIG that inherit from that key. The end result is that SIG images will work for users out of the box, but they will be clearly attributed to a separate key, so in case of any issues will not impact the standing of the main CentOS distribution key.

This does seem like a potential compromise that is worth pursuing.

Alright, what are the next steps to move forward here? If we go down this path, we'd need to:

• create a new key in a secure way (I assume by storing it in a similar / the same HSM that's used for the current one)

Any SIG keys we create will need to be on separate hardware from the distro keys.

• submit it to Microsoft to get it trusted

Microsoft, though, is going to require some sort of hardware token for the private keys. If we want to go this route, we need to plan for and purchase the infrastructure for this.

• setup delegation so that we can create per-SIG keys
• setup a manual process so that SIGs can request their artifacts to be signed
• even better, hook these keys up with koji so that SIGs can sign stuff automatically

I missed @arrfab 's ping in this ticket 2 months ago and didn't know anyone was waiting on me. AIUI, we are looking to find a way to have SIGs produce a kernel that is signed for secure boot so it can be reasonably consumed. Additionally we have a proposal we would like to take to MSFT that should allow us to both ensure the primary key for the project and other SIG keys are both safe from a single SIG having a challenge.

Is this all correct?

Yes, that is correct, @bex. Ideally, the CentOS SIG secure boot keys would be set up to support signing the same way signing works for Fedora kernels (specific people are allowed to build on a builder that has the ability to sign the kernels with the production key, and other builders can't access it and fallback to a test key that's not in the Microsoft trust).

I agree that we can't expect the project to sign arbitrary artifacts it did not produce. However, CentOS SIGs are part of the CentOS project. We already allow SIGs to sign packages with a per-SIG key, and ship centos-release-* packages for them in Extras to install those keys.

Users still have to accept the keys during dnf transactions to actually enroll the key, correct?

If they run dnf install centos-release-hyperscale interactively, yes. If they install this via config management / embed it into an image / use a container / etc., no.

Let me propose a compromise solution here. We make a new "CentOS Secure SIG key", and we work with Microsoft to get it trusted. We then make subkeys for each SIG that inherit from that key. The end result is that SIG images will work for users out of the box, but they will be clearly attributed to a separate key, so in case of any issues will not impact the standing of the main CentOS distribution key.

This does seem like a potential compromise that is worth pursuing.

Alright, what are the next steps to move forward here? If we go down this path, we'd need to:

• create a new key in a secure way (I assume by storing it in a similar / the same HSM that's used for the current one)

Any SIG keys we create will need to be on separate hardware from the distro keys.

• submit it to Microsoft to get it trusted

Microsoft, though, is going to require some sort of hardware token for the private keys. If we want to go this route, we need to plan for and purchase the infrastructure for this.

We will also need shim and grub2 for CentOS adjusted to trust both the CentOS main distribution key and the new key used for SIGs.

@bstinson @bex any update on this?

I am not sure I follow what is needed from me here:

1. Getting a SIG key to sign the kernel, not the main distro key, feels like the route everyone has accepted and a technical problem with a technical solution with no business input needed.
2. Getting a new secure boot key that is unrelated to the existing primary project Key to be used by SIGs via a subkey feels like a "go work the process at MSFT" problem with no business input needed.
3. Obtaining HSM hardware to use with the key in 2 (above) feels like a procurement question and I suspect that the Project's budget can afford this without any additional business directed funding required.

Is there a policy issue/change or business request here? I am not seeing it. I'd prefer to not go "ask for permission" on things unless we know we need to violate a policy/request or believe we are bumping up against a guardrail in spirit. Are we doing those things?

The Kmods SIG is - unsurprisingly - interested in this as well. However our use case is a bit different as we only need to sign kernel modules, not a kernel. Hence we could sign with an un-trusted key and have users add this key to their Machine Owner Key (MOK) list. I.e. our requirements are lower in that sense, but a working solution for the Hyperscale SIG will also work for us. However, we do need this process to be automated. We have to rebuild (and sign) each kernel module for every CentOS Stream kernel release. Nothing I'd like to do manually every time.

Independent of which key is being used from a trust perspective it is required that the private key is secured from any unauthorized access. In the best case scenario not even SIG members have direct access to the private key, but only via the build process and no way to ever export it out of the build system.

Is there anything we can do / required from our side to get this moving forward?

@mobrien : as it needs a Spike to see if that's doable and come with a plan, and also then some time to work on this, is that something you'd want to propose as initiative (or mini-initiative) in quarterly planning for CPE ?

FWIW, (so that people can get a status update), based on multiple reminders last week during the CentOS Dojo, I just started internal discussion about it and so hopefully we'll have some options (including maybe a "no, not possible" answer)

removing myself from this and per different discussion another request asking for clarification was sent to the board : https://git.centos.org/centos/board/issue/67
Once we'll have an official answer from the CentOS board and RH Liaison, we'll see if we have to work on this or not

[backlog_refinement]
Closing it as insufficient data, because it's still blocked by https://git.centos.org/centos/board/issue/67 This could be re-open when the blocking ticket is approved.