#9436 Enable repository metadata signing
Closed: Will Not/Can Not fix by kevin. Opened by marmarek.

Please enable repository metadata signing for all currently supported Fedora versions.
This will allow the users to enable metadata signatures check in addition (or instead - of depending on the configuration) to metadata hash from metalink files.
Repository metadata integrity is important to both avoid (potentially compromised) mirrors manipulate metadata (like, selectively hide updates), but also to reduce the risk of exploiting metadata-parsing bugs (like https://bugzilla.redhat.com/show_bug.cgi?id=1868639).

While enabling DNF to verify metadata signatures by default is blocked on improved UX in DNF (https://github.com/rpm-software-management/libdnf/issues/43, https://bugzilla.redhat.com/show_bug.cgi?id=1768206), it is still beneficial to give a power users options to enable it manually (and accept the key on the first run).

Current state of metadata validation and what it takes to attack it (metadata not signed):
- when using metalink, metadata downloaded from a mirror is verified against a hash downloaded over HTTPS from mirrors.fedoraproject.org
- when using baseurl - no extra metadata verification - any metadata served by the selected mirror will be accepted

State with metadata signing enabled:
- default configuration unchanged
- when using metalink, it's possible to additionally enable metadata signature check, to reduce trust in HTTPS (certificate miss-issuance happens from time to time, and is part of some more extreme threat models) and mirrors.fedoraproject.org integrity (MirrorManager instance?)
- when using baseurl with a specific mirror, it is possible to enable metadata integrity verification at all

Related discussion: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/U6AX4LLBURZNU7HRFIFUV6OA7SUYK23Y/


So, I am going to close this WONTFIX for now.

The problem is that it doesn't get us anything in our default configuration and would require real coding work on robosignatory and pungi.

I have no problem asking those projects to support it at a low level RFE... and someday we could switch it on, but I think we have lots of other bigger fish to fry.

I see robosignatory already has a RFE for it: https://pagure.io/robosignatory/issue/14
As does pungi: https://pagure.io/pungi/issue/506

I'll chime in on the devel discussion as well.

Metadata Update from @kevin:
- Issue close_status updated to: Will Not/Can Not fix
- Issue status updated to: Closed (was: Open)

@kevin that should not be true, we already are able to sign repository metadata, because we do this for the openh264 repo.

The openh262 repo is created differently than updates or base composes. It uses a playbook and dist-repos. So, sure, we could switch bodhi and composes to dist-repos... but again, it would be a ton of work.

Sorry, that wasn't right... the openh264 repo used to be completely manually made by releng on each update (and had a release engineer typing in the passphrase for each reposigning). It's now made from a dist-repo and that does NOT sign the repodata nor does it have any support to do so. ;(

@kevin will https://pagure.io/koji/pull-request/2637 allow this to be implemented?

No, thats distrepos in koji.

That does make it so we could do it for the OpenH264 repo though, right?

no, I don't think so, as those are now done via odcs. :)

additionally, we used to do that for the openh264 repo, but we stopped because it was such a horrible user experience.

no, I don't think so, as those are now done via odcs. :)

ODCS?

additionally, we used to do that for the openh264 repo, but we stopped because it was such a horrible user experience.

Signing the metadata doesn’t affect the user experience at all. It’s bugs in repo_gpgcheck=1 handling that do, and some people (including myself) consider the increase in security to be worth the trouble.

@kevin https://github.com/rpm-software-management/rpm/issues/1482#issuecomment-758094027 suggests that that is merely the first step.

ODCS is on-demand-compose-service: https://pagure.io/odcs/

Signing the metadata doesn’t affect the user experience at all. It’s bugs in repo_gpgcheck=1 handling that do, and some people (including myself) consider the increase in security to be worth the trouble.

Until those bugs are fixed, i'll not be in favor of signing any repodata.

What increase in security to you think this provides? If you are using metalinks (default) it doesn't really provide any additional security as far as I am aware.

@kevin https://github.com/rpm-software-management/rpm/issues/1482#issuecomment-758094027 suggests that that is merely the first step.

Thanks for the link. I am not sure how useful that would be for us...

ODCS is on-demand-compose-service: https://pagure.io/odcs/

Got it, thanks!

Signing the metadata doesn’t affect the user experience at all. It’s bugs in repo_gpgcheck=1 handling that do, and some people (including myself) consider the increase in security to be worth the trouble.

Until those bugs are fixed, i'll not be in favor of signing any repodata.

What increase in security to you think this provides? If you are using metalinks (default) it doesn't really provide any additional security as far as I am aware.

Signing the metadata removes the certificate authority system from the trusted computing base. There are hundreds of publically trusted certificate authorities, and a compromise of any one of them would allow an on-path attack on the metalink files. There are other ways to compromise TLS, too: Fedora’s domain registrar could be compromised, as could Fedora’s account with them. Rogue certificates have been issued in the past, and subsequently used for attacks. I also do not believe that DNF checks for certificate revocation (although I could be wrong); even if it did, such checks were broken in libcurl until very recently.

Because of these factors, an attacker obtaining a trusted certificate for mirrors.fedoraproject.org is in the scope of my threat model. It is also within the threat model of QubesOS, which I help develop. QubesOS recently began signing its own repository metadata to protect against the attacks I mentioned above.

The reason I asked for a Tor hidden service for metalinks is similar: they only require trusting that the hidden service address is correct, which is guaranteed by it being part of a signed package.

It is worth noting that using a private Fedora certificate authority, and pinning it on the client, would solve many of these issues. I am not sure how Fedora’s infrastructure authenticates machine-to-machine communication, but if it already has a CA for that purpose, it could also be used for this. That said, it still requires trusting the metalink server, which is directly exposed to the Internet.

@kevin https://github.com/rpm-software-management/rpm/issues/1482#issuecomment-758094027 suggests that that is merely the first step.

Thanks for the link. I am not sure how useful that would be for us...

You’re welcome!

It is worth noting that using a private Fedora certificate authority, and pinning it on the client, would solve many of these issues. I am not sure how Fedora’s infrastructure authenticates machine-to-machine communication, but if it already has a CA for that purpose, it could also be used for this. That said, it still requires trusting the metalink server, which is directly exposed to the Internet.

This is not something we do today, though in theory it could be done using our FreeIPA server as a CA. It's less than ideal, but it is part of how Red Hat Subscription Manager works to authenticate access to RHEL repositories, so it is supported by DNF.

Signing the metadata removes the certificate authority system from the trusted computing base. There are hundreds of publically trusted certificate authorities, and a compromise of any one of them would allow an on-path attack on the metalink files. There are other ways to compromise TLS, too: Fedora’s domain registrar could be compromised, as could Fedora’s account with them. Rogue certificates have been issued in the past, and subsequently used for attacks. I also do not believe that DNF checks for certificate revocation (although I could be wrong); even if it did, such checks were broken in libcurl until very recently.

DNF's certificate revocation support is derived from libcurl through librepo, so it's only as good as libcurl's.

Signing the metadata removes the certificate authority system from the trusted computing base. There are hundreds of publically trusted certificate authorities, and a compromise of any one of them would allow an on-path attack on the metalink files. There are other ways to compromise TLS, too: Fedora’s domain registrar could be compromised, as could Fedora’s account with them. Rogue certificates have been issued in the past, and subsequently used for attacks. I also do not believe that DNF checks for certificate revocation (although I could be wrong); even if it did, such checks were broken in libcurl until very recently.

How do you obtain the gpg key to check the repodata with? Via CA/tls right? so you are just moving the problem from getting the metadata to getting the key via the same means?

Note that fedora enables dnssec for it's dns, and you can tell dnf to check that (but you need to be using a dnssec enabled/aware resolver stack).

Because of these factors, an attacker obtaining a trusted certificate for mirrors.fedoraproject.org is in the scope of my threat model. It is also within the threat model of QubesOS, which I help develop. QubesOS recently began signing its own repository metadata to protect against the attacks I mentioned above.

But then how do you distribute the keys...

Signing the metadata removes the certificate authority system from the trusted computing base. There are hundreds of publically trusted certificate authorities, and a compromise of any one of them would allow an on-path attack on the metalink files. There are other ways to compromise TLS, too: Fedora’s domain registrar could be compromised, as could Fedora’s account with them. Rogue certificates have been issued in the past, and subsequently used for attacks. I also do not believe that DNF checks for certificate revocation (although I could be wrong); even if it did, such checks were broken in libcurl until very recently.

How do you obtain the gpg key to check the repodata with? Via CA/tls right? so you are just moving the problem from getting the metadata to getting the key via the same means?

No, it would be part of a signed RPM package, or even be the same key that signs the packages.

Note that fedora enables dnssec for it's dns, and you can tell dnf to check that (but you need to be using a dnssec enabled/aware resolver stack).

Because of these factors, an attacker obtaining a trusted certificate for mirrors.fedoraproject.org is in the scope of my threat model. It is also within the threat model of QubesOS, which I help develop. QubesOS recently began signing its own repository metadata to protect against the attacks I mentioned above.

But then how do you distribute the keys...

QubesOS uses the same keys to sign its packages and its metadata. Alternatively, the key could be part of a signed RPM package.

No, it would be part of a signed RPM package, or even be the same key that signs the packages.

I seem to not be explaining this well... ;( at the end you are getting this key via a https connection using CA's right?

I mean there's no place people could travel to and get a hand delivered copy of Fedora's gpg keys. In the end, at least once you are having to get that initial key...

No, it would be part of a signed RPM package, or even be the same key that signs the packages.

I seem to not be explaining this well... ;( at the end you are getting this key via a https connection using CA's right?

I mean there's no place people could travel to and get a hand delivered copy of Fedora's gpg keys. In the end, at least once you are having to get that initial key...

That’s true, but ideally one will cross-check it against multiple different sources. The Qubes Master Signing Key can definitely be so cross-checked.

And in any case, one only needs to get the Fedora GPG key once. That is a much smaller window for an attacker to strike.

Robosignatory PR filed: https://pagure.io/robosignatory/pull-request/51!

Metadata