#5372 maintained cert-pinned location for Atomic Host (and other) content
Opened 2 years ago by walters. Modified 4 months ago

Certificate pinning should be considered a new standard baseline for use of https, particularly in situations like operating system content.

I'd like to request a webserver which pulls the same content as https://kojipkgs.fedoraproject.org/atomic/24/

https://kojipkgs-pin.fedoraproject.org ?

There apparently already a "Fedora Project CA" that's used for https://pkgs.fedoraproject.org/cgit/rpms/ - this could be the same cert.


Replying to [ticket:5372 walters]:

Certificate pinning should be considered a new standard baseline for use of https, particularly in situations like operating system content.

Well, we don't have it setup anywhere yet, but I agree it would be good to explore/do so.
My understanding is that not only would the server have to send the correct headers and hashes of certs, but it would also have to have a backup cert. Additionally, the client would need to support things.

I'd like to request a webserver which pulls the same content as https://kojipkgs.fedoraproject.org/atomic/24/

https://kojipkgs-pin.fedoraproject.org ?

Which does what differently?

There apparently already a "Fedora Project CA" that's used for https://pkgs.fedoraproject.org/cgit/rpms/ - this could be the same cert.

That cert is our local CA, and expires in the next few years. Also, we are trying to replace it ( see https://fedorahosted.org/rel-eng/ticket/6112 ) and it has no backup, so I don't think it would work out.

This all probibly deserves a discussion on list of what all we need to do and where to test it out first.

I'm not sure what you mean by headers and hashes. There's conceptually two kinds of cert pinning - the "limited CA root" model and the "hardcoded key" model. Red Hat Enterprise Linux uses the former via Akamai, and that's what I was suggesting. Here you have a CA that's not in ca-certificates which is used to sign keys.

It's pretty simple from the client, e.g.:

$ curl --head https://pkgs.fedoraproject.org/cgit/rpms/
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
More details here: https://curl.haxx.se/docs/sslcerts.html
...
$ curl --head --cacert certs/FedoraProjectCA.pem https://pkgs.fedoraproject.org/cgit/rpms/
HTTP/1.1 200 OK
...

So then to answer:

https://kojipkgs-pin.fedoraproject.org
...
Which does what differently?

That server presents a certificate which is signed by the (let's call it) "content cert".

I'd rank this a high priority request from the Project Atomic side for infra. See also:

One thing I want to add is that while I'd be happy to start with just pubkey-pinning for ostree, since we're rpm-ostree I'd also like to have it for the RPM metadata.

I've opened a discussion thread on the infrastructure list with a list of questions and issues we need to solve before we can do this.

BTW this morning I happened to see:

https://bugzilla.gnome.org/show_bug.cgi?id=772897
and if you follow the link to e.g.
https://wiki.mozilla.org/CA:WoSign_Issues#Issue_L:_Any_Port_.28Jan_-_Apr_2015.29

One can see the rampant problems with the CA ecosystem, and the reason for discussions like this, particularly because the OSTree content has been waiting for GPG signatures for a long time. (But even after we have that, we still want ca pinning for the reasons linked above)

well, this completely dropped off my radar. Sorry about that.

I was kind of waiting for dnf developers to ack the plan before moving forward on it.

Did we still wish to do this? were we still ok with the plan?

Metadata Update from @kevin:
- Issue priority set to: Waiting on Reporter

8 months ago

key pinning is going to be abandoned:
https://groups.google.com/a/chromium.org/forum/#!msg/blink-dev/he9tr7p3rZ8/eNMwKPmUBAAJ

I guess it would be better to look into requiring certificates be published via certificate transparency nowadays.

Well, this key pinning was not use chrome/chrominum or any browser, but rather in ostree and dnf.

I never really heard anything from dnf maintainers if they would support implementing this, thats likely why it fell off my radar. I suppose I should file a bug or issue and see if I can get them to look.
https://bugzilla.redhat.com/show_bug.cgi?id=1599915 filed on librepo.

Well, my librepo bug says it's been "Triaged", but not sure if that means they want to do it or not.

Metadata Update from @kevin:
- Issue priority set to: Waiting on External (was: Waiting on Reporter)

5 months ago

I think the TL;DR on this is pretty simple:

But in the end...this issue has stalled for a long, long time now. And of course Fedora CoreOS is in the process of being created; one thing that Container Linux does today is a very dev-ops approach where people work on "both ends" of the problem. I personally still want to be able to talk to people at conferences about how we've actually read and considered the TUF paper and addressed the problem; if that requires us taking ownership of the build/delivery side then that's probably what we need to do.

I think the TL;DR on this is pretty simple:

It's what Red Hat Enterprise Linux does by default and for good reason

It stops someone who can subvert CA's you trust, after you have installed.

It does not prevent those same people from subverting CA's you trust and tampering with your initial install.

By "Red Hat Enterprise Linux" you mean "Red Hat Enterprise Linux Atomic Host" ?
or is there some support I am unaware of already in dnf / librepo?

The paper "A Look In the Mirror: Attacks on Package Managers" published here https://theupdateframework.github.io/publications.html goes into a lot of useful detail on the problems

Yes, I know it well. I read it the day it came out and we did a number of things right after that to improve. The mirrormanager metalink work came out shortly after that, and I am pretty sure we now prevent any attack they mention.

But in the end...this issue has stalled for a long, long time now. And of course Fedora CoreOS is in the process of being created; one thing that Container Linux does today is a very dev-ops approach where people work on "both ends" of the problem. I personally still want to be able to talk to people at conferences about how we've actually read and considered the TUF paper and addressed the problem; if that requires us taking ownership of the build/delivery side then that's probably what we need to do.

The TUF paper is 10 years old and as far as I can tell we solved all their issues long ago.

They don't mention CA subverting attacks or cert pinning at all that I can see.

You do what you feel is right...

By "Red Hat Enterprise Linux" you mean "Red Hat Enterprise Linux Atomic Host" ?
or is there some support I am unaware of already in dnf / librepo?

This is the default with yum today in (what I call "classic") RHEL7 - try it! Just boot a VM and subscribe it, no local/internal mirrors. You should see these options in /etc/yum.repos.d/redhat.repo:

sslclientcert = /etc/pki/entitlement/8344246816275404700.pem
sslclientkey = /etc/pki/entitlement/8344246816275404700-key.pem
sslcacert = /etc/rhsm/ca/redhat-uep.pem

That's strong mutual authentication between the CDN and client, but for this case we are just trying to do the sslcacert bit to have the client strongly pin to a particular CA rather than all of ca-certs.

This is also implemented in librepo, and Red Hat Enterprise Linux Atomic Host in libostree.

The TUF paper is 10 years old and as far as I can tell we solved all their issues long ago.

How?

They don't mention CA subverting attacks or cert pinning at all that I can see.

The paper does talk about RHEL, it's possible the current situation there is actually a result of the paper.

During our discus-
sion with the Red Hat security team, they realized there
were issues in how SSL was used in the plug-in as well and
began working on a fix for it.

/cc @puiterwijk - am I missing something here?

So, I've read through the issue, and I've skimmed the TUF paper, but I may still be missing a couple pieces here.

In the case of RHEL, or digital ocean, there's a vested (and monetary) interest in controlling the update streams. That's certainly the case with the example you've listed for RHEL. That's less about cert pinning and more about validating entitlements, as /etc/yum.repos.d/redhat.repo doesn't exist until subscription manager is run.

In the case of Fedora, this feels like it would undermine the various mirrors we rely on for providing updates, as we wouldn't exactly be able to require that mirrors.kernel.org use a fedora generated cert.

With silverblue showing promise, I expect rpm-ostree usage will only increase. To effectively do that, we need to make sure we don't take on the requirement of providing every update without the use of the mirror network.

We prevent all those attacks with metalinks.

Currently the default config out of the box on Fedora installs is for dnf / PackageKit to query https://mirrors.fedoraproject.org for a metalink file. This is ssl encrypted and run by us directly (it's the mirrormanager application that generates this metalink). In the metalink is the checksums for the last 3 valid repos, along with a list of mirrors. The client then queries those mirrors in order looking for those checksums. Once it gets a mirror with those checksums it downloads the packages it needs (failing on to others if that mirror does not have the package, etc).

Looking at the attacks from the paper:

  • Arbitrary Package - The attacker provides a package they created in place of a package the user wants to install.

Won't work because we have the repodata checksums and if they add a package the package manager won't see it at all. If they tamper with a existing package the checksum won't match.

  • Replay Attack An attacker replays older versions of correctly signed packages or metadata, causing clients to install an old package with security vulnerabilities

Won't work because older packages won't be in the repodata. You can't change the metadata or it won't match the checksum and the client will move on to the next mirror. (Note that this works for the last 3 repodatas, but we have means to flush that and only provide the most recent if there's a urgent security update being pushed out)

  • Freeze Attack - Similar to a replay attack, a freeze at-tack works by providing metadata that is not current.

Won't work, metadata doesn't match checksums.

  • Extraneous Dependencies The attacker rewrites the package metadata to have additional packages installed alongside a package the user intends to install.

Won't work, metadata does not match checksums.

  • Endless Data - This attack is performed by returning an endless stream of data in response to any download request.

I think this is fixed in librepo, but I should retest to make sure.

Anyhow, we could cert pin for mirrors.fedoraproject.org, but like I said it only helps you from someone subverting the CA system, which you were already vulnerable to on your initial download. Additionally, all the big players are not doing cert pinning anymore, they are moving to certificate transparency. I am not sure what support for that would look like, perhaps @puiterwijk could say.

For ostree right now we are just hard coding our kojipkgs site, but also serving some parts of it from amazon cloudfront. So, we would need to 'pin' kojipkgs and also cloudfront.

So, two further discussion items from me:

  • Is it desireable/practical to add metalink support to ostree? That might make it easier for us to mirror trees out to other sites (although it's tons of smallish files, so perhaps it's not even easy to mirror on our normal mirror network).

  • Is cert pinning the way to go, or should we look at cert transparency?

For certificate transparency support, librepo/dnf would need to check whether the certificate that it should be using for a connection is published in enough trusted certificate transparency logs (https://www.certificate-transparency.org/ contains more info about it). Using it from OpenSSL seems to be pretty easy:

http://www.certificate-transparency.org/certificate-transparency-in-openssl

Additionally we need to check in infra that there are no rogue certificates published via certificate transparency to identify attacks.

We prevent all those attacks with metalinks.

OK, there's a fundamental root of confusion here - the threat scenario here I'm discussing is an actor with control of a CA that's in ca-certificates. I mentioned some examples earlier.

The linkage between this and the TUF paper is it considers http (non-TLS) + rpms signed with GPG - which is fully symmetric with the "compromised TLS" case.

In the case of Fedora, this feels like it would undermine the various mirrors we rely on for providing updates, as we wouldn't exactly be able to require that mirrors.kernel.org use a fedora generated cert.

It's easy to split metadata from content; this proposal is merely that the metadata supports being fetched from a tls-pinned location. After that one can download the set of (sha256, length) pairs from anywhere (including mirrors.kernel.org) - basically the same way that mirrormanager works today.

For certificate transparency support, librepo/dnf would need to check whether the certificate that it should be using for a connection is published in enough trusted certificate transparency logs

It's an interesting idea...offhand my feeling is that certificate transparency is a good solution for the Web in general, but for base operating system updates where a compromise is total (not just of web sites), it's worth going the extra mile to do pinning.

Currently the default config out of the box on Fedora installs is for dnf / PackageKit

BTW, you seem to have omitted rpm-ostree from "Fedora" here. It links to libdnf just like dnf/PK do. Since @mattdm has been pushing hard for it to be "Fedora CoreOS" I'd hope that trickles down into us being "Fedora" too in discussions like this.

It's easy to split metadata from content; this proposal is merely that the metadata supports being fetched from a tls-pinned location. After that one can download the set of (sha256, length) pairs from anywhere (including mirrors.kernel.org) - basically the same way that mirrormanager works today.

I think you need to redefine the proposal then because that was not clear at all, and the ticket has spread over 2 years into a myriad of side comments about different things. Can you summarize down what you are wanting and how you are wanting and what exactly would be served and NOT served by this website. [A good portion of clients to mirrors and kojipkgs are currently Firefox and IE .. but are they supposed to work with the pinned site without complaints or not?]

Focusing this down to fixing one project for a specific timeline will help us because our solutions need to focus on the general case by default. If we are solving a feature required by rpm-ostree then we can work out the changes we need to make with other groups like Red Hat IT etc without having to make sure we aren't breaking yum for EL6 systems and IE9 clients. Otherwise we are going to spend a lot of time dealing with all the whatifs we normally do while all you are wanting is it for one thing.

On the other-hand, if you are wanting this to be a general case solution, you are going to have to deal with every what-if because we are the ones who get every "why did you break my workflow" for 400+ developers and various outside tools who rely on what exists as is to work without complaints.

Login to comment on this ticket.

Metadata