#12041 CVE-2024-3094 Impact on Container Registry Layers
Closed: Fixed with Explanation 17 days ago by kevin. Opened 2 months ago by jsteffan.

  • Describe the issue

Do we need to clean up the automatically published registries that include the affected packages? Is there any additional CVE data that needs to be collected about what layer hashes were impacted?

The currently in-progress compose will publish new layers and tag them as the latest available. This will prevent any further pulls from getting the impacted layers.


Yeah, latest should be fine now.

I am not sure if we should try and do too much here, and I think if we do it will be hard. ;)

There is at least:
Old oci images with the package
Old docker images with the package
* Old ostree commits that someone could still switch to (I think)? for all the ostree variants.

Trying to remove those all could be difficult. The bad package was in rawhide for about 3 weeks or so, so thats a lot of images/layers/commits.
I'm not even sure how possible it is to remove an ostree commit without messing up the repo off hand.

Also, bigger discussion: we don't do this for 'normal' security bugs... I mean you can still grab old images that have bad security issues in them, we don't do anything to delete them... should there be a threshold?

I'd love to hear more opinions as we learn more... thanks for bringing this up.

I think the difference between normal security bugs and this is that this is an attempt of active malware injection into the distribution. And while it was technically broken in 40 (and wasn't actually shipped in images), it is considered functional in Rawhide, and therefore needs to be treated somewhat differently.

Scrubbing our Rawhide images for the past month is probably "good enough".

I at first thought that removing bad images wasn't needed, but in going over how people 'use' ISOs versus containers, I think they are different enough that the way we 'deal' with problems needs to be different.

1) While we have ISOs with code which could be exploited, we usually have updates and 'end-of-life' flags which say that this ISO is no longer supported and should be considered suspect. Containers do not have this.
2) Containers are hardcoded to pull in layers in various ways which make this code at rest still active years as long as the registry allows for it to be pulled.
3) This was code which was an active attack against the ecosystem. Just as much as we are supposed to reinstall a server from scratch if it has been exploited because we can't really tell how deep the attacker goes, in this case we don't know how far down this 'attack' goes and how many moving parts may have been affected. We also don't know what the long term target was.. maybe it was Fedora 40/SuSE Tumbleweed/Debian+Ubuntu on active systems or maybe it was for longer term attacks where snaps and containers which pulled in this later on.

There are automated tools for detecting CVEs in images, but we shouldn't rely on those being effective or even in use everywhere. Those class of tools are to be used for a defense in depth strategy and not as the primary way to prevent exploitable layers from being used. Tools like these will be useful in detecting anywhere these layers have been pulled to (by the end-user), but we have no way to ensure those layers are not propagated further.

I agree this should be treated differently than a normal CVE because of the intentional nature. The current security response might have been expected and now the original intent is still to be put into action. We have to be 100% whereas the author just has to find the one thing we didn't.

Where I have the largest concern is an innocent looking FROM that is hosted at registry.fedoraproject.org (or any well known/trusted registry) in a random Dockerfile out in a popular tutorial or project makes it trivial to propagate these layers with low scrutiny.

I suggest that we scan/identify all releng automatically published registries for the CVE and then:

1) Document each layer that was found to be an issue (registry/image:hash)
2) Delete the offending layer

I spent a little bit of time poking registry.fedoraproject.org and it's not immediately apparent if it's possible to get a list of all old digests/blobs, only those that are tagged. I briefly reviewed https://github.com/opencontainers/distribution-spec and it didn't appear there is a way to list every digest/blob if you don't already know it. So in this case, someone would have needed to have recorded a digest that did have an issue and use that directly.

I suggest that we just prune everything that is not tagged to be sure.

I've verified that all of the current tags for registry.fedoraproject.org/fedora do not contain CVE-2024-3094 using skopeo and grype[1].

set -x; for repo_tag in $(skopeo inspect docker://registry.fedoraproject.org/fedora|jq -r '.RepoTags[]'); do grype registry.fedoraproject.org/fedora:$repo_tag --scope all-layers; done

[1] https://github.com/anchore/grype

I'll need to look next week, but actually I think we don't keep old rawhide containers. We just copy the new one over the old one... so only the latest one is even there.

It's worth noting that the grype scan was likely useless. I built a vulnerable image and it didn't trigger a finding. This is likely because the CNA for CVE-2024-3094 didn't list any vulnerable Fedora packages. That should be considered for a future update.

To check, I manually looked at the SBOMs using syft[2] directly. I also checked dockerhub and quay.

set -x; for repo_tag in $(skopeo inspect docker://registry.fedoraproject.org/fedora|jq -r '.RepoTags[]'); do syft registry.fedoraproject.org/fedora:$repo_tag --scope all-layers|grep -e "^xz.*"; done

The following tags need to be rebuilt or deleted:

registry.fedoraproject.org/fedora:fedora-aarch64
registry.fedoraproject.org/fedora:fedora-ppc64le
registry.fedoraproject.org/fedora:fedora-s390x
registry.fedoraproject.org/fedora:fedora-x86_64
registry.fedoraproject.org/fedora:fedora-minimal-aarch64
registry.fedoraproject.org/fedora:fedora-minimal-ppc64le
registry.fedoraproject.org/fedora:fedora-minimal-s390x
registry.fedoraproject.org/fedora:fedora-minimal-x86_64
registry.fedoraproject.org/fedora:fedora-toolbox-aarch64
registry.fedoraproject.org/fedora:fedora-toolbox-ppc64le
registry.fedoraproject.org/fedora:fedora-toolbox-s390x
registry.fedoraproject.org/fedora:fedora-toolbox-x86_64
quay.io/fedora/fedora:fedora-aarch64
quay.io/fedora/fedora:fedora-ppc64le
quay.io/fedora/fedora:fedora-s390x
quay.io/fedora/fedora:fedora-x86_64
quay.io/fedora/fedora:fedora-minimal-aarch64
quay.io/fedora/fedora:fedora-minimal-ppc64le
quay.io/fedora/fedora:fedora-minimal-s390x
quay.io/fedora/fedora:fedora-minimal-x86_64
quay.io/fedora/fedora:fedora-toolbox-aarch64
quay.io/fedora/fedora:fedora-toolbox-ppc64le
quay.io/fedora/fedora:fedora-toolbox-s390x
quay.io/fedora/fedora:fedora-toolbox-x86_64

[2] https://github.com/anchore/syft

Yeah, so those are from when we changed our sync script, so I think we just stopped syncing them...

but I am not sure why they are there... what does someone expect from a fedora:fedora tag? The latest stable release? The latest development release? Something else?

I think we may just be able to delete them... do we see them in any docs anywhere?

CC: @adamwill

As a note, we will know that we've correctly cleaned up these tags when this pull is no longer possible:

podman pull registry.fedoraproject.org/fedora@sha256:6d5b5fb6efc6807901870b37fded56ed30c0cf05eedd969380558ecee373dc1b

There might be other digests available, this one is from:

skopeo inspect docker://registry.fedoraproject.org/fedora:fedora-x86_64

registry-inspect-fedora-x86_64.txt

Metadata Update from @kevin:
- Issue tagged with: medium-gain, medium-trouble, ops

2 months ago

I really have no idea, honestly. I just tried to fix the script to do what it was supposed to do. If I changed anything it was because I HAVE NO IDEA WHAT I'M DOING dog meme.

sorry, this dropped off my radar. ;(

I think we should delete those tags, and I will do so later today... or get someone else who can to do so.

I've deleted the quay.io ones... will do the registery ones later this afternoon.

ok. Nuked the rest. Can someone recheck and make sure they are all gone?

I checked the SBOMs for everything returned when inspecting docker://registry.fedoraproject.org/fedora and docker://quay.io/fedora/fedora and the vulnerable package is not on any of those layers.

I also checked that the known bad layer fedora@sha256:6d5b5fb6efc6807901870b37fded56ed30c0cf05eedd969380558ecee373dc1b is unable to be pulled.

Awesome. Thank you. ;)

Metadata Update from @kevin:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)

17 days ago

Login to comment on this ticket.

Metadata
Boards 1
Ops Status: Backlog