Would the shipping be part of the FCOS release pipeline ? Then the ask here is really to get token and permissions to push to the fedora organization in quay.io ?

Would you also like to have the images on registry.fedoraproject.org ?

Would the shipping be part of the FCOS release pipeline ? Then the ask here is really to get token and permissions to push to the fedora organization in quay.io ?

That seems like the most obvious path, although it could also be pull based, i.e. the FCOS pipeline pushes to something like quay.io/fedora-coreos-testing/fcos:stable and there's something with credentials that watches for changes and mirrors from there to quay.io/fedora/coreos:stable.

Would you also like to have the images on registry.fedoraproject.org ?

I personally have no strong opinion on that; it seems like it should behave how other containers behave. If most of them are mirrored then let's mirror.

But this issue is about more than just FCOS note, because other things besides FCOS use rpm-ostree and would probably (maybe?) want this.

Would the shipping be part of the FCOS release pipeline ? Then the ask here is really to get token and permissions to push to the fedora organization in quay.io ?

That seems like the most obvious path, although it could also be pull based, i.e. the FCOS pipeline pushes to something like quay.io/fedora-coreos-testing/fcos:stable and there's something with credentials that watches for changes and mirrors from there to quay.io/fedora/coreos:stable.

We have bodhi currently doing something similar for layered images (OSBS builds and push the images to candidate-registry.fp.o and bodhi pushes the images to registry.fp.o when the update goes stable). Would the os container updates go through bodhi ?

Would you also like to have the images on registry.fedoraproject.org ?

I personally have no strong opinion on that; it seems like it should behave how other containers behave. If most of them are mirrored then let's mirror.

But this issue is about more than just FCOS note, because other things besides FCOS use rpm-ostree and would probably (maybe?) want this.

Yes, there has been some discussion to also move registry.fp.o to be hosted on quay.io (https://pagure.io/fedora-infrastructure/issue/10373) so It don't think it is super important to sort this out now.

In general, our compose artifacts should exist on our infrastructure, even if they're mirrored out externally. For all intents and purposes, that means things should be on registry.fp.o and mirrored to quay.io.

@walters As per the bz you said its almost ready, do you need anything from us? Also, what are your plans for building the images? As in like where are you going to build them? In koji?

At the current time I am focusing on driving this from the Fedora CoreOS side, where we have an integrated, container-native build and test flow. On this topic for example, https://github.com/coreos/fedora-coreos-pipeline/pull/457 merged.

I would love to pursue actually moving quay.io/coreos-assembler/fcos:testing-devel to quay.io/fedora/coreos:testing-devel. That seems like something that could be split out as a separate ticket? I think it'd literally just be "get quay.io robot account with permissions to push there from Fedora infra, change FCOS pipeline to use it".

For non-FCOS editions, anyone else can feel free to pick up those. I do want to emphasize though, it is today a first class operation to take an ostree commit and "encapsulate" it in a container image. See: https://coreos.github.io/rpm-ostree/container/#creating-base-images

So either the Koji-ostree integration could gain a configuration option to run that command, output a container image, and push it to a registry, or a separate controller process could run and watch the repo and synchronize the container. (Hmm, I could try to make this easier to do in an idempotent way in our tools)

I think for the container base images, it would be great if koji-ostree integration could gain that feature, that way we can use pungi to generate these images nightly and push it to the registries.

This is what we do currently for rpm based container base images.

I think for the container base images, it would be great if koji-ostree integration could gain that feature, that way we can use pungi to generate these images nightly and push it to the registries.

OCI container images are just layers of tarballs wrapped with JSON. One thing to bear in mind is that this process of "encapsulating" an ostree commit into an OCI container is hence basically "convert ostree commit into tar format and wrap it with JSON". It's not actually a "build" in any real sense (we'd never run gcc, we don't even run any scripts or solve any dependencies). It's just changing one data format into another.

Yes, for editions using koji-ostree to generate ostree commits, it could make sense to encapsulate the container as part of the same execution flow. But, I think it could also make sense to have it be a separate controller.

This is what we do currently for rpm based container base images.

I'd prefer to say e.g. "non-ostree" or "traditional" here. The container images we're generating using rpm-ostree definitely involve RPM. It just doesn't only involve RPM - it also involves ostree. Another way to say this is, these images aren't "ostree based" - ostree isn't the center of the universe. Neither is rpm. It's a hybrid system.

I raised this issue here: https://github.com/fedora-silverblue/issue-tracker/issues/275#issuecomment-1124160730

But it seems to have great overlap with the discussions here. With that being said if there's any way I can help I would be happy to.

It sounds like the first step (if in agreeance) is to get koji-ostree to push containers while it already generates ostree commits

OK folks, we now have ociarchive images built by pungi during the Rawhide composes.

So now we need to figure out how to push those images to quay.io and disable the previous script doing manual mirroring.

I've been looking at https://pagure.io/pungi-fedora/blob/main/f/container-nightly.sh & https://pagure.io/releng/blob/main/f/scripts/sync-ostree-base-containers.sh (we'll have to remove this one) but I don't see where the classic "Docker" container images are pushed in those scripts.

Could someone from releng point me to the script / code that we use to upload the Fedora container image to the registry?

Thanks!

@cverna pointed me to https://pagure.io/releng/blob/main/f/scripts/sync-latest-container-base-image.sh

If I understand correctly, this will re-download the images and push them to the registry. This would be very time, network and disk space consuming for the ostree images which are rather large.

Hey infra folks. I'm a bit lost between all the scripts and which one runs when.

Could we setup a meeting to move this forward? This is the last missing bit for this Change.

Thanks!

So there's three moving parts:

• These little scripts in ansible
• The scripts in pungi-fedora that are called by those
• The sync scripts in releng (sync-latest-container-base-image.sh and sync-ostree-base-containers.sh) that are called by those

The ansible scripts are run on the compose hosts as cron jobs. They run the various 'nightly' scripts from pungi-fedora - nightly.sh, container-nightly.sh, cloud-nightly.sh - which trigger the composes. After the composes are done, those scripts call one or both of the two 'sync' scripts from the releng repo. sync-latest-container-base-image.sh is capable of publishing the generic container image, minimal generic container image, and toolbox container image to quay.io and registry.fp.o. sync-ostree-base-containers.sh is capable of converting silverblue, kinoite and sericea ostrees to container images and publishing them to quay.io and registry.fp.o.

We probably want to ditch sync-ostree-base-containers.sh and just extend sync-latest-container-base-image.sh to cover the atomic desktop OCI container images too, since it's really the same job and we don't need the ostree conversion magic.

we should also burn it down with fire and replace it with a testable Python script, but nobody had time to do that yet.

After a bit of talking and thinking about this, there seem to quite a lot of moving parts, so at @kevin 's suggestion I created a discourse thread: https://discussion.fedoraproject.org/t/we-need-to-come-up-with-a-consistent-approach-for-generating-and-publishing-containers-both-traditional-and-atomic-desktop-containers-both-stable-and-unstable-releases/109213

I would like to rewrite the sync scripts, but the design for that feels a bit tied up with the larger "how exactly do we want to be doing this?" question.

If I understand correctly, this will re-download the images and push them to the registry. This would be very time, network and disk space consuming for the ostree images which are rather large.

Yes, it does that, and it's very silly. It should get the compose location as an input, and it should just use the images.json metadata to find the images and access the files within the compose; if the compose location is on a local filesystem it should just use it, not redownload. That's one of the things I wanted to do in a rewrite. :D

update: we have now extended the 'cloud' image uploader to handle container images too - https://pagure.io/cloud-image-uploader/pull-request/10 . https://pagure.io/releng/issue/12142 covers deployment of this. that will publish the native oci containers for atomic desktops. it could easily be extended to cover IoT too, but I think IoT still hasn't been native-container-ified yet?

it could easily be extended to cover IoT too, but I think IoT still hasn't been native-container-ified yet?

The fedora-bootc project builds are done for historical reasons under iot, see e.g. https://kojipkgs.fedoraproject.org/compose/iot/latest-Fedora-IoT-41/compose/
and threads like https://gitlab.com/fedora/bootc/tracker/-/issues/15

today we have pushed the new container sync script to production for everything except IoT, so the atomic desktop ostree native containers built by bodhi will be what is actually synced to registries now.

IoT is on hold pending https://pagure.io/fedora-iot/pungi-iot/pull-request/102 .

I've realized that we are not building the right manifests in bodhi. PR: https://pagure.io/fedora-infra/ansible/pull-request/2372

It also looks like we are not pushing each compose under a versioned tag. Opened: https://pagure.io/cloud-image-uploader/issue/37.