Learn more about these different git repos.
Other Git URLs
This is part of https://fedoraproject.org/wiki/Changes/OstreeNativeContainer
Basically, pungi today invokes rpm-ostree compose tree. It can keep doing that, but also invoke rpm-ostree compose container-encapsulate which will convert the generated ostree commit into a container image that can be pushed to a registry, and used client side for updates.
rpm-ostree compose tree
rpm-ostree compose container-encapsulate
A bit more on this here: https://coreos.github.io/rpm-ostree/container/
I'd propose we push to quay.io/fedora/fedora-silverblue for example, matching the already extant https://quay.io/repository/fedora/fedora-coreos
quay.io/fedora/fedora-silverblue
(FCOS uses coreos-assembler, which has learned about ostree native containers for some time and we're slowly reorienting the system around it)
It'd be good to manifest list this image - I think the process of doing so can function similarly to the "plain" base image?
(What process/code today pushes https://kojipkgs.fedoraproject.org/compose/rawhide/latest-Fedora-Rawhide/compose/Container/ to registries?)
Don't remember you engaging with the IoT WG for this feature.
Good point! I filed https://github.com/fedora-silverblue/issue-tracker/issues/359 for the silverblue case and https://bugzilla.redhat.com/show_bug.cgi?id=2129654 for IoT.
minimal and base images are pushed via script:
https://pagure.io/releng/blob/main/f/scripts/sync-latest-container-base-image.sh
Metadata Update from @phsmoura: - Issue tagged with: medium-gain, medium-trouble, ops
I'm a little confused about "(FCOS uses coreos-assembler, which has learned about ostree native containers for some time and we're slowly reorienting the system around it)" -- aren't we moving toward osbuild/ImageBuilder, generally?
aren't we moving toward osbuild/ImageBuilder, generally?
That's a massive sub-thread, lots of discussion on it (I only found https://github.com/osbuild/osbuild/issues/93 but I know there's a more recent one)
In the longer term we'll work to align more but as of now these are independent things. (osbuild/IB can learn to generate ostree-native containers too, all the tools are there for it, but it is subject to discussion by that team).
If anyone just wants to see what this looks like, I pushed https://quay.io/repository/cgwalters/fedora-silverblue (a bit old, but I'm intentionally not keeping it up to date as I don't want people to actually use it from that location). See also https://github.com/coreos/rpm-ostree/issues/4012
We actually have four OSTree variants:
So we'd need to think about this in the context of all four of them. FWIW, there was no engagement for Kinoite either.
Insofar as OSBuild/ImageBuilder usage, I don't know what we're going to do there. In my experience, high-level conversations with that team aren't terribly productive, and I'm not super-appreciative of all this pressure to move to OSBuild for everything when Fedora feedback/requirements are ignored by default anyway.
While COSA is a pain and everything around it is kind of invisible and intangible to Fedora contributors (and seemingly for anyone outside of Red Hat, though it's been a while since I've last checked), it's at least working and seems to be the primary focus for the RPM-OSTree consumers.
This whole situation is kind of a mess. :frowning:
I'd like to challenge this. Please reach out to me before carrying this narrative forward in new conversations so we can dispell any potential/existing confusion.
This is not my experience here, I have found them to be responsive and helpful, they do at times take time to do things but they are a busy team but they actively review submissions and I've never found them to ignore things.
It's the primary focus of FCoS, ATM that is the only thing currently using COSA in Fedora.
While COSA is a pain and everything around it is kind of invisible and intangible to Fedora contributors (and seemingly for anyone outside of Red Hat, though it's been a while since I've last checked), it's at least working and seems to be the primary focus for the RPM-OSTree consumers. I'd like to challenge this. Please reach out to me before carrying this narrative forward in new conversations so we can dispell any potential/existing confusion.
The main problem I have right now with the CoreOS stuff (including COSA) is that there's no obvious visibility for composes, builds, commit promotion, etc.
The stuff that's on the FCOS brochure website? There's no visibility for that.
Insofar as OSBuild/ImageBuilder usage, I don't know what we're going to do there. In my experience, high-level conversations with that team aren't terribly productive, and I'm not super-appreciative of all this pressure to move to OSBuild for everything when Fedora feedback/requirements are ignored by default anyway. This is not my experience here, I have found them to be responsive and helpful, they do at times take time to do things but they are a busy team but they actively review submissions and I've never found them to ignore things.
At least based on the conversations Fedora Cloud members have had so far with the team, we're not particularly interested in using OSBuild. And honestly I'm kind of dreading migrating the spins to it based on that experience alone.
While COSA is a pain and everything around it is kind of invisible and intangible to Fedora contributors (and seemingly for anyone outside of Red Hat, though it's been a while since I've last checked), it's at least working and seems to be the primary focus for the RPM-OSTree consumers. I'd like to challenge this. Please reach out to me before carrying this narrative forward in new conversations so we can dispell any potential/existing confusion. The main problem I have right now with the CoreOS stuff (including COSA) is that there's no obvious visibility for composes, builds, commit promotion, etc. The stuff that's on the FCOS brochure website? There's no visibility for that.
All of the build output can be browsed in the unofficial builds browser. That browser includes all builds (development and production) built by the pipeline (source code). The source configs for the input to the pipeline come from https://github.com/coreos/fedora-coreos-config . The promotion is all done in git. See the checklist executed for the most recent stable build. All changes are linked from that issue.
stable
You can also very easily build FCOS locally and run the exact tests we run in CI. Which is very powerful.
The one thing that isn't transparent right now is build and test logs. We use Jenkins for our builds and are being conservative with access there to protect secrets because AIUI jenkins isn't easy to lock down completely. We'd like to improve on this, though.
So my question is: how much have you really tried to understand what's going on? I'd really like for you to take a new look before you continue to talk about it in this way in the future. We discussed this at devconf.us and I haven't heard from you or seen you trying to understand more (asking questions if you can't find the information). Maybe you could argue that we should have this all written down somewhere nicely and you'd be right, though I don't think it's any more or less difficult to discover and poke around what's going on for pungi/koji in our traditional fedora releng builds. Just different.
While COSA is a pain and everything around it is kind of invisible and intangible to Fedora contributors (and seemingly for anyone outside of Red Hat, though it's been a while since I've last checked), it's at least working and seems to be the primary focus for the RPM-OSTree consumers. I'd like to challenge this. Please reach out to me before carrying this narrative forward in new conversations so we can dispell any potential/existing confusion. The main problem I have right now with the CoreOS stuff (including COSA) is that there's no obvious visibility for composes, builds, commit promotion, etc. The stuff that's on the FCOS brochure website? There's no visibility for that. All of the build output can be browsed in the unofficial builds browser. That browser includes all builds (development and production) built by the pipeline (source code). The source configs for the input to the pipeline come from https://github.com/coreos/fedora-coreos-config . The promotion is all done in git. See the checklist executed for the most recent stable build. All changes are linked from that issue.
Why is all this "unofficial" infrastructure?
You can also very easily build FCOS locally and run the exact tests we run in CI. Which is very powerful. The one thing that isn't transparent right now is build and test logs. We use Jenkins for our builds and are being conservative with access there to protect secrets because AIUI jenkins isn't easy to lock down completely. We'd like to improve on this, though.
This is the main problem I have right now.
I have tried a few times in the last year. But I haven't had much time since DevConf.US about it because of IRL issues.
This is official but on the side build infrastructure because doing what we do in FCOS in the classic Fedora infra would be a major undertaking.
On the Silverblue/Kinoite side, I'm currently working on this issue (tracked in https://github.com/fedora-silverblue/issue-tracker/issues/334 & https://github.com/fedora-silverblue/issue-tracker/issues/359) to do building and testing in CI in GitLab.com in the Fedora namespace. I have a WIP setup in https://gitlab.com/fedora/ostree/ci-test.
And I have the same problem: what's easy for me to setup in GitLab.com CI to get PRs tested will take much more effort to get in Fedora infra, where Silverblue & Kinoite are basically not tested at all before updates are pushed. Our composes are only tested after by OpenQA, but it's too late to prevent issues from getting to users and this happened several times already in Silverblue/Kinoite.
A pre-requisite for that work is also https://fedoraproject.org/wiki/Changes/FedoraSilverblueUnifiedCore (which I'm also working on, tracked in https://github.com/fedora-silverblue/issue-tracker/issues/333).
All of the build output can be browsed in the unofficial builds browser. That browser includes all builds (development and production) built by the pipeline (source code). The source configs for the input to the pipeline come from https://github.com/coreos/fedora-coreos-config . The promotion is all done in git. See the checklist executed for the most recent stable build. All changes are linked from that issue. Why is all this "unofficial" infrastructure?
The main pipeline runs on the Fedora Infra OpenShift at https://console-openshift-console.apps.ocp.fedoraproject.org. That is official Fedora infrastructure. It's just not pungi/koji. All of this is sanctioned by Fedora Release Engineering.
https://console-openshift-console.apps.ocp.fedoraproject.org
pungi
koji
You can also very easily build FCOS locally and run the exact tests we run in CI. Which is very powerful. The one thing that isn't transparent right now is build and test logs. We use Jenkins for our builds and are being conservative with access there to protect secrets because AIUI jenkins isn't easy to lock down completely. We'd like to improve on this, though. This is the main problem I have right now.
It's a valid concern, but I don't think this one thing warrants the full sentiment you are reflecting.
So my question is: how much have you really tried to understand what's going on? I'd really like for you to take a new look before you continue to talk about it in this way in the future. We discussed this at devconf.us and I haven't heard from you or seen you trying to understand more (asking questions if you can't find the information). Maybe you could argue that we should have this all written down somewhere nicely and you'd be right, though I don't think it's any more or less difficult to discover and poke around what's going on for pungi/koji in our traditional fedora releng builds. Just different. I have tried a few times in the last year. But I haven't had much time since DevConf.US about it because of IRL issues.
All I'm asking (similar to what I asked at devconf.us) is to try again before continuing to talk about it in this way and ask questions when you find things you don't understand rather than doing this.
I put together https://github.com/cgwalters/sync-fedora-ostree-containers to demonstrate how easy this is (this does overlap a bit with what Timothée has prototyped in https://gitlab.com/fedora/ostree/ci-test - but that one demonstrates building in gitlab, this repository just re-encapsulates the official ostree commits).
So...this thing can literally be a shell script cron job or I can put it in a container image run in OCP or whatever. (Generating a proper manifest listed image as is done via the app base image sync script above could be put directly into this or done via outside tooling, but if we can get agreement to run this I can tackle that as a separate step (or happy to hand it off to someone else interested!))
Ping on this - looking for some feedback on running this 47 line shell script - should I try to wedge it into the https://pagure.io/releng/blob/main/f/scripts/sync-latest-container-base-image.sh script?
I think just wedging it into there for now makes sense... but sometime we should try and consolidate/saneify things. :)
OK so...I think we need some preparatory cleanup here. I did some in https://pagure.io/releng/pull-request/11120 (not tested locally)
Whoever added handling for the -minimal image just copy pasted everything, which was probably fine at the time, but I have a general rule: two copies of code is sometimes OK, but if you have 3 or more, it's time to deduplicate. Since I'm proposing syncing more containers here, it's time to clean up this script.
-minimal
(Personally I'd use either Go or Rust for this, over here we have another implementation of something similar https://github.com/coreos/coreos-assembler/blob/main/src/cmd-push-container-manifest )
The push the base container image is a bit messy :(, we have 2 different process between DockerHub and registry.fp.o + quay.io.
On the DockerHub side we have a rust cli to help fetch the builds, and prepare the dockerfile expected by DockerHub https://github.com/fedora-cloud/fedora-container-release This is then running in a Github Action https://github.com/fedora-cloud/docker-brew-fedora/blob/master/.github/workflows/update_image.yml
It might be interesting to try to consolidate everything, I wrote down the main steps a few years ago --> https://pagure.io/releng/issue/8270
Thanks for that background. It's also worth cross-referencing that for Fedora CoreOS this is all integrated into the pipeline - including actually testing what we build before it appears on the registry.
I think for right now my inclination is to just make a new bash script that can also be executed by cron. Risk of regression from touching this script seems high.
(But, IMO it would make far, far more sense to have this logic be a proper "controller" executed inside an OCP cluster itself - that makes it then ergonomic and natural to integrate testing too. Though we do want to test with plain podman too)
podman
On the DockerHub side we have a rust cli to help fetch the builds, and prepare the dockerfile expected by DockerHub https://github.com/fedora-cloud/fedora-container-release
This is cool! What's the "productization path" for stuff like this? IOW if I were to decide to write something similar (or fork/extend that), what are the required steps to deploy it?
I don't think it would be too hard to deploy this in fedora's OCP cluster, for fedora-container-release I just build it on quay.io (https://quay.io/repository/cverna/fedora-container-release) and it is running in GitHub Actions because it integrates nicely with GitHub workflow required by DockerHub.
fedora-container-release
But once it is built on quay it could be deployed anywhere :smile:
OK, PR in https://pagure.io/releng/pull-request/11180 !
For now since I had this all mostly already working in shell script, I kept it that way, but just made it a separate script.
I'd definitely like to pursue having more of infrastructure logic being in a Real Programming Language and deployed as containers though!
Status update here: a number of fixes have landed (more in https://pagure.io/releng/pull-request/11180 ) and I think today's rawhide compose is likely to work.
Status update:
$ oc image info quay.io/fedora/fedora-silverblue:39 error: the image is a manifest list and contains multiple images - use --filter-by-os to select from: OS DIGEST linux/amd64 sha256:a8b14684d93a37d9f2fa606e211fd4ebac1c707910b2301134115a9159fe4e50 linux/arm64 sha256:2c6068d5ff0309904b11b7a01fa8bae1c9ecbd5a867db466f9ef767d08c3a409 linux/ppc64le sha256:4423ac5b34493b5e1ea28deeb625b9fd344f2e457c2d34bca9667985e3629b19
There's another PR in https://pagure.io/releng/pull-request/11331 that will fix some things. After that, I think we can start running this on branched too and then we can close this ticket!
OK, we now have reliable sync of quay.io/fedora/fedora-silverblue:39 and quay.io/fedora/fedora-kinoite:39.
quay.io/fedora/fedora-silverblue:39
quay.io/fedora/fedora-kinoite:39
There's...a lot more to do, such as actually running tests on these images before we push them to the world (as we do for FCOS). But this is a useful start.
Metadata Update from @walters: - Issue close_status updated to: It's all good - Issue status updated to: Closed (was: Open)
This worked for me from a rawhide SB boot:
rpm-ostree rebase ostree-remote-registry:fedora:quay.io/fedora/fedora-silverblue:39
Notably by using ostree-remote-registry:fedora we are still verifying the embedded GPG signatures inside the container image.
ostree-remote-registry:fedora
In the future we should also start signing Fedora containers via sigstore (both this and the app container quay.io/fedora/fedora).
quay.io/fedora/fedora
But the main thing of course is you can now build and deploy derived images from this image. More in https://coreos.github.io/rpm-ostree/container/
this one is closed but we are missing sericea in the quay.io/fedora space. Is it something we should do in another issue?
I created the fedora-sericea repo there and it should populate on the next compose run. If it doesn't, please open a new ticket on it. ;)
Thanks!
Login to comment on this ticket.