+1

+1

I would like a fedora-blueprints project to be created that contains a complete osbuild definition for this before giving my assent, but for now I will not vote.

I would like a fedora-blueprints project to be created that contains a complete osbuild definition for this before giving my assent, but for now I will not vote.

What does that even mean?

It means that something like pagure.io/fedora-blueprints should exist, and in it, an osbuild blueprint file should be checked in that koji-osbuild can be pointed to in order to trigger an image build.

We discussed this during today's FESCo meeting with the conclusion that the discussion should continue here.

I'm not seeing any further discussion; I'm putting it back on the meeting agenda.

We didn't get to this topic today, so it will be discussed next week unless more movement happens on the ticket.

So to be clear here:

• This will be integrated into the main compose process
• We need this to be able to support more modern Arm platforms (Like Rock5B)
• I will create the blueprints repo but that will more come into effect for when we move the main desktops to this process (so we can stop using ImageFactory for arm images) - the out of tree image definitions still needs some work and that's why I am only moving the Minimal Image for this cycle so we can at least start to support the more modern arm devices. We were hoping to get those pieces complete in time for F-40 but we're not there yet.

Thanks for the update, @pbrobinson

With that, I think we should proceed with the voting. I'm +1 here.

+1

This will be discussed during the meeting today at 19:30 UTC.

We discussed this issue in today's meeting. We will continue discussions at the next meeting. There are some requirements from FESCO to move this forward.

1, provide documentation for Fedora Release Engineering to be able to replicate/modify the contents of the resulting artifact

2, create space with the blueprint repository for a point of reference for the artifact definition

so I said some things in the meeting that might be a bit wrong. Just wanted to clarify as best I can.

for me - representing quality, with some involvement in the releng side - the principal concern is that Fedora should not be reliant on anyone else to control the definition of our deliverables.

I'm trying to understand exactly how producing osbuild images in Fedora actually works ATM. It seems that, in practice, the components of osbuild itself do a lot of the work of defining the deliverables they produce. So the question of who controls the osbuild instance used to build Fedora deliverables is important. Honestly, if the answer is "releng does, and they can patch it if required, like they can with koji", I am less worried about the details of where exactly things are defined. it is already the case that some things that could be considered attributes of Fedora deliverables are defined in Koji, for instance, and if we need to change those, we can patch Fedora's koji deployment.

It seems like Fedora does have its own osbuild. We have packaged https://github.com/osbuild/koji-osbuild as https://src.fedoraproject.org/rpms/koji-osbuild , and there's a bunch of stuff in ansible about deploying osbuild workers.

If Fedora releng/infra is ultimately in control of the osbuild deployment that builds Fedora osbuild images, and can patch it if necessary, I'm personally less concerned about the details of what elements of image definition are done where in the process, though obviously it's best if it's clear and simple.

edit: I was kinda concerned about https://koji.fedoraproject.org/koji/taskinfo?taskID=113734470 - an osbuild task which failed, for some reason, on a connection to sso.redhat.com - but I'm not sure what the explanation for that is.

for me - representing quality, with some involvement in the releng side - the principal concern is that Fedora should not be reliant on anyone else to control the definition of our deliverables.

We can, and in IoT do, update the definitions in osbuild. We have been using osbuild for IoT for a couple of releases.

I'm trying to understand exactly how producing osbuild images in Fedora actually works ATM. It seems that, in practice, the components of osbuild itself do a lot of the work of defining the deliverables they produce. So the question of who controls the osbuild instance used to build Fedora deliverables is important. Honestly, if the answer is "releng does, and they can patch it if required, like they can with koji", I am less worried about the details of where exactly things are defined. it is already the case that some things that could be considered attributes of Fedora deliverables are defined in Koji, for instance, and if we need to change those, we can patch Fedora's koji deployment.

It was decided by powers that be to use the Red Hat run service.

We can (and regularly do) update the definitions and work with the osbuild team on a regular basis. Some things can be changed via API but not everything. They are changing the way those definitions work, that is taking some time, but before long we'll be able to feed a complete definition via the API so that will make things more straight forward.

It seems like Fedora does have its own osbuild. We have packaged https://github.com/osbuild/koji-osbuild as https://src.fedoraproject.org/rpms/koji-osbuild , and there's a bunch of stuff in ansible about deploying osbuild workers.

koji-osbuild is deployed and running on our koji instance, we use it for many of the IoT deliverables. The main osbuild instance is run by Red Hat as stated above.

If Fedora releng/infra is ultimately in control of the osbuild deployment that builds Fedora osbuild images, and can patch it if necessary, I'm personally less concerned about the details of what elements of image definition are done where in the process, though obviously it's best if it's clear and simple.

See above, one of the main reasons I am just doing the minimal image is to get a process in place for osbuild for the main compose and work out the various kinks and to work with the osbuild team to finish the definition pieces so we can move all the arm images over to it. The current ImageFactory has limitations and it's restricting the arm sig from properly supporting some of the newer arm devices people want. I feel my time is better spent working with the osbuild team on a supported tool rather than spending that time on a dead tool which I could better spend on supporting new arm devices.

ah. so for this Change, is the intent that the RH-run service or the Fedora-run service be used?

ah. so for this Change, is the intent that the RH-run service or the Fedora-run service be used?

I don't believe there is a Fedora-run service as the decision was made to use the hosted service to reduce the load on infra team. I don't remember who exactly made that decision but @mattdm likely knows the details.

To the best of my knowledge, there is no Fedora version of the service. Everything around osbuild is pointed at the one inside the Red Hat infrastructure.

By "Fedora-run service" I guess I meant the thing you described as "koji-osbuild is deployed and running on our koji instance, we use it for many of the IoT deliverables."

By "Fedora-run service" I guess I meant the thing you described as "koji-osbuild is deployed and running on our koji instance, we use it for many of the IoT deliverables."

Correct, it will run as part of the usual compose process in pungi and the builds run through koji-osbuild. You can see example in the iot pungi config here:
https://pagure.io/fedora-iot/pungi-iot/blob/main/f/fedora-iot.conf#_212

I was intending on filing the PR for the main fedora pungi config once it's approved.

so...sorry for the dumb questions...that workflow does not involve the build ultimately happening in the RH-run osbuild service? the actual build happens on a koji worker? or does koji-osbuild still ultimately dispatch to the RH-run service?

koji-osbuild dispatches to the Red Hat-run service. It's not designed to run osbuild in Koji.

oh. That was the piece I was missing. well, that's a shame.

oh. That was the piece I was missing. well, that's a shame.

Why? It's a service that runs and is fully supported by a team we engage with, the reason to choose that service over adding more work to the Fedora infra team, ultimately it's an API we interact with so I am unsure why it matters where it's run.

It's a problem because of the OSBuild architecture: the underlying definitions of what is a "fedora" type image is not under the control of the Fedora community, but instead under an opaque service that we do not lifecycle or have any meaningful way to change.

This can be a problem if we want to change something and we need it to be changed right away. We cannot even hot patch the service to change them.

It's a problem because of the OSBuild architecture: the underlying definitions of what is a "fedora" type image is not under the control of the Fedora community, but instead under an opaque service that we do not lifecycle or have any meaningful way to change.

This can be a problem if we want to change something and we need it to be changed right away. We cannot even hot patch the service to change them.

Even with say koji we can't change things right away, especially in freeze, we need a fix created, it needs to be tested and then we need a freeze break request to have it deployed.

As I stated above the osbuild team is working to resolve this, the reason why I have proposed just the minimal arm image because it doesn't generally change and hasn't changed in ages, is so we can work through all the process to get the definition side of it working in Fedora in conjunction with that team. I have already addressed these points above.

So to return to the previous point:

provide documentation for Fedora Release Engineering to be able to replicate/modify the contents of the resulting artifact

How does releng replicate/modify the contents of some previous artifact?

So to return to the previous point:

provide documentation for Fedora Release Engineering to be able to replicate/modify the contents of the resulting artifact

How does releng replicate/modify the contents of some previous artifact?

Is this a question for me? I don't know what this question means in that context.

The compose happens based on what is defined in pungi and that compose process replicates out the artifact, that part doesn't change in this proposal, we update the pungi config for the minimal image and pungi deals with it. I don't know what other replicate/modify that releng does.

This is exactly the same functionality that is being used for the Workstation Live feature, I am confused why there's a problem with this feature and there was none of these concerns with that:

https://fedoraproject.org/wiki/Changes/FedoraWorkstationImageBuilder

At least from my end, I was confused into thinking that there were blueprints for that image too. I only learned a couple days ago that there aren't.

At least from my end, I was confused into thinking that there were blueprints for that image too. I only learned a couple days ago that there aren't.

And that is something I am at least actively working with that team to improve. I don't feel it is fair to block this because of that for numerous reasons I have outlined above already.

I have explicitly limited this feature to one image that doesn't change very often at all so we can do that.

I thought we were supposed to be able to innovate in Fedora. If I can't improve it in a contained way like proposed while also actually providing Arm users features they've been literally screaming for at least a year I am not sure where else I am supposed to do that! They will probably just go and use Armbian or something else :(

The Workstation Change produces "an additional, non-blocking Fedora Workstation live ISO". It does not replace the existing, livemedia-creator-built Workstation live ISO, which is still the release blocking one that we ultimately put on the download page for users to download.

This was changed during FESCo review of the Change. The Change initially did propose replacing the official, blocking live ISO with an osbuild-built one.

Also, I'm not sure people were fully aware of the details of how osbuild is implemented at the time that Change was reviewed, tbh. I was not aware it was a Red Hat-hosted service rather than a Fedora-controlled one till Neal told me about it.

I'm sorry but I don't have the cycles to maintain and test two images. I currently have so few cycles to work on Fedora as it is I don't believe it's fair to ask people to maintain/test two different versions of the same thing.

Especially with the new arm platforms that don't work on the old image it actually increases the load exponentially. If that is the expectation I will remove this request and produce some remix external to Fedora and basically ignore the official deliverable or simply just not support the new platforms people actually want to use.

I'm sorry but asking to maintain exponentially more to me when I have less time than I had in the past is an unacceptable solution when I feel I have adequately addressed all the concerns that pertain specifically to the actual artifact being produced.

I think if there is concern of the hosted service over Fedora infra team hosting one and doing a whole lot more work that should be taken to the Council and have them address it and make a decision and decouple it from this particular deliverable because they are unrelated, I don't care where the service runs, I just need to consume it.

Personally I don't see why suddenly the use of the hosted service is a problem, IoT is using it without issue and the Live image is using it. Red Hat runs it and we have the ability to work with the team if it saves Fedora resources that can be focused elsewhere to provide overall better value I don't see why it's a problem. In Fedora we currently use Matrix, AWS, IRC and other hosted services.

I'm not trying to say that should be the expectation, and I'm not on FESCo. Just trying to answer the question. I'll butt out and let FESCo members comment from now.

edit: though on the 'replicate/modify' topic - there is https://osbuild.org/docs/hosted/image-builder-koji/#locally-reproducing-koji-image-builds . neal mentioned some concerns with it in https://github.com/osbuild/osbuild/issues/455 , but AFAICT for practical purposes ("oh no, the image build is going wrong, we need to try it locally and see why") that seems like it should be workable? well...assuming we get the "manifest and additional metadata" on failure as well as on success...

So, yes, as noted the flow here is: our pungi calls koji to make some image, it goes to a koji builder that has the plugin and that builder calls out to the api 'make me an image'. The service then builds things and sends information (and the image) back to koji, etc. This is using the same service that Red Hat provides their customers off console.redhat.com. There is also a (staging only so far) version of this service that is going to be provided to all fedora users (ie, login with your fedora account, use api to make images).

From my view this service isn't perfect, but it's also under a lot of development. The hope was that it would become more flexable and allow us to define images ourselves instead of relying more on upstream types, etc. As far as I know, that is still the plan, they just haven't gotten there yet. I am not a big fan of depending on external services, but self run things aren't always better (looking at you ImageFactory and OSBS). I do feel better about this service, since it's actually under heavy use by downstream customers, so if something changed we would at least get a lot of notice.

If Peter thinks this can be done, I'm still +1 to it. If it turns out the image isn't fully workable for some reason, it should hit the blocker process and we can back it out to the old image

We discussed this during today's FESCo meeting:
AGREED: APPROVED (+7, 1, 0)