This does depend to an extent on what the release plan for IoT is, to be fair. If it's intended that we don't build 'gold' IoT images as part of the final release - i.e. we don't expect the Fedora 32 RC compose to have IoT images which must pass validation, but we instead expect to release a separate stream, as we did for two-week Atomic - then keeping a separate stream even during Branched and Rawhide might be OK.

@adamwill I think the separate stream thing is the appropriate model here.

In general in the near future, I want to move to a model where the main compose doesn't produce images at all, and all images are separate streams, although we may have a block of "official release day" ones. (There is a proposal in the works about using Image Builder.) That's definitely out of scope of this conversation but seems like relevant context too.

Well, it would be nice to be consulted on that, I guess?

Our current QA process is quite strongly built around a 'there is one compose' model. I mean, that's ultimately what we validate: we validate a compose for release. The whole wikitcms setup is based on this: validation events are tied to composes, and the design kinda assumes there's one compose stream, not multiple ones.

If it is actually planned to move to such a model, then I'm going to have to do quite a lot of work on the wikitcms stuff, which isn't trivial to work on, and I'd rather like to have some notice on it, as opposed to having to frantically hack everything together in two days like the last time something like this happened (with the modular composes)...

I know you've been talking about this, but talking about it and actually doing it are pretty different. Just handling this for IoT will be work, if we actually want to validate IoT for Fedora 32 the way we validate everything else.

Yes, consulting you is included in the (very nascent) plans. I am familiar with the process and disruption to it. I promise no requests for frantic hacks.

it's just, if we're doing this for F32, it's not like we have a whole bunch of time :) we branched already, Beta is due in just over a month...

@pbrobinson We talked about this a while back? What are you plans for IOT composes? Do you wanna merge them with Fedora composes as a whole or make it a separate stream but still be using all the repos that current fedora releng uses or you wanna totally keep it separate?

well, Fedora 32 rolls along, the IoT images are still in a separate compose stream, no 'plans' have been announced, and no-one's replying to this ticket.

So in this information vacuum I guess we'll just have to do our best. I'll work on re-activating all the wikitcms stuff to cope with a separate compose stream and run a big 's/Modularity/IoT/g' on it all, i guess.

It's my understanding that the IoT images are actually built from the main Everything compose, so it's not really what I think of as a separate "compose stream". From QA perspective, every IoT compose is a "child" of some official, tracked compose.

Do I have something wrong in my thinking here?

I'm pretty sure you do, yes.

Here is a main Fedora 32 compose, with the short name 'Fedora':

https://kojipkgs.fedoraproject.org/compose/branched/Fedora-32-20200310.n.0/

If you examine its image metadata you will note the string 'IoT' does not appear in it. It has no IoT images.

Here is today's F32 IoT compose, with the short name 'Fedora-IoT":

https://kojipkgs.fedoraproject.org/compose/iot/Fedora-IoT-32-20200310.0/

That one's still running so it has no images, but here's the image metadata for the 0307.0 compose: you see it contains only IoT images.

Those are separate composes. There is no relationship between them. If you look at https://kojipkgs.fedoraproject.org/compose/iot/ for e.g. you'll see there was no IoT compose on 0308 or 0309, but on those days we did have regular 'Fedora' composes, as you can see in https://kojipkgs.fedoraproject.org/compose/branched/ .

There is no 'child' relationship between these composes so far as I'm aware. AFAIK 'child composes' aren't really a thing at all; there are layered composes, but that's not quite the same thing, we don't use them in Fedora, and that's not what this is.

The 'Fedora-IoT' compose config doesn't even live in the same repo as the 'Fedora' compose config. The main 'Fedora' compose config is in pungi-fedora. The 'Fedora-IoT' compose config is in a repo called pungi-iot.

"Those are separate composes. There is no relationship between them."

So since I posted that last comment I got the idea that by 'child compose' what you may mean is that the IoT composes use the nightly Branched composes as an input. If so, then I get what you mean by 'child compose', but it doesn't really resolve the process issues I mentioned so far - we still have the problem of the actual deliverables coming in different streams.

@sdharane said in our team meeting today that his understanding from Matt is that we may be able to have the IoT images included in the main Fedora composes, and have those be the deliverables we actually ship with the Final release, while still having the 'Fedora-IoT' compose stream for development purposes. If that was a correct understanding and we're actually planning to do that, it'd be great, but I haven't heard that directly from anyone myself yet!

So for now I'm still proceeding on the assumption we're going to be building the final 'Fedora 32 IoT' deliverables in a 'Fedora-IoT' compose and we'll need to identify and validate that compose somehow. So I'm working to resurrect wikitcms' ability to create events for two streams of composes at once (last used for the initial 'Modularity' stuff) and fix it up, and then hopefully we'll be able to have 'IoT' validation events created alongside the main ones. For now, @pwhalen is creating validation matrices for specific composes manually. I believe for Beta the plan is to ship the images from a 'Fedora-IoT' compose that should match the official Beta compose package set, I'm not sure if a specific compose has been chosen yet.

@pbrobinson Can we please have your response here? Is there any reason why we need to keep the IoT compose separate from the main Fedora compose?