As well as criteria and test cases, we will need validation result tracking for IoT.
Ordinarily we simply fold new editions etc. into the existing Wikitcms 'matrix templates' (the template pages for the big result tables in the result pages like this one). However, for IoT it's not so straightforward because we are not composing IoT as part of the main 'Fedora' composes (Rawhide and Branched) where all other release-blocking images are composed. It is being composed in separate 'Fedora-IoT' composes.
The design of Wikitcms is fundamentally tied to composes: one validation event is associated with exactly one Pungi compose. So when we create a validation event for a mainline 'Fedora' compose, that compose has no IoT images and we cannot sensibly associate any IoT test results with it.
I don't think any kind of awkward hack where we 'associate' a Fedora compose and an IoT compose for validation purposes makes sense here. Apart from anything else, AIUI it's at least theoretically a possibility that mainline and IoT releases could diverge, so that wouldn't be appropriate.
Given that, I see two things we can do here (either, or, both):
Set up a parallel stream of events in Wikitcms - so we have two streams of validation events concurrently, for 'Fedora' and 'Fedora -IoT'. We have precedent for this with the old 'Modularity' composes, and most of the bits to handle that are still in python-wikitcms and associated tools, so hopefully it shouldn't be too painful to implement
Use this as an opportunity to try some shiny tool that isn't a crazy ball of mediawiki hacks. @sumantrom was talking about doing a test deployment of Kiwi; it strikes me this could be a nice opportunity, because it's probably going to be a lot easier to do just enough work to cover IoT than it would be to replicate the whole of Wikitcms' coverage all at once
We could probably do both of those things, the first in production for now, and the second as a test with a view to maybe becoming 'production' for the 33 or 34 cycle if things pan out.
Tagging @pbrobinson , @pwhalen , @coremodule , @kparal for feedback, but I'll probably do the implementation at least for the Wikitcms part, as I know where the bodies are buried...
also @mattdm
Thank you for a detailed description of the situation. We have been talking about trying a real TCMS for quite some time and I think that we should absolutely give Kiwi a try. Especially, when there is an option to use the WIkiTCMS approach with no big pain. Thus a possible failure will not bite us that much.
For the chain of actions itself, I suggest we do it vice versa - let's do the Kiwi as a primary solution and at the same time, let us also report to Wiki for backup.
I watched a video on how to do some things in Kiwi and it seems there will be more work needed to set everything up. So I think that there might be a risk of letting this go, if we arrive into a lack of time - which we probably will. We might end up with everything in WikiTCMS again with an incomplete Kiwi solution and we might throw it overboard before we actually start using it.
I'm very skeptical that we can deploy Kiwi, configure it and learn how to use it during the release validation cycle, especially when Beta Go/NoGo is already next week. But feel free to surprise me :-) The wikitcms hack looks as our best bet in the moment. Or, in the worst case, manually created wiki matrices just for the most important composes (and OpenQA results will be manually checked in OpenQA interface). Kiwi will then probably be a high priority task after F32 is out.
One more additional thought. Assuming F32 IoT is subject to the same release schedule as main Fedora 32, why do we keep the composes separate? Can't we just make IoT part of the same compose, at least for F32? That will save a lot of trouble for QA. Is there a disadvantage?
I see @mohanboddu asked a similar question here, but it's unanswered: https://pagure.io/fedora-iot/issue/24#comment-627576
Ping @mattdm @pbrobinson @mohanboddu @bcotton
Just for the record, I'm working around to the python-wikitcms side of this, but a little indirectly (I want to do some spring cleaning on the project first and write a bunch of missing test coverage for it).
Metadata Update from @sumantrom: - Issue set to the milestone: Fedora 33 (was: Fedora 32)
So finally, I have some progress on this in python-wikitcms. It's still WIP, but getting there. Sorry for the delay, modernizing python-wikitcms first turned out to be a bit of a job.
This is all done now (well, option 1) from my initial post is), with recent python-wikitcms and relvalconsumer releases. We have our first event and it all seems to be working.
Metadata Update from @adamwill: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.