My initial understanding here was the proposal was to drop the full iso and live iso media from release blocking, but I see the netinst iso in the change proposal. Any of those can be used from USB media, which the proposal also mentions, so I am confused.

1) Would any iso files still be included in the release criteria? If so, which ones?

2) I assume we would still try to make iso media but this leaves us a way out if we need to drop one or more to get a release done. Is that a correct read of the proposal?

3) Would Fedora want to alter the preferred method of installation for users with this change proposal? If so, what would that be?

1) All.

2) We would still produce the same iso media. No way out.

3) No.

This change is not about iso files. It's about issues with installation from real optical media. Any other blocking way of using the iso files would remain blocking (as said in the summary). We would still generate and block on the same iso files.

I fully support the QA's motivation not to test this any more.

However, I'm not so sure we should not block a release if some volunteer finds a bug that involves optical media, especially since we don't have an official way to do a fully supported "respins".

I understand @kparal's argument that "without regular QA testing, this will have a tendency to get discovered very shortly before Final release, and then people will be just mad at QA for not detecting it sooner" - however if it ain't blocking at all, I see people getting even angrier if they do discover it, but a new Fedora version is released nevertheless.

That said, consider me +0. I won't block this if other FESCo members approve it, but I'm not confident enough to wave it in, especially given the timing, when a single plus vote can very well become the only thing needed to get things approved.

Thanks for the clarification, @churchyard. I agree that testing installation from optical media is a huge undertaking and the value is questionable these days. I share @churchyard's concern here, especially if a volunteer finds a bug very late.

On the other hand, I may be overly concerned for something that does not have that much of a demand anymore. Given what I work on these days, I am interested to see if rawhide gating can incorporate at least one or more automated install-from-iso tests in qemu emulating an optical drive. I know that is not complete, but it could give us a baseline on the usability of install from optical media at that point in time.

We actually test install-from-iso, @adamwill said:

It would be nearly impossible to do the physical testing before virtual
tests have run, because the virtual tests are automated, openQA runs
them as soon as the compose completes. If virtual optical media boot is
broken we find out approximately 10 minutes after the compose completes
(5 minutes for the tests to be scheduled and the Everything network
install ISO to be downloaded, 5 minutes for the test to time out).

Whether or not we can have a compose based gating and run openQA tests before we break things is a completely different story - I agree that it would be indeed very nice. But I would not strat with install-from-iso tests in qemu, but rather something arguably easier, like testing if an update doesn't break reverse dependencies.

The only bit of Rawhide gating that isn't complete is actually having the compose sync process respect the gating decision. We already run the tests, and have a gating config. The compose check reports even tell you whether each nightly compose passed the gating check or not. releng just hasn't ever actually wired the sync scripts to the gating check.

I think that's a bit academic, though - whether we gate the compose or not, if booting from ISO-as-virtual-optical-disc is broken, so long as anyone is actually looking at openQA results (which at least one person will be so long as I'm around here), we will know about it very quickly. That is the case right now and it would still be the case if this Change was adopted, because I'm not about to go change the openQA tests to boot the install media as USB sticks instead. We might even still want to block on the case of virtual optical booting.

I feel that the proposal is still easy to misunderstand, because it is too easy to confound "optical media" with the "optical media image" and the proposal does not make the difference sufficiently clear, as the mailing list discussion shows.

Proposal to make all Fedora optical media non-blocking.

Maybe "Remove the requirement for release media to pass a test of booting from actual optical drive hardware. Tests with emulated hardware in qemu virtualization, tests with the hybrid ISO image written to a USB drive, and other ways that use the hybrid ISO image will not change." ?

Another thing that is unclear is where booting from virtualized optical media comes into this.

--

There are two aspects to the change: 1. that the requirement is gone from Release Criteria, 2. that QA stops doing the tests.

I'm supportive of QA not doing the tests with physical optical media (because of all the reasons mentioned: that it takes a long time and hybrid images are tested in other ways and bugs that would only affect physical hardware are rare). But OTOH, I think that releasing an ISO image that would not boot from physical optical media would be pretty bad too. Another aspect is that the voices of people who declare that they care about physical optical media should be somehow accommodated. Clearly there's a part of the community which care about this, and even if QA does not assign resources to the problem, those people should still get a way to contribute support for that feature if they want.

I'd propose the following amendment:
1. boot from physical optical media is removed from QA matrices
2. boot from virtualized optical media is tested (I believe that openqa currently does this)
3. passing of 2. is considered enough to satisfy the release criteria
4. community members are encouraged to test with physical optical media. If bugs are reported, they will block the release if the fix is considered feasible and the the bug was found sufficiently early.

With @zbyszek 's amendments from the previous comment, I would vote +1; otherwise ±0.

Pretty much the same for me.

After a week, this is (+0, 1, -0)

Let me just drop in a temporary -1 here, so we talk about this on the meeting and so it is not approved upon the next single plus vote.

I like @zbyszek's amendments, but I propose further revising it to allow booting from physical or virtual media as satisfying the release criteria. This removes the requirement for physical media testing, but leaves QA the option to verifying CD/DVD booting using physical or virtual means.

As written I read it as requiring the iso to boot in a virtual environment only.

The Fedora Workstation working group is not opposed to the change. To the contrary, the working group is more concerned about negative UX resulting from distribution of optical install media for marketing purposes. Statement about this in:
https://pagure.io/fedora-workstation/issue/117#comment-617129

I propose further revising it to allow booting from physical or virtual media as satisfying the release criteria

Yes, that was my intent too to mean either of the tests to satisfy the criteria.

Just a note, mid-December after the initial feedback on devel list, we've added this sentence to the proposal's summary, to clarify the proposal:

Installation from USB devices on bare metal machines and installation from virtual optical drives in virtual machines will remain blocking.

So even if the proposal is accepted in its original version, we intend to keep virtual optical drive in libvirt supported and blocking (our OpenQA is based on this and we test it very frequently).

@zbyszek's proposal sounds as a decent alternative. Miro already linked to my concern, where I prefer having things Fedora blocks on and things QA is responsible for as close as possible. Exceptions make docs harder to read and can cause us get the flak, although unjustified. But it's surely a good enough alternative. We basically need to decide whether we'll rather delay the release if such a problem occurs, or create inconvenience for some part of our user base. And since this would be a departure from a functionality we've considered essential for a very long time, FESCo is, I believe, a good place to make that decision.

I don't think I'd support either/or. Attaching an ISO to a VM as a virtual optical drive is such a basic way to boot a VM that I think we should still block on it. Unlike the 'real world' case, there is no particular advantage to attaching it as a virtual USB media rather than as a virtual optical disc, and I would expect lots of people have existing configurations that rely on the latter. I would not be comfortable shipping a Fedora release which booted on physical optical media but not virtual.