#171 [VOTE] To mark recent failure on Atomic images as non blocking
Closed 6 years ago Opened 7 years ago by kushal.

You can view the error at https://apps.fedoraproject.org/autocloud/jobs/606/output#227

Related bugzilla bus

We want to mark these issues as non-blocking.

Can the WG members please vote in? (Aye, Nay, Abstain).


First I should vote: Aye (to make them non-blocking).

What is the point of having the testcase if we want to release images even though they have bugs?

This testcase is also not a non-gating test AFAIK. https://github.com/kushaldas/tunirtests/blob/master/cloudtests.py#L61-L98

Replying to [comment:2 trishnag]:

What is the point of having the testcase if we want to release images even though they have bugs?

So that we can find and fix bugs faster when they do happen. If we choose to release with the bug anyway then we know it exists and can guide users accordingly until it is fixed.

This testcase is also not a non-gating test AFAIK. https://github.com/kushaldas/tunirtests/blob/master/cloudtests.py#L61-L98

Does that link prove it is a non-gating test? Where is the list of gating vs non-gating?

I vote aye (+1) to release with this known bug.

Replying to [comment:3 dustymabe]:

Replying to [comment:2 trishnag]:

What is the point of having the testcase if we want to release images even though they have bugs?

So that we can find and fix bugs faster when they do happen. If we choose to release with the bug anyway then we know it exists and can guide users accordingly until it is fixed.

Okay, if we choose to release with the bug anyway does it mean that the bug is not going to affect users much?

This testcase is also not a non-gating test AFAIK. https://github.com/kushaldas/tunirtests/blob/master/cloudtests.py#L61-L98

Does that link prove it is a non-gating test? Where is the list of gating vs non-gating?
We put non-gating tests here https://github.com/kushaldas/tunirtests/blob/master/nongatingtests.py
Rest of the files have gating tests. Journal logging test is a gating test. So we have it in a separate file.

Replying to [comment:5 trishnag]:

Replying to [comment:3 dustymabe]:

Replying to [comment:2 trishnag]:

What is the point of having the testcase if we want to release images even though they have bugs?

So that we can find and fix bugs faster when they do happen. If we choose to release with the bug anyway then we know it exists and can guide users accordingly until it is fixed.

Okay, if we choose to release with the bug anyway does it mean that the bug is not going to affect users much?

Yeah. The idea is that we put items to a vote and determine the severity. This bug is not that severe as it is just a logging issue. While important, it shouldn't block release in this case IMHO. My opinion could change in the future, but since this has been going out in our images for some time now we aren't making the currently released image any worse by releasing the next image.

This testcase is also not a non-gating test AFAIK. https://github.com/kushaldas/tunirtests/blob/master/cloudtests.py#L61-L98

Does that link prove it is a non-gating test? Where is the list of gating vs non-gating?
We put non-gating tests here https://github.com/kushaldas/tunirtests/blob/master/nongatingtests.py
Rest of the files have gating tests. Journal logging test is a gating test. So we have it in a separate file.

Thanks

So:

My reason for unblocking these tests is simple:

The version we are currently distributing also fails these tests.

If we had a version on getfedora which passed the tests, and it was only the pending version which was failing, I'd keep them blocking. But I see no point in continuing to block on tests which the "stable" version also doesn't pass. Clearly we've decided it's acceptable to ship a version of Fedora Atomic which fails the log tests.

Replying to [comment:6 dustymabe]:

Replying to [comment:5 trishnag]:

Replying to [comment:3 dustymabe]:

Replying to [comment:2 trishnag]:

What is the point of having the testcase if we want to release images even though they have bugs?

So that we can find and fix bugs faster when they do happen. If we choose to release with the bug anyway then we know it exists and can guide users accordingly until it is fixed.

Okay, if we choose to release with the bug anyway does it mean that the bug is not going to affect users much?

Yeah. The idea is that we put items to a vote and determine the severity. This bug is not that severe as it is just a logging issue. While important, it shouldn't block release in this case IMHO. My opinion could change in the future, but since this has been going out in our images for some time now we aren't making the currently released image any worse by releasing the next image.
Got you. Thanks dustymabe.

This testcase is also not a non-gating test AFAIK. https://github.com/kushaldas/tunirtests/blob/master/cloudtests.py#L61-L98

Does that link prove it is a non-gating test? Where is the list of gating vs non-gating?
We put non-gating tests here https://github.com/kushaldas/tunirtests/blob/master/nongatingtests.py
Rest of the files have gating tests. Journal logging test is a gating test. So we have it in a separate file.

Thanks

Replying to [comment:7 jberkus]:

If we had a version on getfedora which passed the tests, and it was only the pending version which was failing, I'd keep them blocking. But I see no point in continuing to block on tests which the "stable" version also doesn't pass. Clearly we've decided it's acceptable to ship a version of Fedora Atomic which fails the log tests.

This implies a second decision to be made: once there ''is'' a working version, should the non-blocking test become a blocking one?

My inclination is to say "non-blocking" because I do a lot of testing on Atomic Host and never noticed the bug, personally. I generally figure that gating bugs should be ones which either (a) carry significant security risk or (b) make the system unusable, or significantly less usable, than the prior release.

However, surely the Fedora project has some standards around what's considered a gating bug?

Replying to [comment:10 jberkus]:

However, surely the Fedora project has some standards around what's considered a gating bug?

We do for the traditional 6-month releases:

https://fedoraproject.org/wiki/Fedora_Release_Criteria

For example, functional system logging is considered to block an alpha release:

https://fedoraproject.org/wiki/Fedora_25_Alpha_Release_Criteria#System_logging

The Atomic release doesn't have to follow the exact same rules, of course, but I'd highly suggest keeping in the ballpark and coordinating with QA for expertise.

Yeah, so then the recommendation would be to unblock until this bug is fixed, but then make it a gating issue thereafter.

Replying to [comment:12 jberkus]:

Yeah, so then the recommendation would be to unblock until this bug is fixed, but then make it a gating issue thereafter.

Agree. I wonder if we can have a concrete release criteria as emphasis on a issue can change quickly.

I don't know if my vote counts but I am +1 (Aye) to mark these as non-blocking.

Replying to [comment:12 jberkus]:

Yeah, so then the recommendation would be to unblock until this bug is fixed, but then make it a gating issue thereafter.
+1 to this.

@kushal changed the status to Open

7 years ago

Reopeing the ticket to mark the same tests back as gating tests for F26 cycle. Please vote: (Aye, Nay, Abstain)

we should close this ticket - we should do what makes sense and stop bogging ourselves down with "process"

Metadata Update from @dustymabe:
- Issue status updated to: Closed (was: Open)

6 years ago

Login to comment on this ticket.

Metadata