by acceptance test events you mean the automated, RATS stuff? in which case, excellent. twu, if you need any help with this please ask, i'm happy to help.

Replying to [comment:1 adamwill]:

by acceptance test events you mean the automated, RATS stuff?

Yep.:)

I have created a wiki page for clarifying the procedure acceptance test events, and work on it:

https://fedoraproject.org/wiki/QA:SOP_Rawhide_Acceptance_Test_Event

Looks pretty good, here's a few notes:

We should clearly explain how RATs differs from release validation events (the manual tests on TCs and RCs). The easiest way to do this is just to mention the release validation stuff explicitly in the introduction, like this:

RATs are automated test events which happen before the [[QA/SOP_Release_Validation_Test_Event|manual release validation test events]] for each release phase (Alpha, Beta, Final).

https://fedoraproject.org/wiki/QA:Fedora_Rawhide_Acceptance_Test_Template is a bad page and we should probably revise it or stop referring to it (this is kind of a note to self, as I know the release validation SOP also refers to it).

Some of the stuff in 'Make announcements' isn't really relevant to RATs, I don't think, as we don't have much possibility for people to really contribute to RATs; it's more of a case of one person owns RATs, announces they're about to do it, then announces the results. It's not like release validation when we have lots of manual input to get in from different people. So I think we could really lose almost all of it, especially the middle two bullet points - "How and where to add test results" and "Contact information of QA members who are available on test event and can help testers who encounter problems", and all the text after the bullets...really, the whole section should be re-written much shorter, and should just link to an example RATs email (the current example is a release validation email) and mention the things that need to be in a RATs announce.

Same thing for the 'Provide help during test event' section. I think it's just not really necessary for RATs.

For the "Report and Summary" section, again, the example email link needs to be updated to link to a RATs email, not a release validation one.

Other than that, looking good! Nice. Thanks.

Actually, I just realized one big gap: the instructions on how to actually do a RATs run. Since you used the release validation SOP as a template there's no space for this, because for RV, announcing the test and publishing the matrices essentially is the work, but for RATs, we need to explain how you actually do a RATs run. Thanks!

Perhaps linking to the rawhide acceptance test plan might help with the how part of this (https://fedoraproject.org/wiki/QA:Rawhide_Acceptance_Test_Plan).

While this is mostly documented in the test plan and I see the acceptance test milestone primarily ...
1. to identify and resolve compose-related issues
2. file bugs for issues impacting the the rawhide acceptance test plan
3. find more bugs (early pre-TC1 testing)

We touch on #3 a bit in the F15 QA retrospective, so that's certainly an area worth exploring for future test plan enhancement. Another thing I'd like to see in the SOP is some discussion of exploratory testing. Using the RATs images, I've been trying to find and file bugs which I suspect will block or impact Alpha TC1 testing. Given the fix window between TC1 and RC1 is so short ... the earlier we can identify some of the bugs, the better.

Hi adamwill,

Thanks for your great comments, the following is my update:

Replying to [comment:5 adamwill]:

Looks pretty good, here's a few notes:

We should clearly explain how RATs differs from release validation events (the manual tests on TCs and RCs). The easiest way to do this is just to mention the release validation stuff explicitly in the introduction, like this:

RATs are automated test events which happen before the [[QA/SOP_Release_Validation_Test_Event|manual release validation test events]] for each release phase (Alpha, Beta, Final).

Thanks for the nice suggestion, an explain has been added.

https://fedoraproject.org/wiki/QA:Fedora_Rawhide_Acceptance_Test_Template is a bad page and we should probably revise it or stop referring to it (this is kind of a note to self, as I know the release validation SOP also refers to it).

Sorry I might don't understand correctly ... but I have checked again, the url can be opened well and it is not the same as the which release validation SOP also refers to. Can you explain it more in details?

Some of the stuff in 'Make announcements' isn't really relevant to RATs, I don't think, as we don't have much possibility for people to really contribute to RATs; it's more of a case of one person owns RATs, announces they're about to do it, then announces the results. It's not like release validation when we have lots of manual input to get in from different people. So I think we could really lose almost all of it, especially the middle two bullet points - "How and where to add test results" and "Contact information of QA members who are available on test event and can help testers who encounter problems", and all the text after the bullets...really, the whole section should be re-written much shorter, and should just link to an example RATs email (the current example is a release validation email) and mention the things that need to be in a RATs announce.

Same thing for the 'Provide help during test event' section. I think it's just not really necessary for RATs.

For the "Report and Summary" section, again, the example email link needs to be updated to link to a RATs email, not a release validation one.

Other than that, looking good! Nice. Thanks.

Okay, it is good idea , I dropped some content not so necessary to RATs and redirect the example url to the new one.

Thanks to James and adamwill,

The how is and important issue, a section about " how to actually do a RATs run " has just been inserted between the " When a new candidate build available " and " Report and Summary " sections.

Looking nicer. I've made a couple of changes: I gave the 'how to actually do testing' section a better name ('Perform testing'), and moved the 'When a new candidate build available' section to the end, where it makes more sense with the overall flow of the page.

I also rejigged the 'new build available' section, taking out the word 'candidate' as these builds aren't really candidates to be anything, and generally making it reflect reality a bit better.

I also fixed the link in the example email: it shouldn't have redirect=no in it, as we do want people to be redirected when they arrive :)

I'm not sure just linking to the acceptance test plan is sufficient instruction on how to do the testing. It's a planning document, and it has "TODO: Automate these test cases" written in it. :) Just reading that page, I have no idea of the actual steps one has to perform to complete the testing: run this program, click this button, etc. You could do the testing manually by following each test case linked there, but that's not really the point. What we really need are instructions by which someone who'd never done the acceptance testing before could successfully do the acceptance testing - automated, not manually. Thanks! (If it requires access to some Red Hat-only system, or something, then a) that sucks, but b) we should still write it down.)

Replying to [comment:10 adamwill]:

I'm not sure just linking to the acceptance test plan is sufficient instruction on how to do the testing. It's a planning document, and it has "TODO: Automate these test cases" written in it. :) Just reading that page, I have no idea of the actual steps one has to perform to complete the testing: run this program, click this button, etc. You could do the testing manually by following each test case linked there, but that's not really the point.

Just a note ... this isn't fully automated. I'd classify the automation as a mature proof-of-concept. In addition, I've rarely been satisfied with the automated results alone. The automation helps me figure out how far up I need to roll up my sleeves for testing. In my experience, this milestone calls for more hands on attention until the automation has proven itself.

What I've discussed with twu is that this milestone needs to be revisited and possibly extended to cover more test blocking issues ... not just it's original mission of 'rawhide acceptance'. The concept of rawhide has changed since this test plan was born. While the need still exists, this plan doesn't go far enough to find the bugs that will block testing of the 'test compose'. What I'd like to see out of this test run is identification of bugs for items identified in the current acceptance plan, but also bugs that block significant portions of the release validation test matrices. The one week between TC1 and RC1 is not sufficient time to identify, resolve, verify and refix those bugs. I'm convinced this has contributed to slips in all of the previous ALpha slips for ~4 releases (if memory serves).

What I'd recommend is to focus on defining the goal and documenting the process for this effort ... then we can follow-up with focused patching to the rats_* test automation.

What we really need are instructions by which someone who'd never done the acceptance testing before could successfully do the acceptance testing - automated, not manually. Thanks!

Pet peeve of mine ... I recommend ensuring the tests themselves are documented first. From there, we can fill-in the gaps with any automation that exists. The problem I have with the automation is that it's insufficient for what this activity calls for. It's a good start, it needs more work, and in it's current state, it can be a distraction from the goal of finding bugs that will prevent further release validation testing. Let's definitely improve the automation, which twu is looking into, but track that separately.

(If it requires access to some Red Hat-only system, or something, then a) that sucks, but b) we should still write it down.)

It doesn't.

At what point does a 'build some images and test them to see what bugs we'll find when we do test compose #1 validation testing' become test compose #1 validation testing? The way you describe it above, this process is essentially just TC0, it seems to me.

Replying to [comment:12 adamwill]:

At what point does a 'build some images and test them to see what bugs we'll find when we do test compose #1 validation testing' become test compose #1 validation testing? The way you describe it above, this process is essentially just TC0, it seems to me.

Yeah, pretty much! I don't think we need to go so far as running all release validation tests, but running just the existing rats_* automation isn't enough. There's a happy pony in the middle somewhere.

well, is there? What's the benefit of a happy middle pony? To me, the processes are only clearly separate so long as RATs is a simple fire-and-forget automated run. If we're going to start running significant chunks of the matrix manually, what's the point of having multiple processes? Why not just merge them into one TC process and start doing TCs two weeks earlier?

so, to enlarge, my concern is that by limiting the time scale and frequency of RATs runs while simultaneously enlarging the scope, we've made them almost indistinguishable in practical terms from TCs. I'd propose we re-balance things a bit:

we could be doing the fully automated RATs stuff daily, right? Every time there's a Rawhide compose? Is there anything preventing this? Couldn't it just be part of AutoQA that fires, posts a result, and everyone's happy?

we could then strictly define TC and RC: a TC is a published compose of whatever subset of images we decide on, which happens before a freeze so has no chance of being blessed as the release. An RC is the same thing, but after a freeze, so it may be blessed as the release. We then re-align the TC periods for each phase a little earlier, since we've noticed when we slip, it tends to be because we were late getting workable TCs. effectively, we turn the early RATs runs into TCs. We'd wind up with maybe 3-4 TCs per release, the first one or two of which may hit bugs very early in testing.

thoughts?

(oh, when we have Branched, we could do RATs for both Rawhide and Branched, if resources permit.)

Replying to [comment:16 adamwill]:

(oh, when we have Branched, we could do RATs for both Rawhide and Branched, if resources permit.)

We don't create install images for rawhide, so there's nothing for rats_install to do. But iirc, rats_sanity (repo sanity) still runs against rawhide.

Replying to [comment:14 adamwill]:

well, is there? What's the benefit of a happy middle pony? To me, the processes are only clearly separate so long as RATs is a simple fire-and-forget automated run. If we're going to start running significant chunks of the matrix manually, what's the point of having multiple processes? Why not just merge them into one TC process and start doing TCs two weeks earlier?

Sounds good, let's do it! :)

Tao - so, it occurs to me that we eventually mostly went with the 'just have autoqa rats and TCs' plan, but we kept two RATS points on the 17 schedule for the time before Alpha branches from Rawhide. I think we'll probably continue doing that in future, because it's not practical to build a full TC from Rawhide, but we do want those test points. So could you update the SOP (which still exists) to reflect that the RATS points will happen regularly only ahead of Alpha releases, and then we can close up this ticket? Thanks!

this is now all obsolete. We get nightly live and boot.iso from both Rawhide and (when it exists) Branched these days, and we have process for doing full release validation testing on nightlies; we also do TC1s for each milestone much earlier than we did in the days when this ticket was under discussion. We don't do anything called 'RATS' any more. We sadly don't have actual automated install testing working ATM either, though apparently we're getting close to having it in Taskotron.

Anyhow, there's no action left for this ticket, closing it. The SOP for manual release validation is https://fedoraproject.org/wiki/QA/SOP_Release_Validation_Test_Event , it covers nightly validation events as well as TCs/RCs.