Increase automation of release validation tests from 69% to 80%. The nature of tests is such it is very challenging to automate and has been done manually so far. With this we are going to set aside time to work on these hard to automate tests and target 80% coverage
we have actually automated several tests since the 69% estimate I sent you a while back. I'll post an updated estimate here soon.
So, we have automated 5 tests since the previous estimate. That gives 92 automated and 34 not automated out of a total of 126: new coverage percentage is 73%.
Cool, I hope we can reach the 80% threshold before F32.
I've done QA:Testcase_Anaconda_User_Interface_VNC and QA:Testcase_Anaconda_User_Interface_VNC_Vncconnect in the last couple of days, so that's two more.
I did an updated calculation on 2020-01-21 but didn't update this ticket. It came out to 135 total tests, 106 automated, 29 not: that's 78.52% automated. I believe the number is about the same right now.
Obvious targets for automation to get us up to that 80% would be desktop_login, desktop_panel_basic, desktop_keyring, package_install_remove, and GNOME Software upgrade test cases.
I did an updated calculation on 2020-01-21 but didn't update this ticket. It came out to 135 total tests, 106 automated, 29 not: that's 78.52% automated. I believe the number is about the same right now. Obvious targets for automation to get us up to that 80% would be desktop_login, desktop_panel_basic, desktop_keyring, package_install_remove, and GNOME Software upgrade test cases.
Ok, I will look into it, when we are out of the Beta testing phase. However, this will not be super easy, especially the Gnome Software Upgrade - with no possibility to upgrade to Rawhide via Gnome software, we will need to provide R-2 and R-1 images for it, I believe.
we have a ticket for the GNOME Software upgrade thing, I feel like we should be able to hack around it somehow or other. the anaconda help screen test is another one we could automate, that should be easy...
Quick update here - we've been focusing on automating IoT test cases for the last few weeks. Technically that doesn't change the status here because IoT is not release-blocking yet, and for the purposes of this calculation I've been leaving non-blocking tests out entirely, but we are kinda expecting IoT to become release-blocking for F33 or maybe F34 at latest, so I expect I'll re-do this once that is decided.
Ooh, hey, let's re-do this calculation...
I don't remember exactly how I did it before so the method might not be quite the same, but this time I've got:
For a total of (drumroll)...82.14% automated!
This is still a very vague measure - it's just a dumb count of literal wiki test cases which are the test case for at least one release-blocking row and for which openQA (or, in one case, relval) can submit at least one automated result. It doesn't take account at all of how we have multiple environments per test case, and sometimes multiple "result rows" per test case - those factors could distort the result in either direction. But I think that consideration existed with previous calculations too.
The list of non-automated test cases currently is:
most of those are pretty non-automatable or not desired to be automated. We are working on some of the ones that are automatable, @lruzicka is working on the graphical upgrade ones ATM for instance.
Since we're over 80%, let's close this 🥳
Metadata Update from @adamwill: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.