#600 Increase the percentage cover of release validation automation
Opened 2 years ago by sdharane. Modified 2 years ago

Increase automation of release validation tests from 69% to 80%. The nature of tests is such it is very challenging to automate and has been done manually so far. With this we are going to set aside time to work on these hard to automate tests and target 80% coverage


we have actually automated several tests since the 69% estimate I sent you a while back. I'll post an updated estimate here soon.

So, we have automated 5 tests since the previous estimate. That gives 92 automated and 34 not automated out of a total of 126: new coverage percentage is 73%.

Cool, I hope we can reach the 80% threshold before F32.

I've done QA:Testcase_Anaconda_User_Interface_VNC and QA:Testcase_Anaconda_User_Interface_VNC_Vncconnect in the last couple of days, so that's two more.

I did an updated calculation on 2020-01-21 but didn't update this ticket. It came out to 135 total tests, 106 automated, 29 not: that's 78.52% automated. I believe the number is about the same right now.

Obvious targets for automation to get us up to that 80% would be desktop_login, desktop_panel_basic, desktop_keyring, package_install_remove, and GNOME Software upgrade test cases.

I did an updated calculation on 2020-01-21 but didn't update this ticket. It came out to 135 total tests, 106 automated, 29 not: that's 78.52% automated. I believe the number is about the same right now.
Obvious targets for automation to get us up to that 80% would be desktop_login, desktop_panel_basic, desktop_keyring, package_install_remove, and GNOME Software upgrade test cases.

Ok, I will look into it, when we are out of the Beta testing phase. However, this will not be super easy, especially the Gnome Software Upgrade - with no possibility to upgrade to Rawhide via Gnome software, we will need to provide R-2 and R-1 images for it, I believe.

we have a ticket for the GNOME Software upgrade thing, I feel like we should be able to hack around it somehow or other. the anaconda help screen test is another one we could automate, that should be easy...

Quick update here - we've been focusing on automating IoT test cases for the last few weeks. Technically that doesn't change the status here because IoT is not release-blocking yet, and for the purposes of this calculation I've been leaving non-blocking tests out entirely, but we are kinda expecting IoT to become release-blocking for F33 or maybe F34 at latest, so I expect I'll re-do this once that is decided.

Login to comment on this ticket.

Metadata