Several release criteria are currently missing 'enforcing' release validation test cases:
We should create, and add to the appropriate release validation template page, test cases to cover each of these criteria.
I added the KDE package set test the other day - https://fedoraproject.org/wiki/QA:Testcase_Package_Sets_KDE_Package_Install .
Draft of a terminal test case: https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Testcase_desktop_terminal
Draft test cases for bootloader disk selection:
Current status: all Alpha issues mentioned above fixed.
For the start, I will focus on Disk Selection and Terminal Validation test cases.
Metadata Update from @lruzicka: - Issue close_status updated to: None
Just to reinforce this, for folks who said they were interested in working on this: the existing lists here, both the one in the initial description and the one in this comment, are definitely NOT up to date. Things have changed (both criteria and test cases) sufficiently since then that I'd say the first step here should be to draw up a new list: look through the criteria, where a criterion does not link to any related test case, obviously include it in the list; where a criterion does link to related test cases, check whether they cover all of the stuff the criterion talks about, and if not, include them in the list with a note of which bits are not covered. Once there's a new list, you can start working through it.
Lukas, Adam,
Would you be interested in getting together and discussing the criteria that need to be updated as a group? Or just use the Pagure comment system to share changes and thoughts before making changes?
Geoff Marr IRC: coremodule
On Thu, May 24, 2018 at 10:39 AM, Adam Williamson pagure@pagure.io wrote:
adamwill added a new comment to an issue you are following: Just to reinforce this, for folks who said they were interested in working on this: the existing lists here, both the one in the initial description and the one in [this comment](https://pagure.io/ fedora-qa/issue/397#comment-67137), are definitely **NOT** up to date. Things have changed (both criteria and test cases) sufficiently since then that I'd say the first step here should be to draw up a new list: look through the criteria, where a criterion does not link to any related test case, obviously include it in the list; where a criterion *does* link to related test cases, check whether they cover all of the stuff the criterion talks about, and if not, include them in the list with a note of which bits are not covered. Once there's a new list, you can start working through it. To reply, visit the link below or just reply to this email https://pagure.io/fedora-qa/issue/397
adamwill added a new comment to an issue you are following: Just to reinforce this, for folks who said they were interested in working on this: the existing lists here, both the one in the initial description and the one in [this comment](https://pagure.io/ fedora-qa/issue/397#comment-67137), are definitely **NOT** up to date. Things have changed (both criteria and test cases) sufficiently since then that I'd say the first step here should be to draw up a new list: look through the criteria, where a criterion does not link to any related test case, obviously include it in the list; where a criterion *does* link to related test cases, check whether they cover all of the stuff the criterion talks about, and if not, include them in the list with a note of which bits are not covered. Once there's a new list, you can start working through it.
Just to reinforce this, for folks who said they were interested in working on this: the existing lists here, both the one in the initial description and the one in [this comment](https://pagure.io/ fedora-qa/issue/397#comment-67137), are definitely **NOT** up to date. Things have changed (both criteria and test cases) sufficiently since then that I'd say the first step here should be to draw up a new list: look through the criteria, where a criterion does not link to any related test case, obviously include it in the list; where a criterion *does* link to related test cases, check whether they cover all of the stuff the criterion talks about, and if not, include them in the list with a note of which bits are not covered. Once there's a new list, you can start working through it.
To reply, visit the link below or just reply to this email https://pagure.io/fedora-qa/issue/397
honestly I'd say just someone should do it, it's not that hard, it takes like a couple of hours and then there's a list. :P
Hello, I have checked the release criteria pages and realized that some of them did not have their testcases mentioned. I think this should be the starting point to revise that. I think that in some cases, the test cases do exist, but they are simply not mentioned, or they do not exist at all, or they can be outdated, as in USB stick testing, where there are several methods recommended, but I was told by @kparal that we only support USB creation via Media Writer. In that case, those should be replaced by a test case covering that.
This only shows incomplete or outdated records.
There might be others to be created, but I believe that we somehow should focus on these first, and update the wiki pages for release criteria starting Fedora 29 if possible and leave the previous releases to its own fate.
as in USB stick testing, where there are several methods recommended, but I was told by @kparal that we only support USB creation via Media Writer.
Not exactly. The basic criterion says:
Release-blocking live and dedicated installer images must boot when written to a USB stick with at least one of the officially supported methods.
The problem is that that page keeps changing, and as I look at it now, it's no longer clear which methods are officially supported and which are just stated for completeness. We should definitely improve that situation, either adjusting the documentation or our criterion.
What I was saying was that if you try FMW, you've basically tested every dd-based method (dd, gnome-disks). So there's little incentive splitting those into different testcases and increasing testing load for very little gain. My assumption was that we no longer officially support livecd-iso-to-disk (on a former version of the docs page, when it was still on wiki, I believe that was the case).
Aah, thanks for the article @kparal. The question is, whether we really do want to keep all those methods "official" and test for all those tools, or whether we just want that Fedora can be installed from USB using any of the methods that create the USB disk.
I understand that this will need a little discussion to decide, but for me I would suggest getting rid of the livecd-iso-to-disk in both test cases and documentation, if we do not support it any longer, I would retire the UNetbootin (according to documentation, it produces worse results than dd) and I would stick to Fedora Media Writer and dd in both documentation and test cases. If a FMW testcase is able to test the dd testcase at the same time, the better. In this case, we perhaps can only test the FMW.
livecd-iso-to-disk
dd
What do you think?
So, I used to maintain the wiki page that had the USB install instructions, and I kinda had a scheme for how the criteria and the test cases and the wiki page all worked together, but then it got moved to docs and now I'm not maintaining it any more :/
The basic problem is that livecd-iso-to-disk can do some stuff that FMW can't. litd supports persistence and non-destructive writes. Non-destructive writes aren't very important any more, but some people do like persistence, it seems.
There was definitely a discussion which led to us only including the FMW test case in the Installation validation page any more, it'd probably be a good idea to go back and read the archives of that discussion, it would've been on test@ I think.
For the criteria page references, I'd definitely add the FMW test case to the references, and remove the 'Live_luc' one, that one definitely is outdated (in fact we should mark it as an archived test case, I'll do that). I'd probably leave dd and litd in there at least for now, they're not doing much harm.
So far as unetbootin is concerned, we never considered it officially supported, and the wiki page used to make this clear. The docs page is...less clear :(. The reason we documented it at all was that people persisted in using it regardless of whether we documented it, so my call was that listing it on the page with an explicit notice that it was not supported and we did not recommend people to use it was better than ignoring it. The entry on the wiki page basically existed in order to say "you shouldn't use it and we won't help you if you do".
Just for reference, here's how the wiki page looked the last time I touched it.
"Missing list of Featured server roles in Role functional requirements." - yeah, this has always been a bit icky, though practically it's not a problem, as we know they're "domain controller" and "database server".
The Server PRD defines the concept of "Featured server roles" but doesn't say what they are. The Server technical specification lists two "Supported" roles, which is not a term that really occurs anywhere else I can find. The release criteria use the term "release-blocking role" once - which similarly doesn't really occur anywhere else - and then "Featured role" the rest of the time, which does match the PRD. The Basic criteria page claims that there is no list of featured server roles, then specifies a list itself.
This is all a bit off. :P Poking @sgallagh . I know we need to redo all this stuff, but in the meantime perhaps we should change the tech spec to state that "domain controller" and "database server" are the Featured (not "supported") roles from Fedora 21 onwards, drop the one use of "release-blocking role" in the criteria, and have the criteria point to the tech spec list?
"Testcase_anaconda_lvmthin_rootfs_on_disk_partition (page does not exist)." - should probably point to QA:Testcase_partitioning_custom_lvmthin.
Metadata Update from @adamwill: - Issue assigned to adamwill
sigh, I probably need to re-survey this, but I know there are still some such cases.
Metadata Update from @adamwill: - Issue set to the milestone: None (was: Undetermined Future)
Login to comment on this ticket.