= problem =
So far we have similar test cases for a same test area, and it's not quite useful to test all of them.
= analysis =
Take RAID cases for example, as James said:"We aren't hitting any problems anymore that are specific to RAID0 but work on RAID5. The main focus for this tests is to make sure that we identify a real-world RAID install partitioning setup, anaconda can execute that partition scheme, and dracut can boot it."
= enhancement recommendation =
I would like to propose for removing the following cases and keeping a representative one instead:
RAID: * [https://fedoraproject.org/wiki/QA/TestCases/PartitioningRootfsOnRaid1 QA/TestCases/PartitioningRootfsOnRaid1] * [https://fedoraproject.org/wiki/QA/TestCases/PartitioningRaid0OnLvmDevice QA/TestCases/PartitioningRaid0OnLvmDevice] * [https://fedoraproject.org/wiki/QA/TestCases/PartitioningUsrOnRaid0 QA/TestCases/PartitioningUsrOnRaid0] * [https://fedoraproject.org/wiki/QA/TestCases/PartitioningUsrOnRaid5 QA/TestCases/PartitioningUsrOnRaid5] * [https://fedoraproject.org/wiki/QA/TestCases/PartitioningUsrOnRaid6 QA/TestCases/PartitioningUsrOnRaid6] (Replace them with this case: [https://fedoraproject.org/wiki/QA:Testcase_Install_to_Hardware_RAID QA:Testcase_Install_to_Hardware_RAID]) [[BR]]
Preupgrade: * [https://fedoraproject.org/wiki/QA:Testcase_Preupgrade_from_older_release QA:Testcase_Preupgrade_from_older_release] [[BR]]
FTP Install: * [https://fedoraproject.org/wiki/QA/TestCases/InstallSourceFtpAnonymous QA/TestCases/InstallSourceFtpAnonymous] (keep the [https://fedoraproject.org/wiki/QA/TestCases/InstallSourceFtpNonAnonymous NonAnonymous case]) [[BR]]
If you have any thought about it, please feel free to tell, thanks.
I love this idea, thanks for proposing!
Regarding the RAID tests, does it make sense to keep a representative copy for each of the 3 main RAID types? 1. Hardware RAID - [https://fedoraproject.org/wiki/QA:Testcase_Install_to_Hardware_RAID QA:Testcase_Install_to_Hardware_RAID] (as you've noted above) 2. Software RAID - some autopart-like partition scheme using Software RAID devices? 3. BIOS RAID - [https://fedoraproject.org/wiki/QA:Testcase_Install_to_BIOS_RAID QA:Testcase_Install_to_BIOS_RAID]
Replying to [comment:1 jlaska]:
I love this idea, thanks for proposing! Regarding the RAID tests, does it make sense to keep a representative copy for each of the 3 main RAID types?
Regarding the RAID tests, does it make sense to keep a representative copy for each of the 3 main RAID types?
Of course, thanks for clarifying them! My plan was to keep Hardware RAID and BIOS RAID case, but yes, SW RAID one should be added in the matrix.
Hardware RAID - [https://fedoraproject.org/wiki/QA:Testcase_Install_to_Hardware_RAID QA:Testcase_Install_to_Hardware_RAID] (as you've noted above) Software RAID - some autopart-like partition scheme using Software RAID devices?
I will create software RAID case if there's no existing one. For autopart-like partition, does it mean to put root filesystem on LVM volume, which was on new created RAID devices?
BIOS RAID - [https://fedoraproject.org/wiki/QA:Testcase_Install_to_BIOS_RAID QA:Testcase_Install_to_BIOS_RAID]
Replying to [comment:2 rhe]:
Replying to [comment:1 jlaska]: <skip> Software RAID - some autopart-like partition scheme using Software RAID devices? I will create software RAID case if there's no existing one. For autopart-like partition, does it mean to put root filesystem on LVM volume, which was on new created RAID devices?
Replying to [comment:1 jlaska]: <skip>
Software RAID - some autopart-like partition scheme using Software RAID devices?
I've created a software RAID case based on the steps in [https://fedoraproject.org/wiki/QA:Testcase_Partitioning_Rootfs_On_RAID6 QA:Testcase_Partitioning_Rootfs_On_RAID6]:
In this case, root(/) is partitioned on a RAID device, please review it. Thanks!
Software RAID - some autopart-like partition scheme using Software RAID devices? I will create software RAID case if there's no existing one. For autopart-like partition, does it mean to put root filesystem on LVM volume, which was on new created RAID devices?
Oh yes, sorry for not clarifying. I can't think of a critical reason why we would want LVM involved for this scenario. It's certainly a valid test. It's possible there would be problems executing a RAID+LVM partition scheme, or with finding the '/' volume while booting the installed system. If desired, we have a test for LVM on RAID (see [https://fedoraproject.org/wiki/QA:Testcase_Partitioning_LVM_LV_on_RAID_PV QA:Testcase_Partitioning_LVM_LV_on_RAID_PV]). However, I don't have strong feelings here. I trust your judgment.
I've created a software RAID case based on the steps in [https://fedoraproject.org/wiki/QA:Testcase_Partitioning_Rootfs_On_RAID6 QA:Testcase_Partitioning_Rootfs_On_RAID6]: https://fedoraproject.org/wiki/QA:Testcase_Partitioning_On_Software_RAID In this case, root(/) is partitioned on a RAID device, please review it. Thanks!
Looks good, thanks Hurry! Yay, no more running 6 different software RAID installs :)
Except the above raid tests, I plan to remove the following tests: [[BR]]
Reason: In real world, users upgrade directly to the release after the next should be very few, and no release criteria is defined on this.
Reason: After booting with a media, the CD-ROM would be locked, therefore, it's normally impossible to exchange the CD/DVD to add another repo. [https://fedoraproject.org/wiki/QA/TestCases/InstallSourceDvd InstallSourceDvd] and InstallSourceCdrom test installation using DVD/CD package repo.
Reason: In F14, the tests would be grouped according to install media, and default package install is included in each media installation. So I'd like to remove minimal install, since default one has already covered it.
Reason: How many people will install using ext2 filesystem?!
Reason: I think the other partitioning tests and QA/TestCases/BootMethodsKVM(Install on a new created kvm) have somehow covered this test.
What do you think? Feel free to comment if you have any ideas/suggestions/disagreements about these. Thanks.
"Reason: In real world, users upgrade directly to the release after the next should be very few"
This is not actually the case. Quite a lot of people run alternate releases, because that's the longest possible refresh cycle allowed by our maintenance policies. People who don't want to go through the hassle of doing a distro upgrade every six months run alternate releases, so they only have to do it every twelve months.
"and no release criteria is defined on this"
Indeed this is true, but I believe this test exists because we recognize that, in practice, people do run alternate releases so we should at least know about and be able to document any issues with this.
"Reason: In F14, the tests would be grouped according to install media, and default package install is included in each media installation. So I'd like to remove minimal install, since default one has already covered it."
I don't quite understand this. We test the minimal install functionality to make sure it works. Sure, all the same packages are in the default install, but that's not the same as saying the minimal installation function works as intended. Am I misreading you?
"Reason: How many people will install using ext2 filesystem?! "
Agreed, this one is outdated now. (Some people use ext2 on SSDs because they think journalling will kill the hardware, but such people are wrong. :>)
"Reason: I think the other partitioning tests and QA/TestCases/BootMethodsKVM(Install on a new created kvm) have somehow covered this test. "
This seems reasonable.
Replying to [comment:6 adamwill]:
"Reason: In F14, the tests would be grouped according to install media, and default package install is included in each media installation. So I'd like to remove minimal install, since default one has already covered it." I don't quite understand this. We test the minimal install functionality to make sure it works. Sure, all the same packages are in the default install, but that's not the same as saying the minimal installation function works as intended. Am I misreading you?
Hurry: I think we'll need to keep the minimal install test around for Fedora 14. However, I don't think we need to run this test for every usage scenario. Can both the default package set, and the minimal package set tests be moved into the general variations list? I don't think these need to be tested against each boot+install method.
Additionally, I've updated the [https://fedoraproject.org/wiki/QA/TestCases/PackageSetsMinimalPackageInstall minimal install package set test] to ensure the system is able to access the network and install yum updates.
I'd really like to also remove 'QA/TestCases/BootMethodsKVM' and it's XEN counterpart, or at least rename them to something more appropriate. Perhaps a set of storage device tests that confirm the detection and use of virt storage and network devices?
"Reason: In real world, users upgrade directly to the release after the next should be very few" This is not actually the case. Quite a lot of people run alternate releases, because that's the longest possible refresh cycle allowed by our maintenance policies. People who don't want to go through the hassle of doing a distro upgrade every six months run alternate releases, so they only have to do it every twelve months. "and no release criteria is defined on this" Indeed this is true, but I believe this test exists because we recognize that, in practice, people do run alternate releases so we should at least know about and be able to document any issues with this.
I see. If quite many people run alternative releases, I agree to keep this test, but then I hope the release criteria can reflect the priority of it rather than no priority like that in F13.
As you said, my thought was that all the packages are the same in the default install, minimal one can be removed. But you're right, that doesn't mean the minimal install works, especially when James updated this test.
"Reason: How many people will install using ext2 filesystem?! " Agreed, this one is outdated now. (Some people use ext2 on SSDs because they think journalling will kill the hardware, but such people are wrong. :>) "Reason: I think the other partitioning tests and QA/TestCases/BootMethodsKVM(Install on a new created kvm) have somehow covered this test. " This seems reasonable.
Many thanks for your comments Which make things more clear. :)
Replying to [comment:7 jlaska]:
<skip> Hurry: I think we'll need to keep the minimal install test around for Fedora 14. However, I don't think we need to run this test for every usage scenario. Can both the default package set, and the minimal package set tests be moved into the general variations list? I don't think these need to be tested against each boot+install method.
<skip>
No problem. Having default package set test in each scenario is easier for testers to add results. I don't think it's specific to any usage scenario as well, so I'll put it in general list.
Thanks to both you and Adam for clarifying this issue, let's keep minimal install test.
"Reason: I think the other partitioning tests and QA/TestCases/BootMethodsKVM(Install on a new created kvm) have somehow covered this test. " I'd really like to also remove 'QA/TestCases/BootMethodsKVM' and it's XEN counterpart, or at least rename them to something more appropriate. Perhaps a set of storage device tests that confirm the detection and use of virt storage and network devices?
I'm going to remove them and add a case [https://fedoraproject.org/wiki/QA:Testcase_Install_to_KVM QA:Testcase_Install_to_KVM] instead.Thanks.
I further updated the [https://fedoraproject.org/wiki/User:Rhe/Draft2#Test_Matrix matrix], and I'm not sure if the following tests can be removed or not:
Reason: Not commonly used and require special device to support it.
Reason: I don't think No Swap should be separated as a test.
Reason: Is it a usual way to load kickstart file? I seldom use this way.
Reason: [https://fedoraproject.org/wiki/QA:Testcase_Preupgrade_from_older_release QA:Testcase_Preupgrade_from_older_release] is already the case of low /boot disk space. And for upgrading from F13, 500M /boot is enough for preupgrading.
What do you think? Please feel free to add comments and tell me if there are other tests in the removing list.
Apologies, I missed your latest updates in this ticket.
Replying to [comment:10 rhe]:
I further updated the [https://fedoraproject.org/wiki/User:Rhe/Draft2#Test_Matrix matrix], and I'm not sure if the following tests can be removed or not: [https://fedoraproject.org/wiki/QA:Testcase_Boot_Methods_efidisk.img QA:Testcase_Boot_Methods_efidisk.img] Reason: Not commonly used and require special device to support it.
I'd like to remove this test, but I'm afraid we need to ensure this is functional since it is included in the Beta release criteria (https://fedoraproject.org/wiki/Fedora_14_Beta_Release_Criteria). I've relied on feedback from pjones in previous releases to run this test. Mainly because running this test is still a mystery to me. I agree the conditions around this test are not ideal, but I don't have any great ideas for improving this right now :(
[https://fedoraproject.org/wiki/QA/TestCases/PartitioningNoSwap QA/TestCases/PartitioningNoSwap] Reason: I don't think No Swap should be separated as a test.
This test still exists to target the dialog warning that swap is recommended. What do you recommend, should this test be absorbed into another test?
[https://fedoraproject.org/wiki/QA/TestCases/KickstartKsFilePathKsCfg QA/TestCases/KickstartKsFilePathKsCfg] Reason: Is it a usual way to load kickstart file? I seldom use this way.
This is how preupgrade and snake delivery kickstarts. They open up the initrd.img and insert the ks.cfg there. We'll need to continue supporting this.
[https://fedoraproject.org/wiki/QA:Testcase_Preupgrade_low_/boot_disk_space_to_install QA:Testcase_Preupgrade_low_/boot_disk_space_to_install] Reason: [https://fedoraproject.org/wiki/QA:Testcase_Preupgrade_from_older_release QA:Testcase_Preupgrade_from_older_release] is already the case of low /boot disk space. And for upgrading from F13, 500M /boot is enough for preupgrading.
Good point, this should be less of an issue preupgrading from F13 -> F14. The remaining preupgrade test is written to mimic a real-world upgrade scenario, so we should capture any /boot size issues as they surface.
I can't think of any others at this time. Thanks!
Replying to [comment:11 jlaska]:
Apologies, I missed your latest updates in this ticket. No worries. Replying to [comment:10 rhe]: I further updated the [https://fedoraproject.org/wiki/User:Rhe/Draft2#Test_Matrix matrix], and I'm not sure if the following tests can be removed or not: [https://fedoraproject.org/wiki/QA:Testcase_Boot_Methods_efidisk.img QA:Testcase_Boot_Methods_efidisk.img] Reason: Not commonly used and require special device to support it. I'd like to remove this test, but I'm afraid we need to ensure this is functional since it is included in the Beta release criteria (https://fedoraproject.org/wiki/Fedora_14_Beta_Release_Criteria). I've relied on feedback from pjones in previous releases to run this test. Mainly because running this test is still a mystery to me. I agree the conditions around this test are not ideal, but I don't have any great ideas for improving this right now :(
Apologies, I missed your latest updates in this ticket. No worries. Replying to [comment:10 rhe]:
I see, since it's included in beta criteria, let's keep this test.
[https://fedoraproject.org/wiki/QA/TestCases/PartitioningNoSwap QA/TestCases/PartitioningNoSwap] Reason: I don't think No Swap should be separated as a test. This test still exists to target the dialog warning that swap is recommended. What do you recommend, should this test be absorbed into another test?
Hmm, I think it depends on the importance of no swap partitioning. A possible way is to absorb this into the expected results of ext4 and ext3 partitioning tests, saying "If no swap partition is given, a warning dialogue will pop up...", but then no swap is not guaranteed to be tested. So, I think just keeping this test is fine.
[https://fedoraproject.org/wiki/QA/TestCases/KickstartKsFilePathKsCfg QA/TestCases/KickstartKsFilePathKsCfg] Reason: Is it a usual way to load kickstart file? I seldom use this way. This is how preupgrade and snake delivery kickstarts. They open up the initrd.img and insert the ks.cfg there. We'll need to continue supporting this.
I understand now.
[https://fedoraproject.org/wiki/QA:Testcase_Preupgrade_low_/boot_disk_space_to_install QA:Testcase_Preupgrade_low_/boot_disk_space_to_install] Reason: [https://fedoraproject.org/wiki/QA:Testcase_Preupgrade_from_older_release QA:Testcase_Preupgrade_from_older_release] is already the case of low /boot disk space. And for upgrading from F13, 500M /boot is enough for preupgrading. Good point, this should be less of an issue preupgrading from F13 -> F14. The remaining preupgrade test is written to mimic a real-world upgrade scenario, so we should capture any /boot size issues as they surface.
Okay, I'll remove [https://fedoraproject.org/wiki/QA:Testcase_Preupgrade_low_/boot_disk_space_to_install QA:Testcase_Preupgrade_low_/boot_disk_space_to_install]
Thank you all. I'm closing this ticket. Feel free to reopen in need.
Metadata Update from @adamwill: - Issue untagged with: test review - Issue tagged with: test cases
Login to comment on this ticket.