#151 Tests consistent with Criterion
Closed: Fixed None Opened 13 years ago by rhe.

= problem =

Criteria was kept changing during F-14 test cycle, but some installation tests didn't update accordingly. On the other hand, the criterion should be modified to include preupgrade_from_older_release test and new tests created for F-15.

= enhancement recommendation =

Installation tests need be reviewed and updated to fit for the criterion. Criterion for preupgrade_from_older_release test is required.


This is still valid, we have several criteria for which we do not have matching tests, on both installation and 'desktop' side (we may want a third matrix for things that aren't really either). Let's work on this for F16.

I have done a survey of the alpha criteria, checking for associated tests. Here are the results. 'partial' indicates that relevant tests are present but do not entirely cover the criterion's requirements. 'not-in-matrix' indicates a test case is written but not included in the current test matrices. 'NO TEST' indicates that, as far as I could find, there is no test case written for the criterion. Following these keywords, I have listed the relevant test cases for each criterion (where applicable).

We can open subsidiary tickets to this bug for each specific issue identified here.

There must be no file conflicts (cases where the files in some packages conflict but the packages have explicit Conflicts: tags are acceptable) or unresolved package dependencies during a media-based (CD/DVD) install

        https://fedoraproject.org/wiki/QA:Testcase_Mediakit_FileConflicts

All dedicated installer images must boot to the graphical boot menu and allow the user to select install options. If no option is selected, the installer should load after a reasonable timeout

partial     https://fedoraproject.org/wiki/QA/TestCases/BootMethodsBootIso
        https://fedoraproject.org/wiki/QA/TestCases/BootMethodsDvd
        https://fedoraproject.org/wiki/QA:Testcase_Boot_Methods_efidisk.img

The installer must boot (if appropriate) and run on all primary architectures from default live image, DVD, and boot.iso install media

not-in-matrix   https://fedoraproject.org/wiki/QA:Testcase_Live_Image_Boot
        https://fedoraproject.org/wiki/QA/TestCases/BootMethodsBootIso
        https://fedoraproject.org/wiki/QA/TestCases/BootMethodsDvd

The graphical boot menu for all installation images should include an entry which causes both installation and the installed system to use a generic, highly compatible video driver (such as 'vesa'). This mechanism should work correctly, launching the installer and attempting to use the generic driver

NO TEST

The installer must be able to use at least one of the HTTP or FTP remote package source options

        https://fedoraproject.org/wiki/QA:Testcase_Http_Repository (redundant with ftp_repository?)
        https://fedoraproject.org/wiki/QA:Testcase_Additional_Http_Repository
        https://fedoraproject.org/wiki/QA:Testcase_Additional_Ftp_Repository
        https://fedoraproject.org/wiki/QA:Testcase_Ftp_Repository

The installer must be able to use the DVD local package source options

        https://fedoraproject.org/wiki/QA/TestCases/InstallSourceDvd

The installer must be able to complete an installation using the text, graphical and VNC installation interfaces

        https://fedoraproject.org/wiki/QA/TestCases/InstallSourceDvd
        https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_Text
        https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_VNC

The installer must be able to complete package installation with the default package set for each supported installation method

        https://fedoraproject.org/wiki/QA/TestCases/InstallSourceDvd
        https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_Text
        https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_VNC

The installer must be able to complete an installation using IDE, SATA and SCSI storage devices, with the default file system and LVM

        https://fedoraproject.org/wiki/QA/TestCases/PartitioningExt4OnNativeDevice
        https://fedoraproject.org/wiki/QA:Testcase_install_to_SATA_device
        https://fedoraproject.org/wiki/QA:Testcase_install_to_SCSI_device
        https://fedoraproject.org/wiki/QA:Testcase_install_to_PATA_device

The installer must be able to complete an installation using the entire disk, existing free space, or existing Linux partitions methods, with or without encryption enabled

partial     https://fedoraproject.org/wiki/QA:Testcase_Anaconda_autopart_install
        https://fedoraproject.org/wiki/QA:Testcase_Anaconda_autopart_%28use_free_space%29_install
        https://fedoraproject.org/wiki/QA:Testcase_Anaconda_autopart_%28encrypted%29_install
(existing partitions method does not seem to be tested)

The rescue mode of the installer must start successfully and be able to detect and mount an existing default installation

        https://fedoraproject.org/wiki/QA:Testcase_Anaconda_rescue_mode

The installer must be able to report failures to Bugzilla, with appropriate information included

        https://fedoraproject.org/wiki/QA:Testcase_Anaconda_save_traceback_to_bugzilla

In most cases (see Blocker_Bug_FAQ), a system installed according to any of the above criteria (or the appropriate Beta or Final criteria, when applying this criterion to those releases) must boot to the 'firstboot' utility on the first boot after installation, without unintended user intervention. This includes correctly accessing any encrypted partitions when the correct passphrase is supplied. The firstboot utility must be able to create a working user account

NO TEST

Following on from the previous criterion, after firstboot is completed and on subsequent boots, a system installed according to any of the above criteria (or the appropriate Beta or Final criteria, when applying this criterion to those releases) must boot to a working graphical environment without unintended user intervention. This includes correctly accessing any encrypted partitions when the correct passphrase is supplied

NO TEST (partly implied by more sophisticated desktop tests)

When booting a system installed without a graphical environment, or when using a correct configuration setting to cause an installed system to boot in non-graphical mode, the system should boot to a state where it is possible to log in through at least one of the default virtual consoles

partial     https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_Text

It must be possible to run the default web browser and a terminal application from all release-blocking desktop environments. The web browser must be able to download files, load extensions, and log into FAS

        https://fedoraproject.org/wiki/QA:Testcase_desktop_browser

The installed system must be able to download and install updates with yum and the default graphical package manager in all release-blocking desktops

        https://fedoraproject.org/wiki/QA:Testcase_desktop_updates

The default Fedora artwork must either refer to the current Fedora release under development (Fedora 16), or reference an interim release milestone (e.g. Alpha or Beta). If a release version number is used, it must match the current Fedora release under development. This includes artwork used in the installer, firstboot, graphical boot, graphical login and desktop background.

NO TEST

A system logging infrastructure must be available and enabled by default. It must provide at least basic local file-based logging of kernel messages, and allow other components to write log messages. This must be done in accordance with relevant standards accepted by the Project

NO TEST

For "All dedicated installer images must boot to the graphical boot menu and allow the user to select install options. If no option is selected, the installer should load after a reasonable timeout": I have updated the relevant test cases to specify that navigating the boot menu must work. that resolves the 'partial' issue for this criterion.

For "The graphical boot menu for all installation images should include an entry which causes both installation and the installed system to use a generic, highly compatible video driver (such as 'vesa'). This mechanism should work correctly, launching the installer and attempting to use the generic driver" - filed ticket #200: https://fedorahosted.org/fedora-qa/ticket/200

For "The installer must be able to complete an installation using the entire disk, existing free space, or existing Linux partitions methods, with or without encryption enabled", filed ticket #201 to add an 'existing Linux partitions' test.

For "In most cases (see Blocker_Bug_FAQ), a system installed according to any of the above criteria (or the appropriate Beta or Final criteria, when applying this criterion to those releases) must boot to the 'firstboot' utility on the first boot after installation, without unintended user intervention. This includes correctly accessing any encrypted partitions when the correct passphrase is supplied. The firstboot utility must be able to create a working user account": filed #202 to request a firstboot test case.

For "Following on from the previous criterion, after firstboot is completed and on subsequent boots, a system installed according to any of the above criteria (or the appropriate Beta or Final criteria, when applying this criterion to those releases) must boot to a working graphical environment without unintended user intervention. This includes correctly accessing any encrypted partitions when the correct passphrase is supplied": filed #203 to request a test case.

For "When booting a system installed without a graphical environment, or when using a correct configuration setting to cause an installed system to boot in non-graphical mode, the system should boot to a state where it is possible to log in through at least one of the default virtual consoles": I updated https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_Text to better reflect the relevant release criteria.

For "The default Fedora artwork must either refer to the current Fedora release under development (Fedora 16), or reference an interim release milestone (e.g. Alpha or Beta). If a release version number is used, it must match the current Fedora release under development. This includes artwork used in the installer, firstboot, graphical boot, graphical login and desktop background. " (and corresponding Final criterion): filed #204 to request test cases.

For "A system logging infrastructure must be available and enabled by default. It must provide at least basic local file-based logging of kernel messages, and allow other components to write log messages. This must be done in accordance with relevant standards accepted by the Project": filed #205 to request a test case.

Many thanks for your survey, Adam! I've read it carefully and updated sub-tickets accordingly. Please note that the install source and repository tests in the [https://fedoraproject.org/wiki/QA:Fedora_15_Install_Results_Template F-15 install template] have been recategorized and updated to [https://fedoraproject.org/wiki/Category:Installation_Repository install repository category] in [https://fedoraproject.org/wiki/QA:Fedora_16_Install_Results_Template F-16 install template] to cover the repo=, askmethod, and graphically repo adding methods(see ticket#173 ). It wouldn't reduce the test coverage expect that the HTTP and FTP were included in one test now.

update for criterion "The graphical boot menu for all installation images should include an entry which causes both installation and the installed system to use a generic, highly compatible video driver (such as 'vesa'). This mechanism should work correctly, launching the installer and attempting to use the generic driver": rhe points out that there is in fact a test in the install matrix, https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_Basic_Video_Driver .

Updates: created test cases for 'firstboot' and 'initial startup' criteria (see tickets #202 and #203). I also re-considered merging the text install, firstboot and startup tests; they should really be separate, so the new test cases cover both text and graphic cases, and the installer tests have been restricted to considering installer functions only.

I've put the new test cases in a new 'base' category (per their page names, anyway) to reflect that I think we're going to need a new matrix for tests that enforce criteria that aren't really installer or desktop related exactly: firstboot, system logging, startup and so on.

Here are the results of the Beta survey. Will follow up with tickets for the action areas soon.

The network installation image, DVD image, and live images for release-blocking desktops must meet current size requirements

    https://fedoraproject.org/wiki/QA:Testcase_Mediakit_ISO_Size

The installer must boot and run on systems using EFI other than Apple Macs

    https://fedoraproject.org/wiki/QA:Testcase_Boot_Methods_efidisk.img (and all boot_methods tests)

The installer must be able to use the HTTP, FTP and NFS remote package source options

    https://fedoraproject.org/wiki/QA:Testcase_install_repository_NFS_graphical (others are Alpha)

At the package installation stage, the install should show no uncategorized package groups

NO TEST

The installer must be able to use all kickstart delivery methods

        https://fedoraproject.org/wiki/QA:Testcase_Kickstart_Http_Server_Ks_Cfg
        https://fedoraproject.org/wiki/QA:Testcase_Kickstart_File_Path_Ks_Cfg
        https://fedoraproject.org/wiki/QA:Testcase_Kickstart_Hd_Device_Path_Ks_Cfg
        https://fedoraproject.org/wiki/QA:Testcase_Kickstart_Nfs_Server_Path_Ks_Cfg

The installer must be able to create and install to software, hardware or BIOS RAID-0, RAID-1 or RAID-5 partitions for anything except /boot

        https://fedoraproject.org/wiki/QA:Testcase_Partitioning_On_Software_RAID
        https://fedoraproject.org/wiki/QA:Testcase_Install_to_BIOS_RAID
        https://fedoraproject.org/wiki/QA:Testcase_Install_to_Hardware_RAID

The installer must be able to successfully complete an upgrade installation from a clean, fully updated default installation (from any official install medium) of the previous stable Fedora release, either via preupgrade or by booting to the installer manually. The upgraded system must meet all release criteria

        https://fedoraproject.org/wiki/QA:Testcase_Anaconda_Upgrade_New_Bootloader
        https://fedoraproject.org/wiki/QA:Testcase_Anaconda_Upgrade_Skip_Bootloader
        https://fedoraproject.org/wiki/QA:Testcase_Anaconda_Upgrade_Update_Bootloader
        https://fedoraproject.org/wiki/QA:Testcase_Anaconda_Upgrade_Encrypted_Root
        https://fedoraproject.org/wiki/QA:Testcase_Preupgrade

Any installation method or process designed to run unattended must do so (there should be no prompts requiring user intervention)

NO TEST

The rescue mode of the installer must be able to detect and mount (read-write and read-only) LVM, encrypted, and RAID (BIOS, hardware, and software) installations

partial     https://fedoraproject.org/wiki/QA:Testcase_Anaconda_rescue_mode

When booting a system installed without a graphical environment, or when using a correct configuration setting to cause an installed system to boot in non-graphical mode, the system should provide a working login prompt without any unintended user intervention when boot is complete, and all virtual consoles intended to provide a working login prompt should do so

not-in-matrix   https://fedoraproject.org/wiki/QA:Testcase_base_startup

The release must boot successfully as a virtual guest in a situation where the virtual host is running the same release (using Fedora's current preferred virtualization technology)

NO TEST (explicit; in practice, much testing is done in virt environment)

The release must boot successfully as a virtual guest in a situation where the virtual host is running the previous stable Fedora release (using Fedora's current preferred virtualization technology)

NO TEST

The release must boot successfully as a virtual guest in a situation where the virtual host is running a supported Xen implementation

NO TEST

In most cases, the installed system must be able to play back sound with gstreamer-based applications (see Blocker_Bug_FAQ)

        http://fedoraproject.org/wiki/QA:Testcase_audio_basic

No part of any release-blocking desktop's panel (or equivalent) configuration should crash or be entirely non-functional on boot of the installed system using default installation choices

        http://fedoraproject.org/wiki/QA:Testcase_desktop_panel_basic

Automatic mounting on insertion of removable media must work in release-blocking desktops

        http://fedoraproject.org/wiki/QA:Testcase_desktop_automount

The default update manager in release-blocking desktops must not periodically check for updates when the system is booted live, but must periodically check for updates when running on an installed system

        http://fedoraproject.org/wiki/QA:Testcase_desktop_updates

All release-blocking desktops' offered mechanisms (if any) for shutting down, logging out and rebooting must work

        http://fedoraproject.org/wiki/QA:Testcase_desktop_login

Created tickets for all the deficiencies listed above (except the not-in-matrix Testcase_base_startup - I'm handling the proposed Base matrix through the list):

https://fedorahosted.org/fedora-qa/ticket/216
https://fedorahosted.org/fedora-qa/ticket/217
https://fedorahosted.org/fedora-qa/ticket/218
https://fedorahosted.org/fedora-qa/ticket/219

all are currently assigned to rhe, but if anyone wants to pitch in and help out, please do!

Petr, the work remaining here is:

  • Ensure there are test cases in the matrices for all Final criteria
  • Reverse concordance: go through the set of all test cases and make sure all the ones that are listed as 'Alpha', 'Beta' or 'Final' are actually associated with a criterion from the appropriate level. If there's no criterion that matches the test case, then decide whether it makes more sense to create a criterion, or to adjust the 'importance level' of the test case (or drop it altogether).

Please let me know if you have any problems or questions!

er, by 'set of all test cases', I mean all release validation test cases - all the test cases that show up on the Install, Desktop and Base validation matrices.

Petr, I think we got a long way along with this. Would you say we completed it? I don't recall if we've finished the reverse concordance.

I don't think we can complete this any time. Criteria are changing quite quickly. Changes I did before Fedora 17 pre-release testing:

I added new test case for uncovered criterion:
"All services in a default install must start properly" -> https://fedoraproject.org/wiki/QA:Testcase_Services_start (there is a problem with this, which we should discuss: http://lists.fedoraproject.org/pipermail/test/2012-February/105685.html)

I removed http://lists.fedoraproject.org/pipermail/test/2012-February/105685.html test case from installation matrix.

I changed release level of https://fedoraproject.org/wiki/QA:Testcase_Kickstart_Http_Server_Ks_Cfg and https://fedoraproject.org/wiki/QA:Testcase_Mediakit_ISO_Size to beta

I changed release level of https://fedoraproject.org/wiki/QA:Testcase_Anaconda_traceback_debug_mode to non-blocking, because debug mode is not supposed to be used by users.

I changed release level of https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_serial_console to Beta and level of https://fedoraproject.org/wiki/QA:Testcase_Anaconda_User_Interface_Cmdline to Final.

I added new alpha criterion: "A correct checksum must be published for each official release image." and final criterion "If there is embedded checksum on ISO media, it must be correct." and I set release level of https://fedoraproject.org/wiki/QA:Testcase_Mediakit_ISO_Checksums to Alpha / Final

I added new alpha criteria:
* "The installer must be able to download and use an [[Anaconda/Updates| installer update image]] from an HTTP server"

I added new beta criteria:
"The installer must be able to use an [[Anaconda/Updates|installer update image]] retrieved from removable media, remote installation source and HTTP url" - this one was lately moved to the final"
"The installer must be able to complete an installation using the serial console interface"

I added new final criteria:
"All release media must include a standalone memory test utility. A boot menu option to launch this utility must be present and must work correctly."
"The installer must be able to complete an installation using all supported interfaces"

I changed criterion "The installer must be able to report failures to Bugzilla, with appropriate information included" to
"The installer must be able to report failures to Bugzilla and local disk, with appropriate information included" and I changed release level of https://fedoraproject.org/wiki/QA:Testcase_Anaconda_save_traceback_to_remote_system to non-blocking

All those changes were made after discussion on test mailing list.

Because there have been added new criteria since I went through this I propose to revise it again before Fedora 18 pre-release testing. I know about some unclosed topics (http://lists.fedoraproject.org/pipermail/test/2012-February/105468.html http://lists.fedoraproject.org/pipermail/test/2012-January/105260.html). I'm not sure if I should open new ticket or only change milestone to this one, what do you think Adam?

Also, when we will be accepting a new criterion we should probably demand test case for testing feature from this criterion and reverse approach - if we create new test case and add release level to it, we should make sure that it's covered by some criterion.

pschindl: I think it's probably best to open a new ticket, I don't think everlasting tickets are great, it gets messy and doesn't allow us to track what we've achieved in the past. I agree it would probably be a good idea to require newly-proposed validation tests and criteria to match as you suggest.

You have an entry on the retrospective that's broadly about this area, I know - I was already planning to file a new ticket as a follow-up for the retrospective.

There are separate tickets for newer Fedora.

Login to comment on this ticket.

Metadata