From 9787682a20caf164217a631ba4d585f464e0b39d Mon Sep 17 00:00:00 2001 From: Adam Williamson Date: Aug 12 2025 13:35:01 +0000 Subject: [PATCH 1/2] sop_mass_branching: add a note to update fedfind's release metadata This is important...I usually do it, but this SOP is the natural place for it. Signed-off-by: Adam Williamson --- diff --git a/modules/release_guide/pages/sop_mass_branching.adoc b/modules/release_guide/pages/sop_mass_branching.adoc index d180699..be4cbbd 100644 --- a/modules/release_guide/pages/sop_mass_branching.adoc +++ b/modules/release_guide/pages/sop_mass_branching.adoc @@ -651,6 +651,16 @@ https://pagure.io/fedora-infrastructure/issue/9239#comment-671446[fedora infra ticket] to update the database of mirrormanager, though note that none of the 'modular' repos exist any more. +=== Updating fedfind release metadata + +We need to update the `/srv/groups/qa/metadata/release.json` file on +fedorapeople (people01). It is used by fedfind as a source of truth +about release numbers. openQA, fedora-image-uploader and various other +will behave incorrectly if it is not updated. The new branched release +number should be added to the `branched` list in this file - before +the edit, the list should be empty. If you do not have access to edit +this file, ask a member of the Quality team to do so. + === Enable autosigning on branched release Once the branched compose is composed, we need to re-enable From 74b9ca9f23f0c7ee89853b593e5e37db341eda89 Mon Sep 17 00:00:00 2001 From: Adam Williamson Date: Aug 12 2025 13:35:01 +0000 Subject: [PATCH 2/2] sysadmin guide: Update openQA with branching instructions I tried to document all the faffing around we usually have to do with openQA at branching time, since I'm going to be on PTO when it happens this cycle. Signed-off-by: Adam Williamson --- diff --git a/modules/sysadmin_guide/pages/openqa.adoc b/modules/sysadmin_guide/pages/openqa.adoc index 121dff7..9aa7bbd 100644 --- a/modules/sysadmin_guide/pages/openqa.adoc +++ b/modules/sysadmin_guide/pages/openqa.adoc @@ -54,13 +54,13 @@ matching), and test 'assets' like ISO files and disk images. The path is `/var/lib/openqa/share/factory`. In our deployment, the PostgreSQL database for each instance is hosted -by the QA database server. Also, some paths on the server are themselves +by db-openqa01. Also, some paths on the server are themselves mounted as NFS shares from the infra storage server. This is so that these are not lost if the server is re-deployed, and can easily be backed up. These locations contain the data from each executed job. As both the database and these key data files are not actually stored on the server, the server can be redeployed from scratch without loss of -any data (at least, this is the intent). +any data. We've done this successfully several times. Also in our deployment, an openQA plugin (which we wrote, but which is part of the upstream codebase) is enabled which publishes messages on @@ -165,6 +165,114 @@ and click on 'Workers', which will show the status of all known workers for that server, and select 'Working' in the state filter box. This will show all workers currently working on a job. +== Creating needles + +openQA has a "Developer mode" for creating new needles. You need to be +an openQA admin or operator to use it. It's usually best to do this +on the staging instance. On a running test, go to the Live View tab. +You should see a "Developer mode" box above the test video. Click it, +then change "Pause on screen mismatch" to "assert_screen timeout" and +click "Confirm to control this test". Now, the next time a screen match +assertion fails, the test will pause and a button to open the needle +editor will be shown. On the needle editor screen you can select an +existing needle to use as a base at the top right. It's easiest if an +existing needle is *nearly* matched - in this case you can probably +just select it and hit Save. Otherwise, it's still a good idea to pick +an existing needle to start from, as it will at least give you the tags +and filename of that needle, which are probably a good base. If you +don't need to change the tags, just change the match areas (unless +they're already correct), tweak the filename, and hit Save. + +The saved needle will be present on the server in +`/var/lib/openqa/share/tests/fedora/needles`. You should copy it off +the server and into your local system, in a checkout of the +os-autoinst-distri-fedora repository, then place it in the appropriate +subdirectory of `needles/` - probably the same one as existing needles +of the same type. Create a commit with the needle. If you have commit +rights you can push it out directly; if you don't, or you want someone +to check your work, you can create a pull request in the usual way. +Once the needle is merged, you can go back to the server, do a +`git pull`, and remove the copy of the needle that's directly in the +`needles/` directory. Remember to also pull the needle onto the other +server instance! If the staging instance is not on the main branch, +rebase whatever branch it's on (from your local system) then do +`git fetch origin` then `git reset --hard origin/` on staging. + +== Handling Fedora branch events + +Branching of a new release is quite disruptive to openQA and requires +some manual handling. It is best to collaborate closely with the +release engineering team on this. + +In general, tests for the newly-branched release and Rawhide may have +issues until a first Branched compose, and a first post-branching +compose of Rawhide, are both done and synced to mirrors, mirror +manager's configuration is correctly updated, and the relval release +metadata is updated. These steps are all covered in the +xref:release_guide:sop_mass_branching.adoc[mass brancing guide], but +be aware they need to happen and work with releng to monitor them. You +may need to edit the relval release metadata if the release engineer +does not have access to it. + +Until the relval metadata is updated, openQA will believe the to-be- +branched release number is Rawhide, and any updates for the new Rawhide +release number (one number higher) will confuse it. Once the metadata +is updated, openQA will believe the to-be-branched release number is +Branched and the new Rawhide release number is Rawhide; if the change +is made too early, tests of updates for the new Branched number may +fail due to the expected repositories not existing, etc. There is no +perfect time to update the metadata, so just be aware that you may see +odd behavior related to this during the branching process. + +At the time a test is scheduled, several release-number-related +variables are set based on info from fedfind, especially `RAWREL`, +which is set to whatever number relval currently believes is Rawhide. +If this value is wrong, various tests will fail. Watch out for a +situation where tests still have RAWREL set to the old Rawhide, now +Branched, number *after* branching is complete. If this happens, you +need to *re-schedule* the tests, not just re-run them, because +re-running the tests does not change the variables. Run: +``` +fedora-openqa update -f +``` +to re-schedule. + +If tests are failing for reasons that look related to mirroring, ask +releng to look at the problems and fix the mirror config. + +As part of branching, new fedora-repos and fedora-release packages must +be built, but there may be a catch-22 situation where they fail tests, +but we need them pushed stable in order to proceed with branching and +get back to a point where tests will pass. If that happens, just waive +the failures from Bodhi to let the update go through. + +The desktop_background test will fail for the newly-branched release +because there will be no needles for it yet (we do not test the +background on Rawhide). If the backgrounds for the new release are +already present and working, just create a new needle in needles/background +and commit it. If the new backgrounds are not ready yet, you can either +temporarily disable gating for the test - comment out the line for it +in infra ansible roles/openshift-apps/greenwave/templates/fedora.yaml.j2 +and re-run the relevant plays (you may need help from someone with +sysadmin-main access to do this) - or temporarily 'short-circuit' the test +for the Branched release in the same way as it is short-circuited for +Rawhide, so it (incorrectly) passes. + +Install tests on Rawhide updates will fail until new `version_NN_ident` +needles are created - there is a check that the installer correctly +shows the release number. Find a test failing on `version_NN_ident` +and create the needle, then commit it. + +Throughout the process, keep one Branched update and one Rawhide update +handy to use as 'canaries' - keep re-running the tests on them until +they all pass. Once you have all the tests passing for a release, +re-run (or re-schedule, if necessary) all failed tests on other updates +for the same release. It is important to ensure there are no failures +for any update (unless, of course, it's a "real" failure, not a result +of branching). Note that the ostree tests aren't gating, so it's most +important to fix the other tests first, you can fix ostree tests a bit +slower if necessary. + == Troubleshooting === New tests not being scheduled