#371 Update Final Release SOP
Opened a month ago by patrikp. Modified a month ago
patrikp/infra-docs-fpo sop_update  into  master

@@ -11,20 +11,31 @@ 

  

  == Bodhi Changes

  

- Set the bodhi release to `current`

+ Set the Bodhi release to `current`.

+ 

+ [NOTE]

+ .Note

+ ====

+ Ensure your Bodhi configuration is properly set up.

+ `sysadmin-bodhi` FAS group membership is required.

+ ====

+ 

  [source,subs="attributes+"]

  ....

  $ bodhi releases edit --name F{branched} --state current

  $ bodhi releases edit --name F{branched}F --state current

  ....

  

- Set the bodhi stable tag to point to updates instead of base repo

+ Set the Bodhi stable tag to point to updates instead of base repo.

+ 

  [source,subs="attributes+"]

  ....

  $ bodhi releases edit --name F{branched} --stable-tag f{branched}-updates

  ....

  

- Set EOL of oldest release to correct date: four weeks after the official release date of {branched}.

+ Set EOL of oldest release to correct date: four weeks after the official release

+ date of {branched}.

+ 

  [source,subs="attributes+"]

  ....

  $ bodhi releases edit --name F{old_release} --eol YYYY-MM-DD
@@ -32,18 +43,26 @@ 

  $ bodhi releases edit --name F{old_release}F --eol YYYY-MM-DD

  ....

  

- If this date is different from the one in the https://fedorapeople.org/groups/schedule/f-{old_release}/f-{old_release}-key-tasks.html[Fedora schedule],

+ If this date is different from the one in the

+ https://fedorapeople.org/groups/schedule/f-{old_release}/f-{old_release}-key-tasks.html[Fedora schedule],

  ask the program manager to update the date in the schedule.

- If it is different from the one in F{old_release}'s `/etc/os-release`, https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora&format&component=fedora-release&version=40[file a bug against fedora-release] asking for it to be updated.

+ If it is different from the one in F{old_release}'s `/etc/os-release`,

+ https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora&format&component=fedora-release&version=40[file a bug against fedora-release]

+ asking for it to be updated.

  

- You may also wish to check the current EOL estimates for other releases on Bodhi, the schedule, and fedora-release,

- and make sure they are in sync and sensible.

+ You may also wish to check the current EOL estimates for other releases on

+ Bodhi, the schedule, and `fedora-release`, and make sure they are in sync and

+ sensible.

  

  == Atomic desktops changes

  

- Run the releng scripts/update_ostree_refs.sh on a compose machine with /mnt/koji/ mounted.

+ Run the `releng/scripts/update_ostree_refs.sh` script on a compose machine with

+ `/mnt/koji/` mounted.

  

- $ scripts/update_ostree_refs.sh {branched}

+ [source,subs="attributes+"]

+ ....

+ $ ./update_ostree_refs.sh {branched}

+ ....

  

  This updates the refs for Atomic desktops to have users follow updates

  for the new release rather than being stuck on the GA compose forever.
@@ -73,40 +92,138 @@ 

  $ sudo rbac-playbook groups/openqa.yml -t testcase_stats

  ....

  

- If you do not have permissions to run a playbook, ask a sysadmin-main member for help.

- A sysadmin-qa member can run the openQA playbook.

+ If you do not have permissions to run a playbook, ask a `sysadmin-main` member for help.

+ A `sysadmin-qa` member can run the openQA playbook.

  

- For reference, see the https://pagure.io/fedora-infra/ansible/c/7e501dcb9432885c9ef10e78c0418b0d40c85061[Fedora 41 release PR],

- though note it is somewhat old and includes some things that are no longer needed, and leaves out some that should be done.

+ For reference, see the https://pagure.io/fedora-infra/ansible/pull-request/2567[Fedora 42 release PR].

  

- == Stage Final Release for Mirrors

+ == Staging Final Release to Mirrors

  

- To prepare for running the staging script, make sure you have the following information:

+ Staging the Final Release consists of two steps:

  

- . Release Version: This is the numerical version number of the release, for example, {branched}.

- . ComposeID: The ID associated with the Compose, such as "Fedora-{branched}-20160614.0".

- . Label: The label used for the location in stage, for example, "Compsoe label for the location in stage 39_RC-1.2."

- . Key: The name of the release key, which can be "fedora-{branched}" or "fedora-{branched}-secondary," as examples.

- . Prerelease: Set this to 0 or 1 to determine if the release should be placed in a testing environment or not.

- . Arch (Optional): For secondary architectures, this parameter can be used to make changes to some internal locations.

- +

- [source,subs="attributes+"]

+ . Signing the checksums

+ . Staging the release

+ 

+ [NOTE]

+ .Note

+ ====

+ The signing script requires the person running it to have

+ https://docs.fedoraproject.org/en-US/infra/releng_misc_guide/sop_sigul_client_setup/[Sigul]

+ set up.

+ 

+ The staging script can be run on any compose machine where `/pub` is mounted with `rw` permissions.

+ ====

+ 

+ For example:

  ....

- $ scripts/stage-release.sh {branched} Fedora-{branched}-20160614.0 {branched}_RC-1.2 fedora-{branched} 0

+ $ ssh bodhi-backend01.iad2.fedoraproject.org

  ....

- . Sync the release to the Red Hat internal archive following internally

- documented

+ 

+ === Signing the checksums

+ 

+ Run the `releng/scripts/stage-release-checksum-sign.sh` script.

+ 

+ [NOTE]

+ .Note

+ ====

+ This script should be run as the release engineer, not as root.

+ ====

  

  [NOTE]

  ====

  Make sure to grab the directory size usage numbers which is used to send

- an email to [.title-ref]#mirror-admin@lists.fedoraproject.org# list.

+ an email to the `mirror-admin@lists.fedoraproject.org` mailing list.

  ====

  

- == Make sure there are no tagging leftovers from bodhi

+ For example:

+ 

+ [source,subs="attributes+"]

+ ....

+ $ ./stage-release-checksum-sign.sh {branched} Fedora-{branched}-20250411.n.0 fedora-{branched} 0

+ ....

+ 

+ Where:

+ 

+ . `{branched}` is the `Release Version`.

+ . `Fedora-{branched}-20250411.n.0` is the `Compose ID`. It can be found https://kojipkgs.fedoraproject.org/compose/{branched}/[here].

+ . `fedora-{branched}` is the `Signing Key`.

+ . `0` is the `Pre-release`. `1` places the release in a testing environment, `0` does not.

+ 

+ === Staging the release

+ 

+ Run the `releng/scripts/stage.sh` script.

+ 

+ [NOTE]

+ .Note

+ ====

+ This script should be run as the `ftpsync` user.

+ ====

+ 

+ For example:

+ 

+ [source,subs="attributes+"]

+ ....

+ $ sudo -u ftpsync scripts/stage-release.sh {branched} Fedora-{branched}-20250411.n.0 {branched}_RC-1.1 fedora-{branched} 0

+ ....

+ 

+ `{branched}_RC-1.1` is the `Label`.

+ 

+ == Syncing the release to the Red Hat internal archive

+ 

+ This is documented internally.

+ 

+ == Making sure there are no tagging issues in Bodhi

+ 

+ Builds should not be tagged in both `f{branched}` and `f{branched}-updates-testing` at once.

+ Any builds that are in both tags should be untagged from `f{branched}-updates-testing`.

+ 

+ Get the list of builds that are tagged in `f{branched}`.

  

- Sometimes builds are not properly tagged during the freeze process.

- Make sure all builds are properly tagged, for more info about the [issue](https://pagure.io/releng/issue/11775)

+ [source,subs="attributes+"]

+ ....

+ $ koji list-tagged --quiet f{branched} > f{branched}.txt

+ ....

+ 

+ Get the list of builds that are tagged in `f{branched}-updates-testing`.

+ 

+ [source,subs="attributes+"]

+ ....

+ $ koji list-tagged --quiet f{branched}-updates-testing > updates-testing.txt

+ ....

+ 

+ Compare which builds match and put it in a file.

+ 

+ ....

+ $ grep -Fxf <(awk '{print $1}' f42.txt) <(awk '{print $1}' updates-testing.txt) > tagged-in-both.txt

+ ....

+ 

+ We now have a text file (`tagged-in-both.txt`) with only the names of the builds, one word per line.

+ 

+ Use a simple while loop and the `koji untag-build f{branched}-updates-testing <BUILD>`

+ utility to untag all of the builds.

+ 

+ This may take several hours.

+ 

+ If any of the untagging fails because of permission issues the Koji `--force`

+ option can be used.

+ 

+ After the script finishes repeat the process above (comparing matches in two

+ files) to make sure that no builds tagged in `f{branched}` are also tagged in

+ `f{branched}-updates-testing.`

+ 

+ == Pushing the updates to the stable repository

+ 

+ Updates are usually pushed to the stable repository as part of a cronjob that runs during the weekend.

+ Stage Release Day (as opposed to the official Release Day on Tuesdays) usually falls on a Friday.

+ If we want to avoid weekend hiccups we can push the updates to the stable repository manually.

+ 

+ For example:

+ 

+ ....

+ $ sudo -u apache bodhi-push --username <username> --releases F42 --request stable

+ 

+ name:  f42-updates           success:  True

+ ....

  

  == Verification

  
@@ -115,8 +232,52 @@ 

  Infrastructure team to give the tree a second set of eyes.

  

  Some things to check for:

- . are the CHECKSUM files signed?

- . are the spins/labs present?

+ 

+ . Are the CHECKSUM files signed?

+ . Are the spins/labs present?

+ 

+ == One day before Release Day (Monday)

+ 

+ . Check that https://docs.fedoraproject.org/en-US/infra/release_guide/torrentrelease/[torrents] are published and seeding properly.

+ 

+ [NOTE]

+ .Note

+ ====

+ We want to have the torrents up in advance so people can download the content to seed it to other users.

+ We do not typically announce this widely, just to those people who normally seed torrents.

+ ====

+ 

+ [start=2]

+ . Make the directories that were hidden as part of the https://pagure.io/releng/blob/main/f/scripts/stage-release.sh[stage-release.sh] script unhidden.

+ 

+ [NOTE]

+ .Note

+ ====

+ These directories had their permissions set to `750` and now they should be changed to `755`.

+ The directories in question can be found here:

+ 

+ `/pub/fedora/linux/releases/`

+ 

+ `/pub/alt/releases/`

+ 

+ `/pub/fedora-secondary/releases/`

+ 

+ This step ensures that the content officially becomes public and everything is accessible after the release GO decision.

+ ====

+ 

+ [NOTE]

+ .Note

+ ====

+ This should be done late on Monday night, before the release.

+ This is to allow mirrors that are not tier 1 and do not have access to pre-release content to start syncing up

+ so that more of them are in sync by the time of the actual release announcement.

+ We typically do not announce this widely either.

+ Mirrors will just start syncing it, we don't need to tell users about it.

+ ====

+ 

+ == Release Day (Tuesday @ 14:00 UTC)

+ 

+ ?

  

  == Post Release

  

Looks good to me. In the beginning, there are some indentation changes that will render a similar result, I guess, but it's good to maintain the clarity of the document.

Otherwise, the changes seem right. There was an introduction of a checksum file and the stag-release file, which was actually the splitter script of the latter.

Additional Notes

Just make sure to include the following in your changes (some are already there, but I wanted to ensure everything is covered):

  • For this process, Sigul setup is required.
  • The respective script should be run with the correct mount and on a machine that has Sigul configured.
  • For the Bodhi commands:
  • Ensure your Bodhi client configuration is properly set up.
  • You need the right ACLs (I think sysadmin-bodhi will do the trick).

Verification

At the end, verification is done to check for the tag. For example:

name: f42-updates
success: True

Make sure to check if there are any pending updates to the section. This might be misleading since we also do a final push. It is generally run as part of cron later, but if someone wants to avoid weekend hiccups, they should do it manually on stage release day, which often falls on Friday.

Add one more section:

Release Day (Tuesday @ 14:00 UTC)

On release day (usually Tuesday, 14:00 UTC), make sure to:

  • Check that torrents are published and seeding properly.
  • Unhide the directories that were hidden as part of the stage-release script — these should now be made available to the public.
  • This step ensures that the content officially becomes public and everything is accessible after the release GO decision.

rebased onto b52b32f

a month ago

Only the signing one needs sigul. The other one can be run on any compose machine with /pub rw (typically compose-x86-01, but I suppose it could also be bodhi-backend01)

This should be moved to Monday, the day before release. We want to have the torrents up in advance so people can download the content to seed it to other users.
We don't typically announce this widely, just to those folks who normally seed torrents.

This should be moved to monday night (late) before the release. This is to allow mirrors that are not tier1 (and don't have access to pre-release content) to start syncing up, so that more of them are in sync by the time of the actual release announcement.
We typically don't announce this widely either. Mirrors will just start syncing it, we don't need to tell users about it.

A few minor comments, but otherwise looks good! Thanks for updating this!

rebased onto b52b32f

a month ago
Metadata