T Day actions for mass branching

(for reviewers here is a checklist that one can copy this on the releng tracker, so we can keep track of everything needed in one place)

Merge all the prepatory PRs:

• [ ] pungi-fedora
• [ ] fedora-kickstarts
• [ ] fedora-comps
• [ ] workstation-ostree-config/
• [ ] fedora-release
• [ ] fedora-repos
• [ ] Toddlers

Merging and running ansible changes

Push the Changes

• [ ] Commit, push, and apply the changes using the corresponding ansible playbooks for various services.

Disable Rawhide Builds in Koji

• [ ] Configure an outage in Koji to disable Rawhide builds.
• [ ] Cancel all running builds for Rawhide by listing and selecting relevant tasks, and then cancel each task.

PDC (Product Definition Center)

• [ ] Create a new "product-release" in PDC using the provided script.
• [ ] Clone or update the releng repository on PDC backend.
• [ ] Run the create-new-release-branches.py script to set up the new release branches, ensuring to use the --createfile argument.

Koji

• [ ] Run the make-koji-release-tags script from the releng repository to handle builds from the new branch.

Dist-Git

• [ ] Create new branches in Git and update gitolite.conf to allow users to push to the new branches.
• [ ] Run the mass-branching-git.py script to create new branches based on the file generated by PDC.

Bodhi

• [ ] Link empty repos and create empty repos as necessary to prepare for the new release.
• [ ] Create rawhide releases in Bodhi using appropriate commands for various types of releases (e.g., standard, container, flatpak).
• [ ] Update MirrorManager to point to the new Rawhide release.
• [ ] Enable autosigning on the Branched release after the compose is completed.
• [ ] Perform ELN-related work, including updating image configurations and scripts.

Fedora Container Base Image

• [ ] Import new images for Rawhide and update tags for fedora:rawhide and fedora:${RAWHIDE}.

Update Sync Script

• [ ] Update the sync script in the releng repository with the new version.

As @sgallagh mentioned, before we start to branch:

(ELN) need to hit the pause button on our auto-rebuilds a few minutes ahead of the rest of the branch.
So we can let anything in progress finish, do the final import to CentOS Stream 10, update our macros for RHEL 11, and then resume

ELN auto rebuilds are stopped

@decathorpe pointed out: https://bodhi.fedoraproject.org/updates/?search=fc40&releases=F41

Issues we hit:

• Bodhi had some wrong info in it for f40/rawhide. See https://pagure.io/releng/issue/11950

• we switched iot 'stable' to the new rawhide key, but it should stay on the previous/stable release. See: https://pagure.io/releng/issue/11951

• the h264 repo was still pointing to f40... we need to make a empty repo for the new rawhide and point mirrormanager to it. I can provide a SOP for this.

• we needed to run the playbooks/openshift-apps/toddlers.yml playbook to push out the change allowing users to request f40 branches. See https://pagure.io/releng/issue/11948

It would be nice to figure out a better way to stop users from submitting builds, but not sure what.

Overall much nicer than the last cycle IMHO. we have composes and everything is pretty much back to normal now. ;)

Thanks @kevin for these points! I'll make sure to update the sop+checklist as well from these details. Once, we have sop ready for [3] point, I can link that as well to our mass branching sop - we need to restructure the docs as well.

Overall much nicer than the last cycle IMHO. we have composes and everything is pretty much back to normal now. ;)

Sounds like a relief that i didn't mess much of the things <3

Thanks to you and @humaton!!!