#12241 Fedora 41 Mass Branching Tracker
Opened a month ago by jnsamyak. Modified 11 days ago

The Fedora 41 schedule[1] has a mass branching schedule, We need to plan and coordinate all tasks in preparation for it. For the driving changes please refer to [2].

Most importantly for the reviewers or the participators for the PRs and other $stuff here is the checklist we created from our last branching and will be our source to check if we missed anything!
https://docs.fedoraproject.org/en-US/infra/release_guide/mass_branching_checklist/

[1] https://fedorapeople.org/groups/schedule/f-41/f-41-key-tasks.html
[2] https://fedoraproject.org/wiki/Fedora_41_Mass_Rebuild#Driving_Features


Metadata Update from @jnsamyak:
- Issue assigned to jnsamyak

a month ago

Metadata Update from @jnsamyak:
- Issue tagged with: f41, high-gain, high-trouble

a month ago

T Day actions for mass branching

(for reviewers here is a checklist that one can copy this on the releng tracker, so we can keep track of everything needed in one place)

Merge all the preparatory PRs:

Merging and running ansible changes

Push the Changes

  • [ ] Commit, push, and apply the changes using the corresponding ansible playbooks for various services.

Disable Rawhide Builds in Koji

  • [ ] Configure an outage in Koji to disable Rawhide builds.
  • [ ] Cancel all running builds for Rawhide by listing and selecting relevant tasks, and then cancel each task.

PDC (Product Definition Center)

  • [ ] Create a new "product-release" in PDC using the provided script.
  • [ ] Clone or update the releng repository on PDC backend.
  • [ ] Run the create-new-release-branches.py script to set up the new release branches, ensuring to use the --createfile argument.

Koji

  • [ ] Run the make-koji-release-tags script from the releng repository to handle builds from the new branch.

Dist-Git

  • [ ] Create new branches in Git and update gitolite.conf to allow users to push to the new branches.
  • [ ] Run the mass-branching-git.py script to create new branches based on the file generated by PDC.

Bodhi

  • [ ] Link empty repos and create empty repos as necessary to prepare for the new release.
  • [ ] Create rawhide releases in Bodhi using appropriate commands for various types of releases (e.g., standard, container, flatpak).
  • [ ] Update MirrorManager to point to the new Rawhide release.
  • [ ] Enable autosigning on the Branched release after the compose is completed.
  • [ ] Perform ELN-related work, including updating image configurations and scripts.

Fedora Container Base Image

  • [ ] Import new images for Rawhide and update tags for fedora:rawhide and fedora:${RAWHIDE}.

Update Sync Script

  • [ ] Update the sync script in the releng repository with the new version.

Add version 41 to bugzilla?

Two things:

Add version 41 to bugzilla?

@churchyard, do you have anything specific to what we do that is not written here? Or, rephrasing what/where needs to be updated?

Another thing, as mentioned on the channel was, that we did a stable push before we started branching so everything is in non-weird state

I did not see anything bugzilla-related in the checklist. Perpas that happens automatically and all is good, but perhaps it needs to be done manually and hence needs to be listed?

It doesn't happen automatically, it has to be done via the web interface.
We need to get you added to the group that can do this. In the mean time either myself or @amoloney should be able to do this.

Thanks for the heads up @kevin! And this is what we are missing from the docs as well!

One more thing is I have for @humaton, with the PDC gone we should skip creating new releases in PDC as part of branching correct?

Hi folks, these are the changes that are required during the mass_branching process, can we please give all these a quick round of reviews, especially to the new changes in ansible, kiwi changes are introduced etc

Repository PR Link Reviewed Merged
pungi-fedora (rawhide) Link to PR
pungi-fedora (f41) Link to PR
fedora-kickstarts (rawhide) Not required(right?)
fedora-kickstarts (f41) Link to PR
fedora-repos (rawhide) Link to PR
fedora-repos (f41) Link to PR
fedora-release (rawhide) Link to PR
fedora-release (f41) Link to PR
workstation-os-config (rawhide) Link to PR
workstation-os-config (f41) Link to PR
fedora-lorax-template (rawhide) Not required
fedora-lorax-template (f41) Just push a new branch without any changes
workstation-os-config (rawhide) No changes required
fedora-kiwi-descriptions (f41) Link to PR
fedora-comps (main) Link to PR
ansible (main) Link to PR
releng (rawhide) Link to PR

cc: @kevin @humaton @patrikp @amedvede @siosm @ngompa

@sgallagh, the branching is scheduled for tomorrow, like last time, is something that needs to be turned off there? I wanted to check and inform one day prior branching happens so we could be ready!

I went through and reviewed all of them. Most are looking good. A few minor comments on others.

One more important review for disabling builds during the branching: https://pagure.io/fedora-infra/ansible/pull-request/2203

As always PDC is here for being pain in the ass, before the decommissioning we used to create new releases in PDC which in turns gives us a list of file which we fed to dist-git for branching, the file use to had all the active packages in the rawhide!

Now since we don't have it, and we need to branch we did look into the API from the https://src.fedoraproject.org/api/0/projects?namespace=rpms&branch=rawhide which on parsing the parent => full name from the project dictionary gives us only 14Kish approx packages which is a lot less than usual, we branched 23K approx packages last time;

I'm trying to debug this, and @lenkaseg is helping me to look the problems within the script: https://paste.centos.org/view/56ba795c

So we might need to hold off branching if we couldn't;t figure out the list of packages within time

Another intersting findings are:

➜ jnsamyak@fedora  ~/work/scripts/tmp  jq '.rawhide | length' retired_in_rawhide.json
14783

➜ jnsamyak@fedora  ~/work/scripts/tmp  jq '.f40 | length' retired_in_f40.json
213

We had only 213 packages that were retired in the f40 but in rawhide with the new script it shows 14783, this retired packages number also seems a little off to me!

Note that all packages that were ever retired in rawhide throughout the entire history of Fedora dist-git will be a lot more packages compared to packages that are retired on any specific branch.

That's because after retirement we no longer branch and after branching, only a handful of packages get retired.

That makes so much sense!

Okay with my sysadmin-main hat on, and with the help of Lenka (thanks!)

We found the correct packages (guessing by the count), and we got that from the pkg machine by negating the script from the get_retired_packages.sh
and found out non-retired packaged on rawhide:
24344 components_f41.txt (edited)

there are 24344 that will be branched today

24232 seems to be in the repository:

$ repoquery -q --repo=koji-source -a --latest=1 | wc -l
24232

Your query probably has packages that were never built, are not available on my architecture (x86_64) etc. as well, so it is probably OK that it is a bit higher than mine.

Post Branching Retrospective

(these are things we did for the first time, things we can do better the next time, and things that we missed this time):

  • use of custom ip for blocking external folks to submit builds in koji
  • add rawhide defination and new keys to all the stable fedora-repos
  • creation of new bugzilla component is missing from the docs
  • fedora-repos was updated to point to rawhide key and f43 key. Thats not right, it should keep pointing to the previous key until we make the next one (f44) and then we can point it to also f43... i.e, we want it to point to rawhide and the previous one so things signed with the old key keep working
  • https://pagure.io/releng/issue/12255 we need to update the link in /mnt/koji/repos/rawhide/latest
  • check for directory permission for new compose to sync properly
  • perhaps we could use sidetags for the fedora-release/fedora-repos updates and also then update mock-core-configs and distribution-gpg-keys and push them all in one update.
  • we may also want to disable rawhide cron in the branching pr... and make sure we get a branched compose and then push another commit to re-enable rawhide

Two new things to add on as well:

  • make sure to activate the update-testing repository for the branched version (it can be done at the time of branching or before the beta freeze activation, the former is preferred)
  • make sure template_branches in the ostree section inside the pungi-config (pointing to lorax) should correctly point to there branched version

Log in to comment on this ticket.

Metadata