So, I have started on this in stg late last week. @phsmoura is gonna help document this process.

I thought we had a SOP, but the nearest I can find is:
https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/koji-builder-setup/
which is massively out of date and should probibly be deleted. ;(

So, I would say we should start a new SOP... 'Builder reinstall/upgrades' or something.

A bit of background:

Koji has several parts. A database backend, some hubs, some kojipkgs servers (where packages are downloaded from), and builders. Long ago we had builders that were RHEL. This caused some problems because Fedora would update rpm or glibc or kernels in such a way that the new Fedora version was not compatible with the old RHEL version and builds wouldn't work anymore. So, we moved builders to Fedora to keep them mostly up with this. Now days mock uses a bootstrap to setup the build env chroot, which means this is much less important, although there are still sometimes reasons to run a recent Fedora over RHEL on builders (ability to get fixes faster, better ability to interface with maintainers, etc).

So, after each Fedora release is out for at least a few weeks, we look at reinstalling or upgrading all the builders to that new Fedora release. Some builders are difficult to reinstall (namely the bare metal hardware ones, and the s390x z/vm ones), so we usually upgrade those and reinstall all the vm ones. We first start by upgrading staging and doing a bit of testing to make sure all is working well there before starting on production.

First an ansible commit is done to update the virt-install path on the builders. Like for example https://pagure.io/fedora-infra/ansible/c/af94db8ab88588aa8dc2c20893b2487afb7f7777?branch=main then, each builder vm is removed (either via virt-inst-dest or manually) and the buildvm.yml playbook is run to deploy them. Any problems in the kickstart or config is then handled until the builders are all redeployed with the new version. Once thats complete, the hub is re-installed or upgraded with 'dnf --refresh --releasever <release> distro-sync'. Then, test builds are done to make sure everything is working.

For production a similar process is followed, but to avoid outages there, builders are disabled, reinstalled and re-added small groups at a time.

I've reinstalled the staging ones now, not yet started on prod so far.

ok, this is finally all done.

I also upgraded the hubs to the latest version. :)

@phsmoura let me know when you have a start of a SOP document and we can work on that together from there. :)

I'm also going to mail an announcement out about this.

Hi @kevin, I started writing it with the info you provided in the previous comment. I can go back to it anytime