#357 EPEL10+ minor branching SOP
Merged 2 months ago by dherrera. Opened 3 months ago by dherrera.
dherrera/infra-docs-fpo epel_minor_branching  into  master

file modified
+1
@@ -41,6 +41,7 @@ 

  *** xref:release_guide:sop_file_ftbfs.adoc[File FTBFS bugs]

  *** xref:release_guide:mass_branching_checklist.adoc[Mass Branching Checklist]

  *** xref:release_guide:sop_mass_branching.adoc[Mass Branching]

+ *** xref:release_guide:sop_epel_minor_mass_branching.adoc[EPEL Minor Version Mass Branching]

  *** xref:release_guide:beta_freeze.adoc[Beta freeze and updates-testing activation]

  *** xref:release_guide:beta_RC_compose.adoc[Beta RC compose]

  *** xref:release_guide:beta_release.adoc[Beta release]

@@ -6,3 +6,8 @@ 

  :branched_name: Forty Two

  :current: 41

  :old_release: 40

+ :epel_major: 10

+ :epel_minor: 0

+ :epel_minor_new: 1

+ :epel_current: {epel_major}.{epel_minor}

+ :epel_new: {epel_major}.{epel_minor_new}

@@ -0,0 +1,221 @@ 

+ include::_partials/attributes.adoc[]

+ 

+ = EPEL Minor Version Mass Branching

+ 

+ == Description

+ 

+ Originally, the EPEL repository had packages that built against the latest RHEL minor, but since EPEL{epel_major}, the EPEL repository now builds against the latest Centos Stream release. To keep consistency for RHEL releases, EPEL repository goes through a mass branching process to generate a new minor release branch. This process let's the developers continue to maintain their packages against Centos Stream without risking to break packages that already work on the current active RHEL minor target.

+ 

+ == Send announcement

+ 

+ One day before the mass branching, we send out announcement because during mass branching,

+ new koji builds for EPEL are disabled.

+ 

+ === Remove Targets

+ 

+ In Fedora it is recommended to https://docs.fedoraproject.org/en-US/infra/release_guide/sop_mass_branching/#_disable_rawhide_builds_in_koji[stop connections to koji through firewall]. 

+ But it was https://pagure.io/infra-docs-fpo/pull-request/357[decided not to require it] for this process.

+ 

+ In the case of EPEL, just removing the targets is enough to disable it's usage for the time being while the branching process it's being done.

+ 

+ [source,bash,subs="attributes"]

+ ----

+ $ koji remove-target epel{epel_major}-candidate

+ $ koji remove-target epel{epel_major}

+ ----

+ 

+ === Cancel all running build for EPEL{epel_major}

+ 

+ List all running tasks and select only those relevant for current branching.

+ 

+ [source,bash,subs="attributes"]

+ ----

+ $ koji list-builds --state=0 --type=rpm | grep el{epel_major}_{epel_minor} | awk '{print $1}'

+ ----

+ 

+ Cancel each of those tasks

+ 

+ ```

+ $ koji cancel <build>

+ ```

+ 

+ == dist-git

+ 

+ Now we need to update dist-git in two steps:

+ 

+ * Create the new branch in git

+ 

+ For both of these actions you will need the list of active repos in EPEL{epel_major}.

+ 

+ === Get current EPEL active packages

+ 

+ To get the list of all the current active packages, you must run the `scripts/branching-epe/get_all_active_packages_branching.sh` script.

+ 

+ [source,bash,subs="attributes"]

+ ----

+ sh scripts/branching-epel/get_all_active_packages_branching.sh {epel_major}

+ ----

+ 

+ The results will be saved to the components_epel{epel_new}.txt

+ 

+ === Create the git branches

+ 

+ On `pkgs01.stg` (for testing) or `pkgs02` (for production), run:

+ 

+ [source,bash,subs="attributes"]

+ ----

+ $ sudo -u pagure python /usr/local/bin/mass-branching-git.py --branch-from  epel{epel_major} epel{epel_current} components_epel{epel_major}.txt 

+ ----

+ 

+ The first argument is the new epel version and the

+ second one is the path to the file generated in the previous step.

+ 

+ == Ansible

+ 

+ Apps in https://pagure.io/fedora-infra/ansible[ansible] need to be updated

+ to be aware of the new branch.

+ 

+ The following sections describe the things that need to be updated, but it's recommended to use https://pagure.io/fedora-infra/ansible/pull-request/2473#request_diff[this PR]

+ for reference.

+ 

+ === Bodhi

+ 

+ Bodhi needs to be updated to add new release. This needs to be done in

+ https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2[bodhi2 role] in

+ infra ansible repo. This change includes, updating `koji-sync-listener.py`,

+ `new-updates-sync`, pungi configs for rpm updates, bodhi templates.

+ 

+ * `roles/bodhi2/backend/files/new-updates-sync`

+ * `roles/bodhi2/backend/tasks/main.yml`

+ * `roles/bodhi2/backend/templates/koji_sync_listener.toml`

+ 

+ [NOTE]

+ ====

+ In this step in Fedora, `roles/bodhi2/backend/templates/pungi.rpm.conf.j2`

+ would also need to be modified. But here it's not required since the signing key doesn't

+ change between minor versions

+ ====

+ 

+ === Toddlers

+ 

+ ==== Add new SLA to the toddlers App

+ 

+ Make sure that the SLAs are added. Check https://pagure.io/fedora-infra/ansible/pull-request/2190#request_diff[this PR] 

+ for reference on what needs to be available.

+ 

+ === Koji hub

+ 

+ Update the koji hub config to allow side tags for new koji epel{epel_major} minor (currently epel{epel_new}) tag.

+ 

+ * `roles/koji_hub/templates/hub.conf.j2`

+ 

+ === Robosignatory

+ 

+ All minor branches use the same key, so it's not required to generate a new one, but it's important to update the configuration files so that the propper keys are used

+ 

+ * `roles/robosignatory/templates/robosignatory.toml.j2`

+ 

+ === Push the changes

+ 

+ When done editing the files, commit, push and apply them via the corresponding

+ ansible playbooks:

+ 

+ [source,bash,subs="attributes"]

+ ----

+ $ sudo rbac-playbook groups/batcave.yml

+ $ sudo rbac-playbook groups/koji-hub.yml

+ $ sudo rbac-playbook groups/bodhi-backend.yml

+ $ sudo rbac-playbook manual/autosign.yml

+ ----

+ 

+ == Koji

+ 

+ The koji build system needs to have some tag/target work done to handle

+ builds from the new branch and to update where builds from epel{epel_major} go.

+ 

+ Run

+ https://pagure.io/releng/blob/main/f/scripts/branching-epel/make-koji-release-tags[make-koji-release-tags]

+ script in https://pagure.io/releng[pagure releng] repo

+ 

+ [NOTE]

+ ====

+ Make sure that things tagged on epel{epel_current}-testing are tagged correctly.

+ https://bodhi.fedoraproject.org/updates/?search=&status=pending&status=testing&releases=EPEL-{epel_current}

+ ====

+ 

+ == Bodhi

+ 

+ === Linking Empty Repos

+ 

+ We need to link empty repos so that new-updates-sync wont complain about

+ missing repos. The following commands should be run on

+ *bodhi-backend01.phx2.fedoraproject.org*

+ 

+ [source,bash,subs="attributes"]

+ ----

+ $ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/epel{epel_new}

+ $ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/epel{epel_new}-testing

+ ----

+ 

+ === Creating Empty Repos

+ 

+ To create empty repos on the master mirror, run

+ https://pagure.io/releng/blob/main/f/scripts/branching-epel/create_empty_repos.sh[create_empty_repos.sh]

+ from https://pagure.io/releng[pagure releng] repo. This should be run on

+ *bodhi-backend01.phx2.fedoraproject.org*

+ 

+ [source,bash,subs="attributes"]

+ ----

+ $ sudo -u ftpsync sh scripts/branching-epel/create_empty_repos.sh {epel_current}

+ ----

+ 

+ === Creating EPEL{epel_new} release

+ 

+ To create a new minor release in bodhi, you need to run:

+ 

+ [source,bash,subs="attributes"]

+ ----

+ $ bodhi releases create \

+     --name "EPEL-{epel_new}" \

+     --long-name "Fedora EPEL {epel_new}" \

+     --version {epel_major} \

+     --branch epel{epel_major} \

+     --id-prefix FEDORA-EPEL \

+     --dist-tag epel{epel_new} \

+     --stable-tag epel{epel_new} \

+     --testing-tag epel{epel_new}-testing \

+     --candidate-tag epel{epel_new}-testing-candidate \

+     --pending-signing-tag epel{epel_new}-signing-pending \

+     --pending-testing-tag epel{epel_new}-testing-pending \

+     --pending-stable-tag epel{epel_new}-pending \

+     --override-tag epel{epel_new}-override \

+     --mail-template fedora_epel_legacy_errata_template \

+     --state current \

+     --composed-by-bodhi \

+     --no-create-automatic-updates

+ ----

+ 

+ === Recreate targets

+ 

+ Once mass-branching is complete and the new release exists in bodhi,

+ re-create these targets with the following commands:

+ 

+ [source,bash,subs="attributes"]

+ ----

+ $ koji add-target epel${release_major}           epel${release}-build epel${release}-testing-candidate

+ $ koji add-target epel${release_major}-candidate epel${release}-build epel${release}-testing-candidate

+ ----

+ 

+ === Updating MirrorManager

+ 

+ MirrorManager will load the changes automatically, you can check that everything is in order throught the following command:

+ 

+ [source,bash,subs="attributes"]

+ ----

+ $ curl 'https://mirrors.fedoraproject.org/metalink?arch=x86_64&repo=epel-{epel_current}'

+ ----

+ 

+ === Branch new EPEL{epel_major} minor in Koschei

+ 

+ Branch new fedora EPEL{epel_major} minor in

+ https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/koschei/#_branching_a_new_fedora_release[koschei].

This document is based on the Fedora Release Mass Branching SOP, but adapted to reflect what is needed to be done to deploy a new EPEL minor version release.

References:
EPEL10 Work tracker

Signed-off-by: Diego Herrera dherrera@redhat.com

1 new commit added

  • Resolve some of the TODOs
3 months ago

2 new commits added

  • Resolve some of the TODOs
  • Start EPEL10+ minor branching draft
3 months ago

So this essentially blocks any access to koji. We did that for fedora branching because people kept ignoring that it was happening and doing builds in the middle of it and ending them up in weird states. ;(

This may be less of an issue with epel? Perhaps we don't do this at first and see how it goes. It's pretty disruptive.:(

gitolite is not used anymore... this can be dropped (and should be from the fedora one)

pkgs machines are weird auth wise. They don't allow shell logins except to root, so thats syadmin-main. ;(
However, I wonder... could we make this a playbooks/manual playbook we just run from batcave?

yeah, no epel issue here... unless you want to set some variables? They are handy for using in templates so you don't need to change the same '10.0' or 10.1' or whatever in a bunch of places.

2 new commits added

  • Add extra note about testing packages
  • Update some steps of the process
3 months ago

hum I guess koschei is desired for each minor? I guess needs discussed...

1 new commit added

  • Drop more stuff that isn't required for EPEL
3 months ago

So this essentially blocks any access to koji. We did that for fedora branching because people kept ignoring that it was happening and doing builds in the middle of it and ending them up in weird states. ;(

This may be less of an issue with epel? Perhaps we don't do this at first and see how it goes. It's pretty disruptive.:(

Honestly, I agree, but didn't want to decide on this before checking the impact on staging... but the process is a lot smaller than what it's being done on fedora, so it might just be ok I suppose :)

gitolite is not used anymore... this can be dropped (and should be from the fedora one)

pkgs machines are weird auth wise. They don't allow shell logins except to root, so thats syadmin-main. ;(
However, I wonder... could we make this a playbooks/manual playbook we just run from batcave?

If i'm not mistaken, this is the actual script

https://infrastructure.fedoraproject.org/infra/ansible/roles/distgit/files/mass-branching-git.py

Are you talking about building a playbook that runs it? or to add variables to track current EPEL versions and keep the state properly? I'm in to work on either of those options on a future iteration of this process :)

yeah, no epel issue here... unless you want to set some variables? They are handy for using in templates so you don't need to change the same '10.0' or 10.1' or whatever in a bunch of places.

I'll remove this for now then... might consider for future iterations

Yeah, these next two things do not apply to epel

removed :D

drop, doesn't apply to epel

removed :D

hum I guess koschei is desired for each minor? I guess needs discussed...

I don't have a strong opinion on this ... maybe we can keep it on the active minors?

and thank you a lot @kevin for all the feedback ^^

So this essentially blocks any access to koji. We did that for fedora branching because people kept ignoring that it was happening and doing builds in the middle of it and ending them up in weird states. ;(

This may be less of an issue with epel? Perhaps we don't do this at first and see how it goes. It's pretty disruptive.:(

Honestly, I agree, but didn't want to decide on this before checking the impact on staging... but the process is a lot smaller than what it's being done on fedora, so it might just be ok I suppose :)

Yeah, I don't think it's needed. If something causes it to be needed, we can change it next time? :)

gitolite is not used anymore... this can be dropped (and should be from the fedora one)

pkgs machines are weird auth wise. They don't allow shell logins except to root, so thats syadmin-main. ;(
However, I wonder... could we make this a playbooks/manual playbook we just run from batcave?

If i'm not mistaken, this is the actual script

https://infrastructure.fedoraproject.org/infra/ansible/roles/distgit/files/mass-branching-git.py

Are you talking about building a playbook that runs it? or to add variables to track current EPEL versions and keep the state properly? I'm in to work on either of those options on a future iteration of this process :)

Yeah, just a playbook in playbooks/manual/ thats manually run by someone at the right time.

Right now running that requires sysadmin-main, because you have to login as root to pkgs01 and run it.
If we move it to a playbook we can make it so anyone in sysadmin-releng could run it.
But yeah, can be a later iteration, and just get someone from sysadmin-main to run it for now.

yeah, no epel issue here... unless you want to set some variables? They are handy for using in templates so you don't need to change the same '10.0' or 10.1' or whatever in a bunch of places.

I'll remove this for now then... might consider for future iterations

Yeah, these next two things do not apply to epel

removed :D

drop, doesn't apply to epel

removed :D

hum I guess koschei is desired for each minor? I guess needs discussed...

I don't have a strong opinion on this ... maybe we can keep it on the active minors?

Well, it doesn't do a new release unless you add it.

https://koschei.fedoraproject.org/

it doesn't yet even have 10 (although there is a reqest to add it)

I'd say this can be deferred and added later.

and thank you a lot @kevin for all the feedback ^^

Happy to help! Thanks for working on it.

5 new commits added

  • Add variables to the docs
  • Clean up errors
  • Remove iptable instructions
  • Remove warning for Koschei
  • Remove final TODO's
3 months ago

1 new commit added

  • Add fixes
3 months ago

1 new commit added

  • Fix grep on koji list command
3 months ago

1 new commit added

  • Fix explanation
3 months ago

1 new commit added

  • Change new parameter to work as a flag
3 months ago

14 new commits added

  • Change new parameter to work as a flag
  • Fix explanation
  • Fix grep on koji list command
  • Add fixes
  • Add variables to the docs
  • Clean up errors
  • Remove iptable instructions
  • Remove warning for Koschei
  • Remove final TODO's
  • Drop more stuff that isn't required for EPEL
  • Add extra note about testing packages
  • Update some steps of the process
  • Resolve some of the TODOs
  • Start EPEL10+ minor branching draft
3 months ago

1 new commit added

  • Add manual koji steps to the SOP
3 months ago

4 new commits added

  • Remove deprecated steps
  • Fix version that needs new empty repos
  • Fix the new epel link on koji
  • Set the correct playbooks that are needed to run
3 months ago

In order to construct the right grep here, we may want to restructure the attributes so we can use the minor version independently, something like el{epel_major}_{epel_minor_current}.

Lets add to this list:

  • roles/koji_hub/templates/hub.conf.j2
  • roles/robosignatory/templates/robosignatory.toml.j2

No changes to mirrormanager were necessary. We should however modify this section to verify that the mirrormanager regex scanner picked up the new directory.

curl 'https://mirrors.fedoraproject.org/metalink?arch=x86_64&repo=epel-{epel_current}'

4 new commits added

  • EPEL minor branching - Reorganize ansible section
  • EPEL minor branching - Update MirrorManager instructions
  • EPEL minor branching - add files required to modify to make bodhi work
  • Setup compound variables for epel major and minor
3 months ago

rebased onto 3e0e8d3

2 months ago

Pull-Request has been merged by dherrera

2 months ago