epel

Clone
Source Code
GIT

#280 Create basic documents to help deal with EPEL package set EOL
Opened 7 months ago by smooge. Modified 6 months ago
smooge/epel main  into  main

Merge
file added
+24 
@@ -0,0 +1,24 @@ 
 

+ Hello all,
 

+ 

+ The Extra Packages for Enterprise Linux (EPEL) will be ending support
 

+ for packages built against Red Enterprise Linux (RHEL) NN on YYYY-MM-DD.
 

+ This will match the time as RHEL NN moves out of maintenance mode and
 

+ into extended life-cycle support.
 

+ 

+ No more updates of any kind, including security updates or security
 

+ announcements, will be available for EPEL-NN after the said date. All
 

+ the updates of EPEL-NN being pushed to stable will be stopped as well.
 

+ 

+ Packages in the download sites of /pub/epel/NN/ will be moved to
 

+ /pub/archive/epel/NN/ on participating mirrors. The Fedora mirror
 

+ manager used by dnf or yum will be updated to point to those mirrors.
 

+ Existing mirrors of EPEL-NN will be cleared of EPEL-NN package at some
 

+ point due to this movement. Sites using EPEL-NN are encouraged to mirror
 

+ the content locally for their future use as again no updates of this
 

+ material will be happening after YYYY-MM-DD.
 

+ 

+ Regards,
 

+ 

+ Fedora Release Engineering for the EPEL Steering Committee.
 

+ 

+ [1] -> point to the EPEL Release Life Cycle

file added
+81 
@@ -0,0 +1,81 @@ 
 

+ The Fedora Project’s Extra Packages for Enterprise Linux (EPEL) group
 

+ follows the release cycle of Red Hat Enterprise Linux (RHEL) for how the
 

+ different EPEL repositories are installed, updated, and maintained.
 

+ Releases of RHEL tend to be every 3 to 5 years with an expected lifetime

Now that RHEL 7 is EOL, I think it's safe to just say every 3 years here. RHEL 8 and up are on 3 year major, 6 month minor schedule.

 
+ of 10 years for normal maintenance.
 

+ 

+ The EPEL repositories are not a complete operating system, but a '`stone
 

+ soup`' repository where volunteers take packages from equivalent Fedora

I'd drop the word equivalent here, we don't require it or even suggest it as far as I'm aware. I'd even say it's more common for maintainers to merge from the rawhide branch for a new epel branch, unless the software version in rawhide doesn't build against that RHEL for some reason.

 
+ releases and recompile them to work with Red Hat Enterprise Linux.
 

+ Because this is volunteer activity, packages may be in one EPEL
 

+ repository (say the one equivalent for RHEL-7) but not be in other
 

+ repositories (for RHEL-8 or RHEL-9).

I think we should go into a bit more detail here, but avoid talking about specific versions to avoid having to periodically update the versions. Something like:

Not all Fedora packages are appropriate or necessary for EPEL, and some are even not allowed.
This varies between EPEL major versions.
Because of this, we do not automatically build Fedora packages in new major versions of EPEL.
We also do not automatically build packages from previous EPEL major versions in new EPEL major versions.

This would also be a good place to work in a mention of (and link to) the package request guide (xref:epel-package-request.adoc[link text]).

 
+ 

+ [[_release_dates]]
 

+ == Release Dates
 

+ 

+ Release dates for EPEL repositories are tied to multiple items:
 

+ 

+ [arabic]
 

+ . How much work it takes for the EPEL release team to get tooling to
 

+ work with the Red Hat Enterprise Linux software. While working with
 

+ CentOS Stream has sped up recent releases, there may be various problems
 

+ which take longer for volunteers to work out.
 

+ . What packages are shipped in the main RHEL release and what packages
 

+ may be '`build-root-only`' in that they only exist inside of Red Hat’s
 

+ build system. These packages may be build dependencies for many other
 

+ packages and some amount of rebuilding and porting is needed to get this
 

+ done.

I would elaborate a bit here on the difference between the release of the EPEL repo, versus the release of individual package. The shipped and buildroot-only packages info is the right segue to this distinction I think.

 
+ 

+ [[_development_schedule]]
 

+ == Development Schedule
 

+ 

+ In the past, EPEL development schedules were tied with when Red Hat
 

+ publicly released their beta for the upcoming operating system. This
 

+ would allow for initial porting and code work in the Fedora build
 

+ systems to support the RHEL release as a base for building against. Due
 

+ to various differences between the Beta and the first RHEL release, a
 

+ lot of work might need to be redone for the final release.
 

+ 

+ With the addition of '`CentOS Stream`' as part of the Red Hat Enterprise

We can drop the quotes here around CentOS Stream. We don't quote Fedora or Red Hat Enterprise Linux in other parts of the doc.

 
+ development model, the speed of initial release of an EPEL release has
 

+ improved. However there may still be delays between the initial release
 

+ of a Red Hat Enterprise Linux major release and an EPEL package set.
 

+ 

+ Furthermore, because packages in any EPEL package set are offered by
 

+ volunteers, there may be gaps or lack of packages for months as various
 

+ amounts of additional packages which are required to build specific
 

+ packages. This can only be sped up by volunteering help in porting,
 

+ checking spec files, and testing.
 

+ 

+ [[_development_planning]]
 

+ === Development Planning
 

+ 

+ Fedora development planning is handled by the EPEL Steering committee.

s/Fedora/EPEL/

 
+ 

+ *Other parts need to be filled out by people working on EPEL-10 and

I'm working on separate docs going in to the EPEL 10 changes. For this doc I think we can avoid specifics on it for now.

 
+ newer releases*
 

+ 

+ [[_end_of_life_eol]]
 

+ == End of Life (EOL)
 

+ 

+ EPEL package sets reach an end of life when their corresponding Red Hat
 

+ Enterprise Linux moves from Maintenance Support to Extended Life-cycle

Instead of mentioning the extended stuff, we can simplify this to just say "reaches the end of the Maintenance Support Phase".

 
+ Support.
 

+ 

+ [cols="<,<,<,<",options="header",]
 

+ |===
 

+ |RHEL ver |Release Date |EoMaint |EoExtended
 

+ |5 |2007-03-15 |2017-03-31 |2020-11-30
 

+ |6 |2010-11-10 |2020-11-30 |2024-06-30
 

+ |7 |2014-06-10 |2024-06-30 |2028-06-30
 

+ |8 |2019-05-07 |2029-05-?? |2031-05-??
 

+ |9 |2022-11-08 |2032-05-?? |2034-05-??
 

+ |10 |2025-??-?? |2035-??-?? |2037-??-??

I think we can leave the extended dates out of this table, as they aren't relevant to EPEL. We also already have specific dates for a few of these question marks:

  • RHEL 8 EOM 2029-05-31
  • RHEL 9 EOM 2032-05-31

We should also include a note that these dates are not authoritative and people should refer to the RHEL lifecycle page.

https://access.redhat.com/support/policy/updates/errata#Life_Cycle_Dates

 
+ |===
 

+ 

+ In general, when RHEL updates are no longer available under normal
 

+ contracts, the corresponding EPEL release is considered End of Life
 

+ (EOL). Branches and updated packages for the release are no longer
 

+ allowed in the Fedora build system, and EOL tasks are performed as
 

+ documented in the link:[EPEL End of Life SOP]

file added
+207 
@@ -0,0 +1,207 @@ 
 

+ [[_description]]
 

+ == Description
 

+ 

+ Each release of an EPEL package set is maintained according to the
 

+ following two factors:
 

+ 

+ [arabic]
 

+ . The Red Hat Enterprise Linux is in its main lifetime.

This may be too vague. Instead we could say "The corresponding Red Hat Enterprise Linux version is still in the Full or Maintenance Support Phase."

 
+ . There are sufficient volunteers to take care of the packages in that
 

+ release which is judged either by the EPEL Steering committee or in case
 

+ of no remaining members of that body then by Fedora Engineering Steering
 

+ Committee (FESCo).
 

+ 

+ If either of these parameters are met, then Fedora Release engineering

s/are met/are not met/

 
+ will begin putting that EPEL package set into `+end of life+` status.
 

+ This document describes the tasks necessary to move a package set to
 

+ that status.
 

+ 

+ [[_actions]]
 

+ == Actions
 

+ 

+ [[_reminder_announcement]]
 

+ === Reminder announcement
 

+ 

+ Send an email to `+devel@+`, `+devel-announce@+`, `+epel-devel@+`,
 

+ `+epel-announce+` and `+announce@+` lists as remainder about the release
 

+ EOL. Use the link:[EPEL EOL template] from the release engineering repo.
 

+ 

+ [NOTE]
 

+ ====
 

+ Due to the slow nature of Enterprise users, this reminder should
 

+ scheduled to be sent 1 month befoee the end of life of the appropriate

s/befoee/before/

 
+ RHEL release. In some cases this may not be possible so the note should
 

+ be sent at least 1 week before the remaining actions occur.
 

+ ====
 

+ 

+ [[_koji_tasks]]
 

+ === Koji tasks
 

+ 

+ * Disable builds by removing targets. Due to the 10 year length of time
 

+ an EPEL release has, the first step is to find all releases which
 

+ generally will have the pattern `+epel-{old_release}+` as in `+epel-8+`.
 

+ Confirm with another Release engineer that these are the ones to be
 

+ removed.
 

+ 

+ ....
 

+     $ koji remove-target epel-{old_release}
 

+     $ koji remove-target epel-{old_release}-candidate
 

+     $ koji remote-target <side-targets> #any side targets that are still around
 

+ ....
 

+ 

+ * Purge from disk the signed copies of rpms that are signed with the
 

+ EOL’d release key. To acheive this, add the release key to
 

+ *koji_cleanup_signed.py* script in https://pagure.io/releng[releng] repo
 

+ and the script on compose-branched01.iad2.fedoraproject.org
 

+ 

+ ....
 

+ ./scripts/koji_cleanup_signed.py
 

+ ....
 

+ 

+ [[_pdc_tasks]]
 

+ === PDC tasks

We may as well leave out the PDC section, as I understand it it's very close to being decommissioned.

https://pagure.io/fedora-infrastructure/issue/11820

 
+ 

+ [NOTE]
 

+ ====
 

+ PDC is to be replaced with some other tool and may have been replaced
 

+ several times over the lifetime of an EPEL release. Update this section
 

+ as needed.
 

+ ====
 

+ 

+ * Set PDC *active* value for the release to *False*
 

+ 

+ ....
 

+     curl -u: -H 'Authorization: Token <token>' -H 'Accept: application/json' -H 'Content-Type:application/json' -X PATCH -d '{"active":"false"}' https://pdc.fedoraproject.org/rest_api/v1/releases/epel-{old_release}/
 

+ ....
 

+ 

+ * Set the EOL dates in PDC for all the components to the release EOL
 

+ date if they are not already set. Run the following script from
 

+ https://pagure.io/releng[releng] repo
 

+ 

+ ....
 

+     python scripts/pdc/adjust-eol-all.py <token> f{old_release} 2020-11-24
 

+ ....
 

+ 

+ [[_bodhi_tasks]]
 

+ === Bodhi tasks
 

+ 

+ * Find the appropriate names for that particular EPEL package set. For
 

+ this document, we will assume that it was set up was
 

+ `+EPEL-{old_release}+` but it may have been different.
 

+ * Run the following bodhi commands to set the releases state to
 

+ *archived*
 

+ 

+ ....
 

+     $ bodhi releases edit --name "EPEL-{old_release}" --state archived
 

+     $ bodhi releases edit --name "EPEL-{old_release}M" --state archived

We don't need the second command here, we already archived EPEL 8 Modular and there won't be any more.

 
+ ....
 

+ 

+ [[_fedora_infra_ansible_changes]]
 

+ === Fedora Infra Ansible Changes
 

+ 

+ * We need to make changes to bodhi, koji, mbs, releng, autosign roles in
 

+ ansible repo.
 

+ * Run the associated playbooks on _batcave_
 

+ 

+ ....
 

+     $ sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/bodhi-backend.yml
 

+     $ sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/koji-hub.yml
 

+     $ sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/mbs.yml
 

+     $ sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/releng-compose.yml
 

+     $ sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/proxies.yml -t pkgdb2
 

+     $ sudo ansible-playbook /srv/web/infra/ansible/playbooks/manual/autosign.yml
 

+     $ sudo ansible-playbook /srv/web/infra/ansible/playbooks/openshift-apps/bodhi.yml
 

+ ....
 

+ 

+ [NOTE]
 

+ ====
 

+ Another way to run the playbook is using rbac-playbook, in case one
 

+ don’t have sysadmin-main rights or can’t become root. Syntax: sudo
 

+ rbac-playbook mbs.yml
 

+ ====
 

+ 

+ [[_final_announcement]]
 

+ === Final announcement
 

+ 

+ * Send the final announcement to `+devel@+`, `+devel-announce@+`,
 

+ `+test-announce@+`, `+announce@+` lists.
 

+ 

+ Use the link:[EPEL EOL template] from the release engineering repo
 

+ 

+ [[close-out-all-epel-nn-bugzilla-reports]]
 

+ === Close out all EPEL-NN bugzilla reports.
 

+ 

+ This will require working with administrators who have token access to
 

+ https://bugzilla.redhat.com. Scripts can be run which will collect all
 

+ open EPEL-NN tickets, and then auto-close then with an appropriate
 

+ template letting the user know that the Red Hat Enterprise Linux release
 

+ has reached its End of regular Maintenance and this package will not get
 

+ any further updates.
 

+ 

+ [[_move_the_eol_release_to_archive]]
 

+ ==== Move the EOL release to archive
 

+ 

+ [arabic]
 

+ . Log into to bodhi-backend01 and become root
 

+ +
 

+ ____
 

+ $ ssh bodhi-backend01.iad2.fedoraproject.org $ sudo su $ su - ftpsync
 

+ ____
 

+ . Then change into the releases directory.
 

+ +
 

+ ____
 

+ $ cd /pub/epel/ $ export DATE=$(date -I) $ export
 

+ OLD_REL=\{old_release_number} $ export MINOR=\{last minor release number
 

+ of RHEL like 9 or 10}
 

+ ____
 

+ . Check to see that the target directory doesnt already exist.
 

+ +
 

+ ____
 

+ $ ls /pub/archive/epel/ $ mkdir -p
 

+ /pub/archive/epel/latexmath:[{OLD_REL}.]\{DATE} $ pushd
 

+ /pub/archive/epel/ $ ln -s latexmath:[{OLD_REL}.](date -I)
 

+ latexmath:[{OLD_REL}.]\{MINOR} $ popd
 

+ ____
 

+ . Do a recursive rsync to update any changes in the trees since the
 

+ previous copy.
 

+ +
 

+ ____
 

+ $ rsync -avAXSHP
 

+ ./latexmath:[{OLD_REL}/ /pub/archive/fedora/linux/releases/]\{OLD_REL}.$\{DATE}/
 

+ ____
 

+ . We will now do testing in similar ways.
 

+ +
 

+ ____
 

+ $ cd ../updates/ $ mkdir /pub/archive/epel/latexmath:[{OLD_REL}.]\{DATE}
 

+ $ rsync -avAXSHP
 

+ /pub/latexmath:[{OLD_REL}/ /pub/archive/testing/]\{OLD_REL}.$\{DATE} $
 

+ ____
 

+ . Announce to the mirror list this has been done and that in 2 weeks you
 

+ will move the old trees to archives.
 

+ . In two weeks, log into mm-backend01 and
 

+ [arabic]
 

+ .. Determine what the directory name being used for that release is. It
 

+ might be /\{old_release}/Everything like Fedora or something else.
 

+ .. Run the archive script
 

+ +
 

+ ____
 

+ $ sudo -u mirrormanager mm2_move-to-archive –originalCategory="`EPEL`"
 

+ –archiveCategory="`EPEL Archive`" –directoryRe='`/\{old_release}`'
 

+ ____
 

+ . If there are problems, the postgres DB may have issues and so you need
 

+ to get a DBA to update the backend to fix items.
 

+ . Wait an hour or so then you can remove the files from the main tree.
 

+ 

+ ____
 

+ $ ssh bodhi-backend01 $ cd /pub/epel/\{old_release} $ ls # make sure you
 

+ have stuff here $ rm -rvf * $ ln ../6/README . $ cd
 

+ /pub/epel/testing/\{old_release} $ ls # make sure you have stuff here $
 

+ rm -rf * $ ln ../6/README .
 

+ ____
 

+ 

+ [[_consider_before_running]]
 

+ == Consider Before Running
 

+ 

+ * Resource contention in infrastructure, such as outages
 

+ * Extenuating circumstances for specific planned updates, if any
 

+ * Send the reminder announcement, if it isn’t sent already

Using the Fedora Release Engineering documents as a template,
begin to create a set of documents which will be followed in
future EPEL and EPEL-next EOL.

Signed-off-by: Stephen Smoogen ssmoogen@redhat.com

:thumbsup:

A related EPEL Issue:
https://pagure.io/epel/issue/279

Now that RHEL 7 is EOL, I think it's safe to just say every 3 years here. RHEL 8 and up are on 3 year major, 6 month minor schedule.

I'd drop the word equivalent here, we don't require it or even suggest it as far as I'm aware. I'd even say it's more common for maintainers to merge from the rawhide branch for a new epel branch, unless the software version in rawhide doesn't build against that RHEL for some reason.

I think we should go into a bit more detail here, but avoid talking about specific versions to avoid having to periodically update the versions. Something like:

Not all Fedora packages are appropriate or necessary for EPEL, and some are even not allowed.
This varies between EPEL major versions.
Because of this, we do not automatically build Fedora packages in new major versions of EPEL.
We also do not automatically build packages from previous EPEL major versions in new EPEL major versions.

This would also be a good place to work in a mention of (and link to) the package request guide (xref:epel-package-request.adoc[link text]).

I would elaborate a bit here on the difference between the release of the EPEL repo, versus the release of individual package. The shipped and buildroot-only packages info is the right segue to this distinction I think.

We can drop the quotes here around CentOS Stream. We don't quote Fedora or Red Hat Enterprise Linux in other parts of the doc.

s/Fedora/EPEL/

I'm working on separate docs going in to the EPEL 10 changes. For this doc I think we can avoid specifics on it for now.

Instead of mentioning the extended stuff, we can simplify this to just say "reaches the end of the Maintenance Support Phase".

I think we can leave the extended dates out of this table, as they aren't relevant to EPEL. We also already have specific dates for a few of these question marks:

  • RHEL 8 EOM 2029-05-31
  • RHEL 9 EOM 2032-05-31

We should also include a note that these dates are not authoritative and people should refer to the RHEL lifecycle page.

https://access.redhat.com/support/policy/updates/errata#Life_Cycle_Dates

This may be too vague. Instead we could say "The corresponding Red Hat Enterprise Linux version is still in the Full or Maintenance Support Phase."

s/are met/are not met/

s/befoee/before/

We may as well leave out the PDC section, as I understand it it's very close to being decommissioned.

https://pagure.io/fedora-infrastructure/issue/11820

We don't need the second command here, we already archived EPEL 8 Modular and there won't be any more.
Metadata
None
No Tags
Changes Summary 3
+24
file added
modules/ROOT/pages/epel-eol-email.adoc
+81
file added
modules/ROOT/pages/epel-lifecycle.adoc
+207
file added
modules/ROOT/pages/epel-release_eol.adoc
Powered by Pagure 5.14.1
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.