In the %dist tag section. You have

Packages built against epel8-rawhide-candidate will be built with a
%dist tag of .epel8_Y where “Y” is the latest stable release of RHEL
8.

That should be

Packages built against epel8-stable-candidate will be built with a
%dist tag of .epel8_Y where “Y” is the latest stable release of RHEL
8.

In the %dist tag section. You have
Packages built against epel8-rawhide-candidate will be built with a
%dist tag of .epel8_Y where “Y” is the latest stable release of RHEL
8.
That should be
Packages built against epel8-stable-candidate will be built with a
%dist tag of .epel8_Y where “Y” is the latest stable release of RHEL
8.

You are correct. I had a typo in my original email that was copied here. Thank you for catching it.

Thanks for writing this up smooge!

A few comments:

This branch will be left alone until and unless the packager decides
that they want to stage a major (possibly incompatible) change for the
next RHEL 8.Y minor release. At that time, they will need to remove
the package.cfg file from the epel8 branch and manually merge the
proposed changes desired into the epel8-rawhide branch and build
there.

I think we should also leave this open to people who want to build things that are just inherently unstable or rapidly changing.
ie, for those things they will always have a rhel8-rawhide package and may not have a epel8 package ever.

The side-effect here is that the build will also produce a build that goes to the epel8-rawhide repository without Bodhi intervention.

We may want to talk to rawhide gating folks about setting up or even using epel8-rawhide as a testing ground for gating.

Packages built against epel8-rawhide-candidate will be built with a
%dist tag of .epel8_rawhide
Packages built against epel8-rawhide-candidate will be built with a
%dist tag of .epel8_Y where “Y” is the latest stable release of RHEL
8.

Does this then mean we have to mass rebuild every single epel package at minor releases? Thats adding a bunch of work on us I think... so, I'd prefer to avoid that and just use '.epel8' for everything...

This dist tag structure ensures that the version of the package in the
stable epel8 repository will win out over the one in the epel8-rawhide
repository if all other aspects of the EVR are the same. (So one would
only pick up a newer version from epel8-rawhide if it was indeed a
higher version number.)

Well, the epel8-rawhide one should always be a newer version if there is a seperate one no? Or else doing a seperate build for rawhide makes little sense.

Side note: we really need a better name than epel8-rawhide... overloading rawhide is bad I think.

At 00:00 UTC of the day following a new RHEL 8.Y release, an updated
epel-release package will be pushed, updating the %dist tag to the new
.epel8_Y value. All new builds will thus have the new dist tag. A
script will be run at this time to apply a new Koji tag (epel8-8.Y) to
the latest build of a package with one of the following tags: [
epel8-stable, epel8-stable-pending ]. A compose of the epel8.Y
repository will be created at this time from all packages currently
tagged as epel8-8.Y.

I think we should use the method fedora does: after rhel8.Y comes out, we tag all current stable builds with rhel8_Y in koji. Then we know exactly what they were and can compose the archive set (or just take the last composedset and copy it). If we don't change dist, we also don't need to update epel-release all the time for it. :) The next updates pushes will contain packages built against the new minor release. (and any old ones not yet rebuilt).

So I can keep this straight in my head, I'm going to summarize both proposals.

1 - (smooge) - At each release we rebuild all the packages, with a %dist_tag of .epel8_Y where “Y” is the latest stable release of RHEL. All of the older packages are archived (and probrubly tagged).

2 - (kevin) - All epel8-stable builds get %dist_tag of .epel8. At each release we tag the full set with epel8_(Y-1)-stable and archive it.

Here are the pro's and con's that I see.
1 - Pro
p-a - EPEL8 packages will always be built with the libraries for their release.
p-b - Broken packages will get fixed or not be in the release
p-c - It is very easy to see what release the package is compatible with, even without koji.
p-d - Maintainers will need to fix bugs instead of just letting their package rot
1 - Con
c-a - Upfront work for someone to rebuild them all
c-b - Maintainers will need to fix bugs instead of just letting their package sit

2 - Pro
p-a - Less upfront work for maintenance team.
p-b - Less work for maintainers
2 - Con
c-a - On your local system, it is harder to tell if an EPEL8 package is still compatible with the release you are on.
c-b - Maintainers can let packages rot

@tdawson and @kevin thanks for the review and questions

I think it should be @smooge to @sgallagh (as in thank you sgallagh for doing the work on the proposal) and (sgallagh's) proposal

Gallagher could you agree/disagree with tdawson's questions?

Thanks for writing this up smooge!
A few comments:

This branch will be left alone until and unless the packager decides
that they want to stage a major (possibly incompatible) change for the
next RHEL 8.Y minor release. At that time, they will need to remove
the package.cfg file from the epel8 branch and manually merge the
proposed changes desired into the epel8-rawhide branch and build
there.

I think we should also leave this open to people who want to build things that are just inherently unstable or rapidly changing.
ie, for those things they will always have a rhel8-rawhide package and may not have a epel8 package ever.

That's for EPSCo to decide on. The implementation would be the same; this is a policy decision.

The side-effect here is that the build will also produce a build that goes to the epel8-rawhide repository without Bodhi intervention.

We may want to talk to rawhide gating folks about setting up or even using epel8-rawhide as a testing ground for gating.

+1

Packages built against epel8-rawhide-candidate will be built with a
%dist tag of .epel8_rawhide
Packages built against epel8-rawhide-candidate will be built with a
%dist tag of .epel8_Y where “Y” is the latest stable release of RHEL
8.

Does this then mean we have to mass rebuild every single epel package at minor releases? Thats adding a bunch of work on us I think... so, I'd prefer to avoid that and just use '.epel8' for everything...

No, a mass-rebuild isn't necessary. It's not a requirement that this gets bumped. It's about the below statement:

This dist tag structure ensures that the version of the package in the
stable epel8 repository will win out over the one in the epel8-rawhide
repository if all other aspects of the EVR are the same. (So one would
only pick up a newer version from epel8-rawhide if it was indeed a
higher version number.)

Well, the epel8-rawhide one should always be a newer version if there is a seperate one no? Or else doing a seperate build for rawhide makes little sense.
Side note: we really need a better name than epel8-rawhide... overloading rawhide is bad I think.

Don't we generally want people to stick with whichever version is in the stabilized branch if everything else is equal? It's always possible that behavior can change in epel8-rawhide depending on what's in the buildroot. So if someone does yum update foo --enablerepo epel8-rawhide because they need something from it right now, I would want them to end up certain to be back on the stable branch as soon as it becomes available. Is that wrong?

I suppose we can also do this with an unchanging %dist tag as long as we make sure it sorts higher than whatever we end up calling epel8_rawhide.

At 00:00 UTC of the day following a new RHEL 8.Y release, an updated
epel-release package will be pushed, updating the %dist tag to the new
.epel8_Y value. All new builds will thus have the new dist tag. A
script will be run at this time to apply a new Koji tag (epel8-8.Y) to
the latest build of a package with one of the following tags: [
epel8-stable, epel8-stable-pending ]. A compose of the epel8.Y
repository will be created at this time from all packages currently
tagged as epel8-8.Y.

I think we should use the method fedora does: after rhel8.Y comes out, we tag all current stable builds with rhel8_Y in koji. Then we know exactly what they were and can compose the archive set (or just take the last composedset and copy it). If we don't change dist, we also don't need to update epel-release all the time for it. :) The next updates pushes will contain packages built against the new minor release. (and any old ones not yet rebuilt).

I don't see a problem with the tagging.

So I can keep this straight in my head, I'm going to summarize both proposals.
1 - (smooge) - At each release we rebuild all the packages, with a %dist_tag of .epel8_Y where “Y” is the latest stable release of RHEL. All of the older packages are archived (and probrubly tagged).

I didn't actually mandate a mass-rebuild at each minor release point. I hadn't considered the "Maintainers will need to fix bugs instead of just letting their package sit" case, but that may indeed make some sense.

Don't we generally want people to stick with whichever version is in the stabilized branch if everything else is equal? It's always possible that behavior can change in epel8-rawhide depending on what's in the buildroot. So if someone does yum update foo --enablerepo epel8-rawhide because they need something from it right now, I would want them to end up certain to be back on the stable branch as soon as it becomes available. Is that wrong?
I suppose we can also do this with an unchanging %dist tag as long as we make sure it sorts higher than whatever we end up calling epel8_rawhide.

Well, I would expect we want epel8_playground to be 'newer' than whats in epel8 so people can upgrade to it and test, but I agree it's tricky then once that version is promoted in stable (the playground one will be 'newer' still). I would submit we just ask people to use distro-sync for this?

example:

epel8 has foo-0.9-9.epel8 in it.
maintainer builds new version in playground: foo-1.0-1.epel8_playground
people test and it looks good, 8.1 comes out and we push that update out as foo-1.0-1.epel8.
people who installed the playground one however don't see an update and have to distro-sync

I think the only way around this would be to require another bump of N-V-R on moving to stable, but that seems... hacky.

so, some more questions around minor release time.

Say we have a 8.0 setup with a updates and updates--testing repos and epel8-playground
8.1 is released.

We can tag all the existing epel8 packages with 'epel8_0' and archive them.
What do we do with all the pending testing updates? Archive them also? Obsolete them all?
The major items that were being played with in epel8_playground, how do we promote them? Are they epel8 updates? Do they bypass the updates process?
Perhaps they use it and the actual big changes land 2 weeks after the minor point?

I guess I would say we should revoke all updates at that point so maintainers can at least rebuild against the new minor release and we should just note that in the next 2 weeks major items may land.

Also, for epel8 and epel8_playgrounds, do we have seperate buildroots? I think we do... they both inherit from rhel8-build, but after that, epel8 is like current epel's: stable updates and overrides only in the buildroot, while epel8-playground has all packages in it in the buildroot as soon as they are built.

Any other process/things around this we need to decide?

BTW, @mattdm was ok with us using playground... so, lets just do that? epel8_playground

I am +1 to epel8_playground

+1 epel8_playground

+1 for epel8_playground (although I thought of epel8_pilot).

example:
epel8 has foo-0.9-9.epel8 in it.
maintainer builds new version in playground: foo-1.0-1.epel8_playground
people test and it looks good, 8.1 comes out and we push that update out as foo-1.0-1.epel8.
people who installed the playground one however don't see an update and have to distro-sync
I think the only way around this would be to require another bump of N-V-R on moving to stable, but that seems... hacky.
so, some more questions around minor release time.
Say we have a 8.0 setup with a updates and updates--testing repos and epel8-playground
8.1 is released.
We can tag all the existing epel8 packages with 'epel8_0' and archive them.
What do we do with all the pending testing updates? Archive them also? Obsolete them all?
The major items that were being played with in epel8_playground, how do we promote them? Are they epel8 updates? Do they bypass the updates process?
Perhaps they use it and the actual big changes land 2 weeks after the minor point?
I guess I would say we should revoke all updates at that point so maintainers can at least rebuild against the new minor release and we should just note that in the next 2 weeks major items may land.
Also, for epel8 and epel8_playgrounds, do we have seperate buildroots? I think we do... they both inherit from rhel8-build, but after that, epel8 is like current epel's: stable updates and overrides only in the buildroot, while epel8-playground has all packages in it in the buildroot as soon as they are built.

Its really good to have these questions answered.

Well, I would expect we want epel8_playground to be 'newer' than whats in epel8 so people can upgrade to it and test, but I agree it's tricky then once that version is promoted in stable (the playground one will be 'newer' still). I would submit we just ask people to use distro-sync for this?
example:
epel8 has foo-0.9-9.epel8 in it.
maintainer builds new version in playground: foo-1.0-1.epel8_playground
people test and it looks good, 8.1 comes out and we push that update out as foo-1.0-1.epel8.
people who installed the playground one however don't see an update and have to distro-sync
I think the only way around this would be to require another bump of N-V-R on moving to stable, but that seems... hacky.

I guess I'm okay with folks staying on the version they had and getting moved back to stable whenever the next stable update happens that sorts higher than their playground version.

so, some more questions around minor release time.
Say we have a 8.0 setup with a updates and updates--testing repos and epel8-playground
8.1 is released.
We can tag all the existing epel8 packages with 'epel8_0' and archive them.

Anything that is presently stable or enroute to stable (Bodhi update is requested for stable but not yet pushed).

What do we do with all the pending testing updates? Archive them also? Obsolete them all?

Why would we obsolete them? I think we might want to rerun any CI tests against them in case the minor release somehow breaks them, but otherwise I think it's fine to just let them proceed as normal.

The major items that were being played with in epel8_playground, how do we promote them? Are they epel8 updates? Do they bypass the updates process?
Perhaps they use it and the actual big changes land 2 weeks after the minor point?

I think having a period where the big changes are queued but not landed would be wise. Since we can't know ahead of time when the minor release will come out, I guess this is the only sane way.

I'd suggest a three-week policy, though. Users have one week from the minor release to submit their major changes. They then have the remaining two weeks to get karma or hit the 14-day point, at which they push to stable or withdraw the update until the next minor release.

I wonder if we want to (ab)use the "newpackage" update type (or add a new one for "incompatible" in Bodhi so that we could automate abandoning the update if it doesn't hit stable in time.

I guess I would say we should revoke all updates at that point so maintainers can at least rebuild against the new minor release and we should just note that in the next 2 weeks major items may land.

I think revoking updates would be overkill and annoying to many contributors. RHEL has a strong compatibility promise; it'll be rare that an existing update wouldn't work on a new minor release. I think rerunning any available CI tests is probably sufficient.

Also, for epel8 and epel8_playgrounds, do we have seperate buildroots? I think we do... they both inherit from rhel8-build, but after that, epel8 is like current epel's: stable updates and overrides only in the buildroot, while epel8-playground has all packages in it in the buildroot as soon as they are built.

Yes, that's exactly what I was envisioning.

Any other process/things around this we need to decide?
BTW, @mattdm was ok with us using playground... so, lets just do that? epel8_playground

This was implemented as playground