And Miro pointed out:

the default mock config used for fedpkg --release epel8 mockbuild should IMHO be decided by EPSCo 🧑‍⚖️ and not by fedpkg engineers who are not EPEL packagers at all (sorry if I got that claim wrong, it's based on my view as an outsider)

which should be decided here as well. And then passed to fedpkg maintainers.

My opinion: Make Alma Linux the default, not RHEL. I would settle for Rocky Linux.
Setting the default epel mock to RHEL adds complications.

If you are not going to set it to Alma, or Rocky, then I propose get rid of it entirely, and make sure there are almaepel and rockyepel configs. There are already rhelepel configs.

should be decided here as well. And then passed to fedpkg maintainers.

Absolute agreement. I just don't have the processes sorted out in my head,
so the first reaction was to open a ticket against fedpkg.

For an idea of the impact of no ppc64le or s390x using Alma, here are the numbers from the countme data for last week:

epel-8 users, arch
687, s390x
1823, ppc64le
30005, aarch64
1351645, x86_64

Of those 1823 ppc64le users, there are 1423 using Red Hat Enterprise Linux and 238 using CentOS.

If you are not going to set it to Alma, or Rocky, then I propose get rid
of it entirely, and make sure there are alma epel and rockyepel configs.
There are already rhelepel configs.

Removing the configs will have the same, if not worse effects than
adjusting it. It's a compromise, not a correct solution. So as I claimed
elsewhere (see Neal's citations), it's a good opportunity to finally alter
the configuration to its real "native" EPEL meaning, I would change
the opinion if Alma was considered to be the base for Koji's EPEL.

I don't want to blindly promote RHEL (and I wish any fork wasn't
promoted by others), the change would actually bring some
inconvenience to mock users. But also 100% honesty, ... how many
mock users have an idea they are building locally against
CentOS (in Copr we at least have the chroot icon)? Shouldn't they
know they are building against Alma?

If you are not going to set it to Alma, or Rocky, then I propose get rid
of it entirely, and make sure there are alma epel and rockyepel configs.
There are already rhelepel configs.

Removing the configs will have the same, if not worse effects than
adjusting it. It's a compromise, not a correct solution.

Yep, about a minute after I wrote that I realized that would break alot of things.
Let's just call my "toss it out" a bad idea and move on.

I don't want to blindly promote RHEL (and I wish any fork wasn't
promoted by others), the change would actually bring some
inconvenience to mock users. But also 100% honesty, ... how many
mock users have an idea they are building locally against
CentOS (in Copr we at least have the chroot icon)? Shouldn't they
know they are building against Alma?

There is honesty, and then there is getting a task done.

How many mock users "care" that they are building against CentOS.

I for one knew that mock was building against CentOS when I did a local epel build. I also knew that there was rhelepel and I would use it if I wanted to build against epel and rhel.

Regardless of what the plain epel config turns out to be, I think we need almaepel and rockyepel configs.

Regardless of what the plain epel config turns out to be, I think we need almaepel and rockyepel configs.

• alma+epel-8-aarch64.cfg
• alma+epel-8-x86_64.cfg
• rhel+epel-8-aarch64.cfg
• rhel+epel-8-ppc64le.cfg
• rhel+epel-8-x86_64.cfg
• oraclelinux+epel-8-aarch64.cfg
• oraclelinux+epel-8-x86_64.cfg

Any other RHEL clones are welcome to submit similar configs to mock upstream.

I think there are only 2 reasonable defaults: Rocky Linux or AlmaLinux (just pick whatever of those works better in practice, it should not really matter). Anything that requires a subscription of any sort, even a free-as-in-beer one, should be a non-starter. The default EPEL config needs to just work and to be free of field-of-use restrictions, including technically enforced policy restrictions breaking use cases such as emulation. So requiring a RHEL developer subscription is not an acceptable solution. The discrepancy with the EPEL Koji and/or Copr configuration can easily be fixed by changing those, too.

I think there are only 2 reasonable defaults: Rocky Linux or AlmaLinux (just pick whatever of those works better in practice, it should not really matter).

While it may technically not matter, there is a difference in the organizational structure/governance. The AlmaLinux OS Foundation is a 501(c)(6) non-profit [0] and its Board [1] is not controlled by CloudLinux. The Rocky Enterprise Software Foundation (RESF) is a Public Benefit Corporation owned by one person [2]. IMHO that makes AlmaLinux the better choice as the default as it better aligns wit Fedora's Freedom value.

[0] https://almalinux.org/
[1] https://almalinux.org/foundation/members/
[2] https://rockylinux.org/organizational-structure/

It is fun because I have the opposite impression, AlmaLinux being a corporate distribution (even with the non-profit foundation inbetween) and Rocky Linux a community one. But I am fine with AlmaLinux being the default, let us not bikeshed over that.

We discussed this at today's EPEL Steering Committee meeting.

• minutes
• log

We ran out of time and did not reach a decision. We'll revisit this issue at next week's meeting.

I wrote down the options and try my best to write down all pros and cons
https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SGy4VxUA/edit?usp=sharing
Feel free to add technical comments in that doc and I will amend it.

So, one other option (5?):

Treat epel like a 'add on' to other el configs

That is, remove the existing epel configs (or replace with a doc pointer/readme) and make it so you can include epel in any of the compatible el configs (rhel, rocky, alma, etc).I guess with a commented / disabled by default entry in those configs? Or all of them including a config that could be enabled for them all?

Hmmm…

One one hand, I like this in principle, because it is the most logical choice. EPEL has never been a distribution, after all, just an add-on repository. (Would mock -r alma-8-x86_64 --enablerepo=epel make sense as a CLI syntax? And enabling it by default being just a matter of changing enabled=no to enabled=yes in the .cfg file? And would we also allow something like epel-8-x86_64.cfg containing:

include alma-8-x86_64
enablerepo epel

or similar?)

On the other hand, this still means that the existing epel-8-x86_64.cfg is going to be removed, so it will have the same issues as the option proposing just that:

• Should it be replaced with a symlink (or include or something)? To what?
• If you do not provide a symlink/include/… at all, it will be a backwards-incompatible change for users and break their scripts.
• What should fedpkg default to for EPEL branches? (Enabling EPEL is a no-brainer there, but on which config?)

Let's look at history. The mock epel config has use CentOS for many years before before the CentOS team came to Red Hat.
It originally was pointed at Fedora for the base os, and then in sometime around 2007 and 2008 it switched from Fedora to CentOS Linux.

From what I see, it has always been "always allow anyone to use it, but it doesn't have to be perfect" .

We now have a way for it to be perfect, rhelepel. Great. But that puts barriers in front of many people.
We can rip it out. Great. But that also puts barriers in front of many people, and breaks current configs.
Or we can follow in the original spirit of the mock epel config. "always allow anyone to use it, but it doesn't have to be perfect". My opinion on this is to use Alma or Rocky Linux to fulfill the spirit of the original configs. My preference is Alma, but I'm fine with either.

We just had our EPEL Steering Committee meeting. It was a good discussion, and we believe we came up with a solution.
Before I get to the solution, there were two major concerns that steered the discussion towards this solution. Out Of The Box (OOTB) feel for developers. Favoritism to any certain clone / distribution.

In the end it was decided that it would be best to remove the default epel configs and put several things in place to still have a good OOTB experience for developers.

1 - patch mock so that if the default epel config was called, instructions would be given to easily choose what to do.[1]
2 - Have rpm triggers set the default epel links when the choice is clear.
So on an Alma Linux machine, the default epel link would point to alma+epel, Rock Linux would point to rocky+epel, RHEL to rhel+epel, and so forth. These triggers could be on the <distro>-release package that trigger when mock is installed, or they could be on mock when it sees <distro>-release. Where to put the triggers wasn't determined.
3 - patch fedpkg so that a default mock config could be set instead of hard coding it. Patching mock will lessen the need for this, but we still think it's a good idea.

Note: If I misunderstood what we agreed on, please correct me.

[1]

This is alternative behavior
============================

$ mock -r epel-8-x86_64 --rebuild my-beloved.src.rpm
ERROR: epel-8-x86_64 configuration doesn't exist

There are those available configs:

[1] /etc/mock/alma+epel-8-x86_64.cfg
    Use by: 'mock -r alma+epel-8-x86_64 ...'
    You will build against AlmaLinux together with the official EPEL.
[2] /etc/mock/oraclelinux+epel-8-x86_64.cfg
    Use by: 'mock -r oraclelinux+epel-8-x86_64 ...'
    You will build against Oracle Linux together with the official EPEL.
[3] /etc/mock/rhel+epel-8-x86_64.cfg - Red Hat Enterprise Linux + official EPEL, requires a subscription
    You will build against Oracle Linux together with the official EPEL.  This
    is closest to the official EPEL builds, but RHEL subscription is needed.

Or you can install a permanent symlink to one of those configurations
as ~/.config/mock/epel-8-x86_64.cfg and re-try the same mock command.

// interactive terminal only
Do you want to install a concrete epel-8 variant you prefer? (number, or just hit enter cancel):


Configuration
=============

config_opts["non-chroots"] = {}
config_opts["non-chroots"]["epel-8-*"] = {
    "alma+epel-8": {
        "description": " ... the description ...",
    },
    "rhel+epel-8": {
        "description": " ... the description ...",
    },
}

3 - patch fedpkg so that a default mock config could be set instead of hard coding it. Patching mock will lessen the need for this, but we still think it's a good idea.

Not just lessen the need, it will eliminate it. Once mock is patched to output the above message when epel-8-* is not present, then fedpkg mockbuild will inherit it with no changes because it already defaults to epel-{releasever}-{arch}.

I disagree with patching Mock. Mock shouldn't have special logic for any configs. Tools that build on Mock should have that logic.

This "favoritism" argument is absurd. Those configs used to hardcode CentOS even when there were other rebuilds such as Scientific Linux around. And that long before CentOS was absorbed by Red Hat. Yet, nobody complained about that.

I think just picking a reasonable default for Fedora users (which should be one that works out of the box and in all usage contexts, i.e., does not require a subscription of any kind) would be much simpler than adding complicated logic to mock and/or fedpkg. (And I disagree with @ngompa over the assertion that pushing the logic out of Mock into higher layers is a solution, because users also use Mock directly and expect that to just work.)

This "favoritism" argument is absurd. Those configs used to hardcode CentOS even when there were other rebuilds such as Scientific Linux around. And that long before CentOS was absorbed by Red Hat. Yet, nobody complained about that.

I think just picking a reasonable default for Fedora users (which should be one that works out of the box and in all usage contexts, i.e., does not require a subscription of any kind) would be much simpler than adding complicated logic to mock and/or fedpkg.

Obviously, I prefer us just having a good default (like AlmaLinux as I've proposed).

(And I disagree with @ngompa over the assertion that pushing the logic out of Mock into higher layers is a solution, because users also use Mock directly and expect that to just work.)

When you're using Mock directly, you have to select a chroot config. When using fedpkg(1), it auto-selects a chroot config.

I'm confused, so "no epel-8 config" is not an option now? Or that the current message is more user-friendly?

$ mock -r non-existing-x86_64 --shell
ERROR: Could not find required config file: /etc/mock/non-existing-x86_64.cfg
ERROR:   If you're trying to specify a path, include the .cfg extension, e.g. -r ./target.cfg

For the record, the discussion is here https://meetbot.fedoraproject.org/fedora-meeting/2021-12-01/epel.2021-12-01-21.00.log.html starting at 21:28:55

Alright, per the new decision, here's a new PR: https://github.com/rpm-software-management/mock/pull/814

I'm confused, so "no epel-8 config" is not an option now? Or that the current message is more user-friendly?

This is sufficient, fedpkg(1) will need work to give people the ability to configure an distro+epel config to use.

I still think patching Mock in this case is appropriate, what's wrong on that?

I still think patching Mock in this case is appropriate, what's wrong on that?

If we add a user-friendly message, we should add a generally applicable friendly message. I don't think it's appropriate for Mock to do weird things for a specific subset of configs.

You know best that I give anyone enough time to review my pull requests, so
whatever is found inappropriate can be fixed. I wish you could give us some
time to review, too.
It may look like I don't feel it, but it's not joyful to hear that whatever I propose
is unfriendly. :-(

The promised pull-request is here:
https://github.com/rpm-software-management/mock/pull/817

Pull request has been merged.
Built and tagged and in testing in bodhi.
There is negative karma on the updates, looks like some more work needed in rpkg, but from the epel issue perspective it's "resolved"