Learn more about these different git repos.
Other Git URLs
Earlier today we switched from the koji rpmautospec plugin to the mock rpmautospec plugin. From what I can tell, this approach requires rpmautospec to be available for each release and installed in each chroot. However, the rpmautospec epel10 build failed because python-click-man and python-pytest-xdist were not available. This meant that rpmautospec was not available for epel10 builds, and epel10 builds started failing once we made this switch (one example).
To get epel10 builds working again, I set out to get rpmautospec available in the release. Attempting to install the Fedora 40 package in a CentOS 10 container showed me several missing run-time dependencies. I had access to python-click-plugins via the epel-packagers-sig, and built that normally and created a bodhi override. I had to manually tag the remaining run-time dependencies from f40, along with rpmautospec itself, to epel10.0-override.
koji tag-build epel10.0-override \ http-parser-2.9.4-11.fc40 \ libgit2-1.7.2-1.fc40 \ libssh2-1.11.0-4.fc40 \ python-pygit2-1.14.0-1.fc40 \ python-rpmautospec-0.7.1-1.fc40
I noticed there is also a python-rpmautospec-core override which expires on 2024-08-30. I set my python-click-plugins override to expire on the same day. When those expire, epel10 builds will be broken again because rpmautospec will not be able to be installed in chroots. We must get this sorted out properly soon.
I have a few questions for specific people to determine our direction on this.
@tdawson @sgallagh python-rpmautospec is built in the CentOS koji. What are the chances of including it in the base distro (i.e. CentOS and RHEL)? That would be quite beneficial for EPEL to avoid having to bootstrap this with each new major EPEL version.
@nphilipp I can see that the python-rpmautospec spec file is using %autorelease and %autochangelog. If we can't add it to the base distro and have to build it from scratch for each new major EPEL version, won't this be a self-imposed recurring bootstrap problem? If we do need to bootstrap this every three years, I'd at least like to see the infra-sig added to all of these dependencies to make it easier to work on.
Before the bodhi overrides expire on 2024-08-30.
When RHEL 10 is EOL (approximate 2035-05-31).
EPEL 10 builds will be blocked.
For the record, c10s python-rpmautospec is in builrdoot-only and we would need it in CRB.
I can see that the python-rpmautospec spec file is using %autorelease and %autochangelog. If we can't add it to the base distro and have to build it from scratch for each new major EPEL version, won't this be a self-imposed recurring bootstrap problem?
I don’t think so – redhat-rpm-config ships with /usr/lib/rpm/macros.d/macros.rpmautospec which contains fallback macros (%autorelease expands to 1%{?dist}, %autochangelog to a mock changelog entry). This should be sufficient to bootstrap.
redhat-rpm-config
/usr/lib/rpm/macros.d/macros.rpmautospec
%autorelease
1%{?dist}
%autochangelog
Unfortunately, that's not going to work. rpmautospec has dependencies that we (Red Hat) are unwilling to support in RHEL.
I'll look into getting the remaining deps working in EPEL 10 though.
Anyway, I've been building some %autorelease packages in EPEL 10 and it seems to work. See e.g.
How is that possible?
Yes, this is what I was trying to describe when I mentioned it was built in the CentOS koji but not included in the base distro. I should have been a little more specific that I was talking about what EPEL considers the base distro, i.e. what we build against (BaseOS, AppStream, and CRB). Including it in CRB would be the easiest solution, but based on @sgallagh's reply it seems we won't be able to go that route.
That sounds like it may help with, but not eliminate, the bootstrap problem. What would the bootstrap process look like? If I understand things right, the shift from a koji plugin to a mock plugin seems to mean that rpmautoautospec must be available to be installed in each chroot. Simply having a macro expansion fallback is not enough if not being able to install rpmautospec halts builds. We need to document this process so that we can do it with each new EPEL major version. I can see the infra-sig has access to python-rpmautospec, but ideally we will also add the infra-sig added to all of its dependencies that aren't in RHEL/CentOS BaseOS/AppStream/CRB. Then members of the SIG can request new EPEL major branches and follow the documented bootstrap procedure.
That's what I was afraid of, but thanks for confirming.
Before the switch to the mock plugin, builds using %autorelease were working because the koji plugin was installed on the builders. After the switch to the mock plugin, builds started failing for a period of time. They work now because I tagged the necessary Fedora 40 packages into the epel10.0-override tag as a stop gap until we can get them built properly in EPEL 10.
They work now because I tagged the necessary Fedora 40 packages into the epel10.0-override tag as a stop gap until we can get them built properly in EPEL 10.
I see. You said that in the ticket, I must have brainfarted. Sorry about that.
Metadata Update from @phsmoura: - Issue tagged with: medium-gain, medium-trouble, ops
I've been looking through the dependency tree, and here are the packages I believe we need to ensure infra-sig is added to so we can bootstrap rpmautospec in the future. There are other dependencies, but these are the ones that are not already provided in the RHEL/CentOS target base.
@sgallagh mentioned in matrix that the libgit2 dependency on http-parser has been replaced by llhttp, and the list above reflects that. There is also a pending pull request to python-rpmautospec to remove the python-xdist dependency.
EDIT: adjusting the list above as we go
I've been looking through the dependency tree, and here are the packages I believe we need to ensure infra-sig is added to so we can bootstrap rpmautospec in the future. There are other dependencies, but these are the ones that are not already provided in the RHEL/CentOS target base. python-poetry-core
The RHEL build has this removed and I thought we did the same for EPEL, but it appears that was left in. I'll add that to the merge request
python-click-man
The merge request I submitted also makes this optional for bootstrapping.
python-pytest-xdist
Also getting removed in the merge request
python-rpmautospec-core
Pretty obvious, yeah.
python-click python-click-plugins
Newly added upstream, but we need it.
python-pygit2 libgit2
These are the core libraries it uses to function, no avoiding them at all.
llhttp
This isn't needed in EPEL. It's provided by RHEL 10.
libssh2
This dependency doesn't exist in the RHEL/EPEL builds; it's only on the list because the Fedora build that was tagged in depends on it. It will go away when we build python-rpmautospec with the correct macros enabled.
Final list after my MR lands:
python-rpmautospec-core python-click python-click-plugins python-pygit2 libgit2
Correction, I think we do indeed want the python-poetry-core dependency in EPEL, to stay closer to upstream.
I held a video call with @nphilipp and @sgallagh to talk through the details of fixing this.
Short term action items: - @carlwgeorge will switch from f40 builds to eln builds in the override tag - @nphilipp will file bugzillas for the packages in this comment to request the infra-sig be added as a maintainer. - @carlwgeorge will create EPEL 10 builds of those packages via infra-sig permissions - @carlwgeorge will remove the overrides
Long term action items: - include rpmautospec bootstrap steps in releng/EPEL SOP - rethink how this works with @sgallagh's vision for epel11
@carlwgeorge will switch from f40 builds to eln builds in the override tag
ELN uses Python 3.13. EL 10 uses Python 3.12.
Additional highly opinionated suggestions to make this easier for next time:
I am happy to help with both.
@carlwgeorge will switch from f40 builds to eln builds in the override tag ELN uses Python 3.13. EL 10 uses Python 3.12.
We are aware of the python dep; we'll be pulling in older ELN builds where necessary for the current batch of packages, but for the next time around (EPEL 11), we'll be using ELN to avoid these dependency hiccups.
Indeed, this was something we talked about during the call that I failed to capture in sufficient detail in my comment. I plan to use an older ELN build that was still built against Python 3.12. The motivation for this is to avoid the libssh2 (conditionally disabled in ELN/RHEL) and http-parser (switched to llhttp, part of the target base) dependencies.
Upstream should drop click and replace it with argparse.
This was discussed as a possibility as well. I'm in favor, but it's not a blocker for now. We'll still have a bootstrap process, and the only difference is how many builds we put in the override tag. But it's still a good idea.
Upstream should drop poetry(-core) and replace it with flit(-core) or hatch(ling).
We touched on this briefly because the python-rpmautospec spec file currently has the fallback you're describing. I don't have strong opinions on the build system used other than it would be a clear benefit to use one that is shipped in the target base (which it looks like flit-core and hatchling both are).
As always, happy to have your help, you're awesome.
ELN uses Python 3.13. EL 10 uses Python 3.12. Indeed, this was something we talked about during the call that I failed to capture in sufficient detail in my comment. I plan to use an older ELN build that was still built against Python 3.12. The motivation for this is to avoid the libssh2 (conditionally disabled in ELN/RHEL) and http-parser (switched to llhttp, part of the target base) dependencies.
To be clear, both of those deps are coming only from libgit2, which is not Python. So we don't need an older build for those. If we pull any of the python deps from ELN, they'll be from before the 3.13 builds.
In the future (EPEL 11), we will be doing this pre-branch.
Upstream should drop click and replace it with argparse. This was discussed as a possibility as well. I'm in favor, but it's not a blocker for now. We'll still have a bootstrap process, and the only difference is how many builds we put in the override tag. But it's still a good idea.
I'm deferring this to @nphilipp I tend to agree; we should minimize our dependencies.
Upstream should drop poetry(-core) and replace it with flit(-core) or hatch(ling). We touched on this briefly because the python-rpmautospec spec file currently has the fallback you're describing. I don't have strong opinions on the build system used other than it would be a clear benefit to use one that is shipped in the target base (which it looks like flit-core and hatchling both are).
I dislike poetry intensely and would love to see it go away, but as @nphilipp is the primary upstream maintainer, he's the one to convince.
I am happy to help with both. As always, happy to have your help, you're awesome.
Yes, thank you @churchyard !
Looks like this isn't going to work for reducing the builds in the override tag. The ELN build of libgit2 went from depending on libhttp_parser.so.2 to libllhttp.so.9.2. CentOS Stream 10 only has libllhttp.so.9.1. For now, it seems we will just stick with the f40 builds in the override, and finish up the non-bootstrap builds.
I think we have all the builds we need.
carl ~ ❯ koji list-tagged epel10.0-build-side-94526 Build Tag Built by ---------------------------------------- -------------------- ---------------- libgit2-1.7.2-4.el10_0 epel10.0-build-side-94526 sgallagh python-click-man-0.4.1-18.el10_0 epel10.0-build-side-94526 sgallagh python-pygit2-1.14.0-5.el10_0 epel10.0-build-side-94526 sgallagh python-rpmautospec-0.7.1-2.el10_0 epel10.0-build-side-94526 carlwgeorge
Everything else has already been built or removed. Once this sidetag become an update, and that update is stable, I'll remove the f40 builds from the override tag.
Update from the sidetag: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2024-be26b5b32e
That update has moved to stable, so I untagged the f40 builds from the override tag. I fired off a scratch build of a package that uses %autorelease/%autochangelog, and it completed successfully.
For future EPEL major versions, we'll have to account for bootstrapping rpmautospec in our SOP.
Metadata Update from @carlwgeorge: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.