This seems to be caused by the builders all moving to f40 and picking up: https://fedoraproject.org/wiki/Changes/DNFConditionalFilelists

I am not sure what the best fix is.

We could just add --setopt optional_metadata_types=filelists to the default mock dnf_command... but that will affect all builds. ;(

Oh, so this tallies with why I've been unable to do local mock builds for rhel+epel-8 since... I upgraded to Fedora 40

This seems to be caused by the builders all moving to f40 and picking up: https://fedoraproject.org/wiki/Changes/DNFConditionalFilelists

Yes, it is that feature.

We could just add --setopt optional_metadata_types=filelists to the default mock dnf_command... but that will affect all builds. ;(

Add it only to EPEL ≤ 9 mock configs.

(That's the same story as with createrepo_c compression. If you want to use one tool for all distributions, you need to be able to configure that tool per distribution.)

ll /usr/libexec/platform-python
lrwxrwxrwx. 1 root root 20 abr 24 22:38 /usr/libexec/platform-python -> ./platform-python3.6

rpm -qf /usr/libexec/platform-python -i 
Name        : platform-python
Version     : 3.6.8
Release     : 62.el8
Architecture: x86_64
Install Date: qui 27 jun 2024 00:54:44 WEST
Group       : Unspecified
Size        : 41214
License     : Python
Signature   : RSA/SHA256, ter 30 abr 2024 14:08:23 WEST, Key ID 05b555b38483c65d
Source RPM  : python3-3.6.8-62.el8.src.rpm
Build Date  : qua 24 abr 2024 23:15:32 WEST
Build Host  : x86-04.stream.rdu2.redhat.com
Relocations : (not relocatable)
Packager    : builder@centos.org
Vendor      : CentOS
URL         : https://www.python.org/
Summary     : Internal interpreter of the Python programming language
Description :
This is the internal interpreter of the Python language for the system.
To use Python yourself, please install one of the available Python 3 packages,
for example python36.

I'm using Centos Stream, platform-python is available what is the version you got ?

also could be because many repos moved to vault due epel-next EOL

cat templates/centos-stream-8.tpl
44:baseurl=https://vault.centos.org/centos/$releasever-stream/BaseOS/$basearch/os/
52:baseurl=https://vault.centos.org/centos/$releasever-stream/AppStream/$basearch/os/
66:baseurl=https://vault.centos.org/centos/$releasever-stream/centosplus/$basearch/os/
73:baseurl=https://vault.centos.org/centos/$releasever/cr/$basearch/os/
88:baseurl=https://vault.centos.org/centos/$releasever-stream/extras/$basearch/os/
96:baseurl=https://vault.centos.org/centos/$releasever-stream/PowerTools/$basearch/os/
104:baseurl=https://vault.centos.org/centos/$releasever-stream/RT/$basearch/os/
112:baseurl=https://vault.centos.org/centos/$releasever-stream/HighAvailability/$basearch/os/
119:baseurl=https://vault.centos.org/centos/$releasever-stream/Devel/$basearch/os/
126:baseurl=https://vault.centos.org/centos/$releasever-stream/BaseOS/Source/
133:baseurl=https://vault.centos.org/centos/$releasever-stream/AppStream/Source/
140:baseurl=https://vault.centos.org/centos/$releasever-stream/PowerTools/Source/
147:baseurl=https://vault.centos.org/centos/$releasever-stream/extras/Source/
154:baseurl=https://vault.centos.org/centos/$releasever-stream/centosplus/Source/

other tip , also today koschei started to fail on mlt epel8

https://koschei.fedoraproject.org/package/mlt?collection=epel8

ok. I worked around this by just adding 'platform-python' to build and srpm-build groups for epel8-build in koji.

I'd suggest someone file a mock bug on the mock part to get that working for local builds?

This work around doesn't seem to work.

https://koji.fedoraproject.org/koji/taskinfo?taskID=120398268

It's failing at the step of installing python3-dnf and dnf-plugins-core, before the group install commands.

...
DEBUG util.py:558:  Executing command: ['/usr/bin/dnf-3', '--installroot', '/var/lib/mock/epel8-build-52225935-6249544-bootstrap/root/', '--setopt=install_weak_deps=0', '--disableplugin=local', '--disableplugin=spacewalk', '--disableplugin=versionlock', 'install', 'python3-dnf', 'dnf-plugins-core', '--setopt=tsflags=nocontexts'] with env {'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOME': '/var/lib/mock/epel8-build-52225935-6249544-bootstrap/root/installation-homedir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'PROMPT_COMMAND': 'printf "\\033]0;<mock-chroot>\\007"', 'PS1': '<mock-chroot> \\s-\\v\\$ ', 'LANG': 'C.UTF-8', 'LC_MESSAGES': 'C.UTF-8'} and shell False
DEBUG util.py:461:  Unable to detect release version (use '--releasever' to specify release version)
DEBUG util.py:461:  No matches found for the following disable plugin patterns: local, spacewalk, versionlock
DEBUG util.py:463:  Last metadata expiration check: 0:00:07 ago on Fri Jul 12 23:08:55 2024.
DEBUG util.py:461:  Error: 
DEBUG util.py:461:   Problem 1: conflicting requests
DEBUG util.py:461:    - nothing provides /usr/libexec/platform-python needed by python3-dnf-4.0.9.2-5.el8.noarch from build
...

This is an indication that koji is running mock with the bootstrap image feature disabled. I confirmed this in the koji_builder ansible role, which has this setting:

config_opts['use_bootstrap_image'] = False

The error can be reproduced locally by running this command in a distgit checkout:

mock -r rhel+epel-8-x86_64 --spec *.spec --sources . --no-bootstrap-image

I can get a build to work with the bootstrap image disabled by enabling the filelists metadata, but only if I explicitly run a scrub first (which is a bit unwieldy on the command line because you have to specify all dnf common options as separate flags, and you can't just extend the default ones).

mock -r rhel+epel-8-x86_64 --scrub all
mock -r rhel+epel-8-x86_64 --spec *.spec --sources . --no-bootstrap-image \
    --config-opts dnf_common_opts='--setopt=deltarpm=False' \
    --config-opts dnf_common_opts='--setopt=allow_vendor_change=yes' \
    --config-opts dnf_common_opts='--allowerasing' \
    --config-opts dnf_common_opts='--setopt=optional_metadata_types=filelists'

The bootstrap image feature was disabled in koji last August, but it's not clear from that commit message why it was needed. The easiest fix would be to remove that setting to go back to the default enabled state. This sort of this is essentially why that feature was created. If there is some other reason why it still needs to be disabled, we can enable the filelists metadata in the config.

config_opts['dnf_common_opts'] = ['--setopt=deltarpm=False', '--setopt=allow_vendor_change=yes', '--allowerasing', '--setopt=optional_metadata_types=filelists']

I can send a PR for either of these fixes depending on which one is preferred.

I'll also point out that this works in mock with the default config, so I'm not sure a bug is needed for that. I suspect if one is filed the answer will be "that is the expected outcome and the reason we added the bootstrap image feature".

Odd.. I tested it and it worked... will take a look more when I get a chance.

We have never used bootstrap images, we disabled them when they were implemented. They present a number of problems (like network access from builders, etc).

We can't set an arbitrary mock config with koji. We can set a global mock site config that affects all builds, but koji makes the mock configs.

ok, so we could:

1) enable bootstrap images. However, this would require changes in builder firewalls (especially the RHIT controlled site filewall on the vlan). It also would be anoying because registry.access.redhat.com is a akami endpoint and it's ip(s) can change at any point. Additionally, it would add a network dependence, we couldn't do builds if we couldn't reach that endpoint. We could of course copy the ubi-8 image somewhere, but then we would need to keep it in sync. All doable, but... not quickly or without discussion.

2) Get a patch in koji to allow us to set optional_metadata_types=filelists as a tag option. This should be pretty easy to add, but of course we need to get it in and revewed and then carry a local patch or wait for the next koji release. This seems to me worth doing anyhow, as other koji instances may hit this and need this option on a per tag basis.

ok, so I think the only short term option is to enable optional_metadata_types=filelists globally in site_defaults.
I am testing that in staging now... but koji in staging seems... off. I wonder if it's not time to repave it and do a resync with prod.
(but I will hold off on that for now as I know carl is testing epel10 things there right now)

ok, here is a scratch build in prod using the above:

https://koji.fedoraproject.org/koji/taskinfo?taskID=120438954

it completed ok. Can others confirm the workaround?

ok, here is a scratch build in prod using the above:

https://koji.fedoraproject.org/koji/taskinfo?taskID=120438954

it completed ok. Can others confirm the workaround?

I did a quick EPEL 8 scratch build and it seems to be working as expected, thanks!

Thanks a lot @kevin ! Two successful builds [1] [2], workaround LGTM :)

[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=120525653
[2] https://koji.fedoraproject.org/koji/taskinfo?taskID=120525701

Why don't we add an explicit Provides for the path to the platform-python package?

If thats a change that can happen at this point to rhel8 platform-python, sure!

Let's see: https://issues.redhat.com/browse/RHEL-48605

(If it happens, it will take a while, so a workaround is probably still needed for now.)

Why don't we add an explicit Provides for the path to the platform-python package?

Because that will only fix the case of /usr/libexec/platform-python. There can be other packages which rely on file dependencies. Simply put until RHEL 10 the file dependencies are granted.

There can be other packages which rely on file dependencies.

This got me curious about what this actually looked like. There are 345 packages in RHEL 8 that require or build require /usr/libexec/platform-python. There are 84 more in EPEL 8. If you expand the search to everything that requires or build requires /usr/libexec/*, there is only one additional package: gnome-keyring depends on /usr/libexec/gcr-ssh-askpass.

In RHEL 9, there is no platform-python, although python3 does provide a symlink. The are only two packages that have file dependencies in /usr/libexec/: insights-client requiring /usr/libexec/platform-python, and gnome-keyring requiring /usr/libexec/gcr-ssh-askpass.

While it's true that there may be additional packages outside RHEL and EPEL that use path dependencies, based on the available data I think it's safe to say this would be a vanishingly small use case. But I would love to hear about examples if anyone is aware of any.

For the general mock use case, this isn't even a problem because the bootstrap image feature is enabled by default. For the releng use case with that feature disabled, it's only a problem for the early python3-dnf/dnf-plugins-core installation step, which is using the host dnf (F40 in this case). Subsequent steps use RHEL 8's dnf within the chroot.

Based on all of this, I think it's fair to say that adding an explicit provides to platform-python in RHEL 8 is a quite comprehensive solution, if we can get it accepted. I don't think it would be appropriate to hold back progress in Fedora by reverting the change to accommodate RHEL, especially since it only affects a non-default mock configuration and multiple workarounds exist.

The platform-python was a hack I put in place in order to get EPEL-8 out the door due to the initial strong modularity focus EL8 had. I would look at any package which uses platform-python now with an eye of 'can it use python3.12 instead?'

MR: https://gitlab.com/redhat/centos-stream/rpms/python3/-/merge_requests/12

The platform-python was a hack I put in place in order to get EPEL-8 out the door due to the initial strong modularity focus EL8 had.

I agree.

I would look at any package which uses platform-python now with an eye of 'can it use python3.12 instead?'

I disagree. This is dnf. It MUST use platform-python on RHEL 8.

So, I don't think there's anything here left for releng/infra to do right?

If so, please reopen and let us know.

platform-python-3.6.8-67.el8_10 has been released, which includes the /usr/libexec/platform-python provides. I verified that the reproducer I used in this comment now works fine. Here's a PR to drop the workaround in staging, and if it works there we can do the same in production.