I researched the issue and the problem is that CentOS Stream 8 pushed a new python2 for 8.7.0 which only contains about 1/2 of the packages which the older ones did. All of the debug and development items do not seem to be in it. This means that packages won't build or install in it.

Bad module python27:2.7:8070020220624173536:056aacbc:x86_64 has only 37 packages in it. And the last good module python27\:2.7\:8060020220426160402\:8cdc2268\:x86_64/ has 62 with many devel items.

Now this may be a planned change inside of CentOS Stream to get ready for the EOL of the python2 module in 2024. It may also have been a broken push. Possible scenarios to fix:

1. Do nothing and wait until CS8 team (@bstinson ?) clarifies the problem. Have someone open a ticket in the correct place for them to track it.
   a. if this planned then start building epel-* packages which will replace the missing ones.
   b. if this was not planned work with Stream releng to get working module into release stream.
2. Do surgery with the koji buildroot to work around this: (from my irc notes)

 releng will need to turn off the cron job which makes the koji root. They will then need to copy in the directory from 2022-06-28 into the /mnt/fedora/app/fi-repo/centos/stream8-kojitarget/latest and remove the bad python2 directory. They will then need to rerun createrepo and test

Here is the relevant part of the modulemd that I took from http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/repodata/6ce7a4d5107d4bbf9a62901c35deaf7c2c8de0925ddb025c15ac2dacd5c88e63-modules.yaml.xz . This is the modulemd for the last good module.

FWIW, python2-rpm-macros is still in the module api, but the actual rpms are not shipped.

From IRC #centos-stream

[2022-07-02-12:21] <carlwgeorge> smooge: I suspect the latest module didn't pick up all the artifacts.  it's happened before and afaik it's an mbs bug.  will check in a minute.
[2022-07-02-12:21] <carlwgeorge> modularity - the gift that keeps on giving
[2022-07-02-12:28] <carlwgeorge> yup, missing artifacts in the latest python27 module as compared to the rhel8 one
[2022-07-02-12:30] <carlwgeorge> i've triggered a module rebuild to hopefully pick up all the correct artifacts

Next compose and push of modules will hopefully get this.

I want to confirm that we haven't deliberately removed any packages from the python27 module in RHEL 8.

I proposed a change in epel-rpm-macros that prevents non-Python 2 packages from being broken by modularity issues like this one, see https://src.fedoraproject.org/rpms/epel-rpm-macros/pull-request/44#comment-109676

I want to confirm that we haven't deliberately removed any packages from the python27 module in RHEL 8.

It looks like @carlwgeorge came to the conclusion that this was an error on the CentOS Stream side and it wasn't intentionally removed.

If that was the case, we'd have a bit of a problem on our hands:

$ sudo dnf repoquery -q --repo=epel8{,-source} --whatrequires python2-devel |grep '\.src$' | wc -l
87

I am so sad that so many packages were imported to EPEL 8 with a dependency on Python 2 :(

Well what is done is done. Most of the consumers of EPEL were moving from EL5 and EL6 and would have had software requirements on 2 items that needed to keep working.

At this point what needs to be done is go through the packages and start giving them an EOL of 2024 when RHEL8 no longer supports python2.

This has been fixed as of the 20220702.n.0 CentOS Stream 8 compose. Thanks to @carlwgeorge for helping fix it. There's now another issue with the python38 module being broken, but that's not something that can be fixed here.