https://koji.fedoraproject.org/koji/taskinfo?taskID=83566863
DEBUG util.py:444: No matching package to install: 'python38-pytest'
When do you need this? (YYYY/MM/DD) Preferably before the RHEL 8.6 release 2022/05/??
If we cannot complete your request, what is the impact? Cannot build many python38 modules in EPEL8
As one of the RHEL Python maintainers, I confirm that our intention was that python38-pytest is to be available in CRB and hence in the EPEL8 buildroot.
Metadata Update from @mohanboddu: - Issue tagged with: medium-gain, medium-trouble, ops
ok. I spent far too long digging into this tonight. Here's what I have so far:
We have the rhel8-base external repo set in koji with merge mode 'bare'
This causes koji to use /usr/bin/mergerepo_c --pkgorigins --all
This should cause it to include all packages in the merged repodata, but somehow it does not.
python38-pytest is built from a pytest src.rpm.
There is also a python3-pytest rpm built from another older pytest src.rpm.
python38-pytest is in the external repo repodata, but not in the merged repodata in koji once it's merged.
So, I think somehow mergerepo_c is seeing the first one and not including the second one. I am not sure why thought.
I think next step is to make a small repoducer...
@orion until we get this figured out, as a workaround can you just disable tests?
(note also: you will need to mv %{buildroot}/%{_bindir}/jp.py %{buildroot}/%{_bindir}/jp38.py to avoid conflicting with python3-jmespath...)
FWIW I cannot reproduce this way:
├── repo36 │ └── python3-pytest-3.9.1-7.el8ost.1.noarch.rpm └── repo38 └── python38-pytest-4.6.6-3.scrmod+el8.6.0+14528+0d1dce30.noarch.rpm
createrepo_c .
mergerepo_c --repo=file:///.../merge-reproducer/repo3{6,8}/ --pkgorigins --all
I'm sure I can disable tests if I get desperate, but we really can't start producing lots of python38- packages confidently in EPEL8 until this is resolved. Thanks for everyone's efforts.
The python38-devel module is not enabled by default. Is that related to this issue?
python38-devel
On a regular RHEL 8 VM with CRB enabled:
$ sudo dnf install python38-pytest Updating Subscription Management repositories. Last metadata expiration check: 0:04:23 ago on Fri 20 May 2022 06:00:05 PM EDT. All matches were filtered out by modular filtering for argument: python38-pytest Error: Unable to find a match: python38-pytest $ sudo dnf module enable python38-devel Updating Subscription Management repositories. Last metadata expiration check: 0:04:37 ago on Fri 20 May 2022 06:00:05 PM EDT. Dependencies resolved. ==================================================================================================================== Package Architecture Version Repository Size ==================================================================================================================== Enabling module streams: python38-devel 3.8 Transaction Summary ==================================================================================================================== Is this ok [y/N]: y Complete! $ sudo dnf install python38-pytest Updating Subscription Management repositories. Last metadata expiration check: 0:04:46 ago on Fri 20 May 2022 06:00:05 PM EDT. Dependencies resolved. ==================================================================================================================== Package Arch Version Repository Size ==================================================================================================================== Installing: python38-pytest noarch 4.6.6-3.module+el8.4.0+8888+89bc7e79 codeready-builder-for-rhel-8-x86_64-rpms 487 k Installing dependencies: python38-atomicwrites noarch 1.3.0-8.module+el8.4.0+8888+89bc7e79 codeready-builder-for-rhel-8-x86_64-rpms 22 k python38-attrs noarch 19.3.0-3.module+el8.4.0+8888+89bc7e79 codeready-builder-for-rhel-8-x86_64-rpms 71 k python38-more-itertools noarch 7.2.0-5.module+el8.4.0+8888+89bc7e79 codeready-builder-for-rhel-8-x86_64-rpms 60 k python38-packaging noarch 19.2-3.module+el8.4.0+8888+89bc7e79 codeready-builder-for-rhel-8-x86_64-rpms 59 k python38-pluggy noarch 0.13.0-3.module+el8.4.0+8888+89bc7e79 codeready-builder-for-rhel-8-x86_64-rpms 41 k python38-py noarch 1.8.0-8.module+el8.4.0+8888+89bc7e79 codeready-builder-for-rhel-8-x86_64-rpms 202 k python38-pyparsing noarch 2.4.5-3.module+el8.4.0+8888+89bc7e79 codeready-builder-for-rhel-8-x86_64-rpms 160 k python38-wcwidth noarch 0.1.7-16.module+el8.4.0+8888+89bc7e79 codeready-builder-for-rhel-8-x86_64-rpms 34 k Transaction Summary ==================================================================================================================== Install 9 Packages Total download size: 1.1 M Installed size: 4.5 M Is this ok [y/N]: n Operation aborted.
I'll just note that with RHEL8.6 released and ansible-core using python38, my need for python38 modules has gotten much more pressing. I guess I'll just be building them locally for now but would be nice to be able to provide them in EPEL.
If you really wanted to, as a temporary workaround, you could disable tests by default with %bcond_with and then run a local build --with test before pushing to epel. It's possible to enable the python38-devel module in a local mock chroot.
%bcond_with
--with test
ok, with help from @sgallagh and @gotmax23 we finally tracked this down. :)
The way we consume things is to sync them over, then split out the modules. But we have been keeping old versions of modules in the same repo. mergerepo_c sees that there's say 5 versions of python38-pytest (one each in the last 5 versions of the module) that have the exact same name, version, checksum , but NOT mtime. So it yells about it and tosses those out.
So, now we need to either make sure the mtime is the exact same on all the same packages, or stop carrying the old modules along for the ride. We are testing out both approaches, but hopefully this will be fixed in tne next few days.
Things appear to have morphed a bit, now I'm getting:
DEBUG util.py:443: Error: DEBUG util.py:443: Problem 1: conflicting requests DEBUG util.py:443: - nothing provides python38-atomicwrites needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch DEBUG util.py:443: - nothing provides python38-attrs needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch DEBUG util.py:443: - nothing provides python38-more-itertools needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch DEBUG util.py:443: - nothing provides python38-packaging needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch DEBUG util.py:443: - nothing provides python38-wcwidth needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch DEBUG util.py:443: - nothing provides python38-pluggy >= 0.12 needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch DEBUG util.py:443: - nothing provides python38-py >= 1.5.0 needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch DEBUG util.py:443: Problem 2: conflicting requests DEBUG util.py:443: - nothing provides python38-urllib3 >= 1.21.1 needed by python38-requests-2.22.0-9.module_el8.4.0+647+0ba99ce8.noarch
https://koji.fedoraproject.org/koji/taskinfo?taskID=87497860
These are other packages in the python38-devel module except for python38-urllib3 which is part of the python38 module. But it seems is can see python38-pytest from python38-devel and python38-requests from python38.
One possible solution for ensuring the mtime values are all the same: https://github.com/fedora-modularity/GrobiSplitter/pull/7
Glad to see some progress here. Thanks everyone.
It is definitely 100% not fixed yet. :)
I fixed python38-pytest as a test... we have not yet decided on the best fix, not deployed it. ;)
I have worked with @sgallagh and @kevin to put in some fixes to the script. I believe I have it 'working' for at least some test parts. Please test
This appears to have been fixed for epel8 but not epel8-next. Has the script been run for the c8s mirror yet?
epel8
epel8-next
c8s
DEBUG util.py:443: No matches found for the following disable plugin patterns: local, spacewalk, versionlock DEBUG util.py:443: Error: DEBUG util.py:443: Problem: conflicting requests DEBUG util.py:443: - nothing provides python38-atomicwrites needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch DEBUG util.py:443: - nothing provides python38-attrs needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch DEBUG util.py:443: - nothing provides python38-more-itertools needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch DEBUG util.py:443: - nothing provides python38-packaging needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch DEBUG util.py:443: - nothing provides python38-wcwidth needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch DEBUG util.py:443: - nothing provides python38-pluggy >= 0.12 needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch DEBUG util.py:443: - nothing provides python38-py >= 1.5.0 needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch DEBUG util.py:445: (try to add '--skip-broken' to skip uninstallable packages) DEBUG util.py:596: Child return code was: 1 DEBUG util.py:168: kill orphans
https://kojipkgs.fedoraproject.org//work/tasks/5102/87565102/root.log
I don't know how the CentOS side runs. Looking at the directories.. the last time it successfully ran to completion was 2 years ago.
drwxr-xr-x. 5 root root 4096 2022-05-26 10:16 2022-05-26T10/ drwxr-xr-x. 5 root root 4096 2022-05-27 10:16 2022-05-27T10/ drwxr-xr-x. 3 root root 4096 2022-05-28 10:10 2022-05-28T10/ drwxr-xr-x. 5 root root 4096 2020-12-08 19:48 latest/ lrwxrwxrwx. 1 root root 13 2022-05-27 10:21 staged -> 2022-05-27T10/
but that seems wrong so I don't know where koji is looking.
@kevin @carlgeorge @tdawson This needs to be looked at by someone who is more fulltime on EPEL and koji configs. I would be bothering you enough to tell me what to fix :dizzy_face:
Koji is indeed pulling from https://infrastructure.fedoraproject.org/repo/centos/stream8-kojitarget/latest/$arch/CS-8-001/. See https://koji.fedoraproject.org/koji/externalrepoinfo?extrepoID=47.
latest is not a symlink, but the individual CS-8-001 directories (e.g. in https://infrastructure.fedoraproject.org/repo/centos/stream8-kojitarget/latest/x86_64/) are from yesterday. I believe this is how the mirror was configured before your script changes. It looks like the updated cents8-split.sh either failed or didn't run, because the files in the aforementioned directory have varying mtimes.
latest
CS-8-001
cents8-split.sh
It also looks like populating https://infrastructure.fedoraproject.org/repo/centos/stream8-kojitarget/2022-05-28T10/ failed part way through; only aarch64 is present.
aarch64
I will rerun the cent8-split.sh . OK the updated splitter code does not like stream.
# /usr/local/bin/splitter.py --debug --action hardlink --target CS-8-002 /mnt/fedora/app/fi-repo/centos/centos-8-stream/AppStream/aarch64/os/ --only-defaults /usr/lib64/python3.6/site-packages/hawkey/__init__.py:348: DeprecationWarning: The class hawkey.Repo is deprecated. Please use dnf.repo.Repo instead. The class will be removed on 2019-12-31. warnings.warn(msg, DeprecationWarning) Traceback (most recent call last): File "/usr/local/bin/splitter.py", line 524, in <module> main() File "/usr/local/bin/splitter.py", line 505, in main repos = parse_repository(args.repository) File "/usr/local/bin/splitter.py", line 479, in parse_repository mod = _parse_repository_modular(repo_info, package_sack) File "/usr/local/bin/splitter.py", line 153, in _parse_repository_modular idx = _get_modulemd(repo_info=repo_info) File "/usr/local/bin/splitter.py", line 87, in _get_modulemd mmdcts = gzf.read().decode('utf-8') File "/usr/lib64/python3.6/gzip.py", line 276, in read return self._buffer.read(size) File "/usr/lib64/python3.6/gzip.py", line 463, in read if not self._read_gzip_header(): File "/usr/lib64/python3.6/gzip.py", line 411, in _read_gzip_header raise OSError('Not a gzipped file (%r)' % magic) OSError: Not a gzipped file (b'\xfd7')
Here is what is in the repodata that crashes it
# ls -l /mnt/fedora/app/fi-repo/centos/centos-8-stream/AppStream/aarch64/os/repodata/ total 38144 -rw-rw-r--. 1 1004 1005 5645664 May 26 02:41 1182694afc27bfd77140350529cb80221ada17472090ed8c98eeae472c0370c4-primary.sqlite.xz -rw-rw-r--. 1 1004 1005 8398900 May 26 02:41 30d38eb4c34ecaaef0043c2a3e2af53076ce80b4033faba7db0913c637e4a920-filelists.sqlite.xz -rw-rw-r--. 1 1004 1005 100576 May 26 02:40 3c2db7d0904b792773d68a1a90f01be660518deee2eb761ee5092cab73c75d90-modules.yaml.xz -rw-rw-r--. 1 1004 1005 2963100 May 26 02:41 85cbffe91205dbe0275bbdf548a9b49dc5b20d6d5e7a00624775c2c2897d2885-primary.xml.gz -rw-rw-r--. 1 1004 1005 16300172 May 26 02:41 8e289556ce1b1a80f6bc9d5e91481e530ad249c8665241e2090cb914ccd5dfcf-filelists.xml.gz -rw-rw-r--. 1 1004 1005 2433351 May 26 02:41 9619d89f0feefc5bf204db0f5a7df87952775a6b4e15453a03fb9326da25d16d-other.xml.gz -rw-rw-r--. 1 1004 1005 73504 May 26 02:41 a6b041d34e4ece8cd8a309880990ec8cc9d1e984b1db94bbc706ac41a09c9bed-comps-AppStream.aarch64.xml.xz -rw-rw-r--. 1 1004 1005 426685 May 26 02:40 b5deb7cdeaa11606ba9342bdfaec8a91f8544b95bb6431239e2fd5f1013cf849-comps-AppStream.aarch64.xml -rw-rw-r--. 1 1004 1005 2497664 May 26 02:41 dc7728ea2bd631fce3922364191368bf826888e8b800c9c8ccef3d7cee198d6a-other.sqlite.xz -rw-rw-r--. 1 1004 1005 4474 May 26 02:41 repomd.xml -rw-r--r--. 1 1004 1005 811 May 26 12:57 repomd.xml.asc
RHEL instead ships those as
-rw-r--r--. 1 root sysadmin-main 172315034 May 28 14:19 10d070ba7d692c744106499e5b55399a09b4b839286a4862203db690d1bbf270-other.xml.gz -rw-r--r--. 1 root sysadmin-main 32175077 May 28 14:19 1a64a2be0011c7191cbc4fc3f3e28fa08ba762d0c7f9e707f4e22547ff6542aa-filelists.xml.gz -rw-r--r--. 1 root sysadmin-main 158871797 May 28 14:19 2bfe77b233df8da805ef2a8f12294f3809aad3cac044b4107aaef847d42ae4c6-other.sqlite.bz2 -rw-r--r--. 1 root sysadmin-main 423545 May 28 14:19 377ac3c40be6e75c3d2a3177865b863ec3c8c941c0707aadc9276e6c22639b53-modules.yaml.gz -rw-r--r--. 1 root sysadmin-main 2289476 May 28 14:19 588619576b18b1f6d5b58b4ee272c39c5ef812e9e2808417cb802a8f222767d7-updateinfo.xml.gz -rw-r--r--. 1 root sysadmin-main 10567053 May 28 14:19 595ffedcc88831b2f511bb65cb9380fe5501acf6577d62804272658fb567bd27-primary.xml.gz -rw-r--r--. 1 root sysadmin-main 34034214 May 28 14:19 86ed0793bb27fa8917551d78ee0d82af40e9b84b686e05923da9c79dc8cae1e5-filelists.sqlite.bz2 -rw-r--r--. 1 root sysadmin-main 17724576 May 28 14:19 a80fc44247f301e7c56ccf8f25573ee2e1b38123169dc4b1d2fc7b0205ffc278-primary.sqlite.bz2 -rw-r--r--. 1 root sysadmin-main 2171 May 28 14:19 c281b795-db8a-40e4-bb51-f5823d0a0d4e -rw-r--r--. 1 root sysadmin-main 500319 May 28 14:19 d5da7c8e34701c9709259f8bdb19199fe63a5e1b30df83140afc5ec3d8aa2be4-comps.xml -rw-r--r--. 1 root sysadmin-main 4631 May 28 14:19 repomd.xml
so splitter.py needs to gain some smarts to determine how the file is compressed, then uncompress it in the way which will work. So currently the cs8 is broken on creating any new repos.
I have done a 'temp' fix by using the last good one and touch -r timestamp all the files then rerun createrepo. The splitter.py code will need a patch.
ok I think I fixed it to work with the two different file formats. It seems to be running correctly now for c8s
I can confirm! Both my epel8 and epel8-next test builds with python38-pytest succeeded.
Thank you all!
Where do we stand on this? Last I recall there were some issues and things were reverted. Is that still the case? My epel8 scratch build fails to find python38-attrs which should be in python38-devel.
Where do we stand on this? Last I recall there were some issues and things were reverted. Is that still the case?
It should be fixed now. As part of this change, grobisplitter was updated to the latest upstream version, but that caused problems, so it was reverted back to our downstream modified version. The changes that were made to fix the python38-devel content being dropped by koji remained intact.
My epel8 scratch build fails to find python38-attrs which should be in python38-devel.
I just submitted a scratch build that depends on python38-attrs and it worked fine.
Sorry, my bad. I forgot to change %py3_dist to %py38_dist. Should we close this now, or are we waiting for anything else at this point?
I think we can close this bug.
FYI, %py38_dist doesn't exist. %py3_dist will work properly once the epel-rpm-macros maintainers merge my PR.
%py38_dist
%py3_dist
Yeah, all the python38 module rpms should be available now... so, closing this.
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.