releng

Clone

#10679 Cannot install python38-pytest into EPEL8 buildroot
Closed: Fixed 3 days ago by kevin. Opened 4 months ago by orion.

Closed: Fixed
Reopen Issue
  • Describe the issue
    I'm trying to build python38-jmespath which requires python38-pytest from the python38-devel module, but it fails:

https://koji.fedoraproject.org/koji/taskinfo?taskID=83566863

DEBUG util.py:444: No matching package to install: 'python38-pytest'

  • When do you need this? (YYYY/MM/DD)
    Preferably before the RHEL 8.6 release 2022/05/??

  • If we cannot complete your request, what is the impact?
    Cannot build many python38 modules in EPEL8

As one of the RHEL Python maintainers, I confirm that our intention was that python38-pytest is to be available in CRB and hence in the EPEL8 buildroot.

Metadata Update from @mohanboddu:
- Issue tagged with: medium-gain, medium-trouble, ops
4 months ago

ok. I spent far too long digging into this tonight. Here's what I have so far:

We have the rhel8-base external repo set in koji with merge mode 'bare'

This causes koji to use /usr/bin/mergerepo_c --pkgorigins --all

This should cause it to include all packages in the merged repodata, but somehow it does not.

python38-pytest is built from a pytest src.rpm.

There is also a python3-pytest rpm built from another older pytest src.rpm.

python38-pytest is in the external repo repodata, but not in the merged repodata in koji once it's merged.

So, I think somehow mergerepo_c is seeing the first one and not including the second one. I am not sure why thought.

I think next step is to make a small repoducer...

@orion until we get this figured out, as a workaround can you just disable tests?

(note also: you will need to mv %{buildroot}/%{_bindir}/jp.py %{buildroot}/%{_bindir}/jp38.py to avoid conflicting with python3-jmespath...)

FWIW I cannot reproduce this way:

├── repo36
│   └── python3-pytest-3.9.1-7.el8ost.1.noarch.rpm
└── repo38
    └── python38-pytest-4.6.6-3.scrmod+el8.6.0+14528+0d1dce30.noarch.rpm
  1. call createrepo_c . in both directories
  2. call mergerepo_c --repo=file:///.../merge-reproducer/repo3{6,8}/ --pkgorigins --all
  3. create a repo file with the merged_repo
  4. repoquery for all packagesin that repo -> both pytests are visible
Edited 4 months ago by churchyard

I'm sure I can disable tests if I get desperate, but we really can't start producing lots of python38- packages confidently in EPEL8 until this is resolved. Thanks for everyone's efforts.

The python38-devel module is not enabled by default. Is that related to this issue?

On a regular RHEL 8 VM with CRB enabled:

$ sudo dnf install python38-pytest
Updating Subscription Management repositories.
Last metadata expiration check: 0:04:23 ago on Fri 20 May 2022 06:00:05 PM EDT.
All matches were filtered out by modular filtering for argument: python38-pytest
Error: Unable to find a match: python38-pytest
$ sudo dnf module enable python38-devel
Updating Subscription Management repositories.
Last metadata expiration check: 0:04:37 ago on Fri 20 May 2022 06:00:05 PM EDT.
Dependencies resolved.
====================================================================================================================
 Package                    Architecture              Version                      Repository                  Size
====================================================================================================================
Enabling module streams:
 python38-devel                                       3.8                                                          

Transaction Summary
====================================================================================================================

Is this ok [y/N]: y
Complete!
$ sudo dnf install python38-pytest
Updating Subscription Management repositories.
Last metadata expiration check: 0:04:46 ago on Fri 20 May 2022 06:00:05 PM EDT.
Dependencies resolved.
====================================================================================================================
 Package                Arch   Version                               Repository                                Size
====================================================================================================================
Installing:
 python38-pytest        noarch 4.6.6-3.module+el8.4.0+8888+89bc7e79  codeready-builder-for-rhel-8-x86_64-rpms 487 k
Installing dependencies:
 python38-atomicwrites  noarch 1.3.0-8.module+el8.4.0+8888+89bc7e79  codeready-builder-for-rhel-8-x86_64-rpms  22 k
 python38-attrs         noarch 19.3.0-3.module+el8.4.0+8888+89bc7e79 codeready-builder-for-rhel-8-x86_64-rpms  71 k
 python38-more-itertools
                        noarch 7.2.0-5.module+el8.4.0+8888+89bc7e79  codeready-builder-for-rhel-8-x86_64-rpms  60 k
 python38-packaging     noarch 19.2-3.module+el8.4.0+8888+89bc7e79   codeready-builder-for-rhel-8-x86_64-rpms  59 k
 python38-pluggy        noarch 0.13.0-3.module+el8.4.0+8888+89bc7e79 codeready-builder-for-rhel-8-x86_64-rpms  41 k
 python38-py            noarch 1.8.0-8.module+el8.4.0+8888+89bc7e79  codeready-builder-for-rhel-8-x86_64-rpms 202 k
 python38-pyparsing     noarch 2.4.5-3.module+el8.4.0+8888+89bc7e79  codeready-builder-for-rhel-8-x86_64-rpms 160 k
 python38-wcwidth       noarch 0.1.7-16.module+el8.4.0+8888+89bc7e79 codeready-builder-for-rhel-8-x86_64-rpms  34 k

Transaction Summary
====================================================================================================================
Install  9 Packages

Total download size: 1.1 M
Installed size: 4.5 M
Is this ok [y/N]: n
Operation aborted.

I'll just note that with RHEL8.6 released and ansible-core using python38, my need for python38 modules has gotten much more pressing. I guess I'll just be building them locally for now but would be nice to be able to provide them in EPEL.

I'll just note that with RHEL8.6 released and ansible-core using python38, my need for python38 modules has gotten much more pressing. I guess I'll just be building them locally for now but would be nice to be able to provide them in EPEL.

If you really wanted to, as a temporary workaround, you could disable tests by default with %bcond_with and then run a local build --with test before pushing to epel. It's possible to enable the python38-devel module in a local mock chroot.

ok, with help from @sgallagh and @gotmax23 we finally tracked this down. :)

The way we consume things is to sync them over, then split out the modules. But we have been keeping old versions of modules in the same repo. mergerepo_c sees that there's say 5 versions of python38-pytest (one each in the last 5 versions of the module) that have the exact same name, version, checksum , but NOT mtime. So it yells about it and tosses those out.

So, now we need to either make sure the mtime is the exact same on all the same packages, or stop carrying the old modules along for the ride. We are testing out both approaches, but hopefully this will be fixed in tne next few days.

Things appear to have morphed a bit, now I'm getting:

DEBUG util.py:443:  Error: 
DEBUG util.py:443:   Problem 1: conflicting requests
DEBUG util.py:443:    - nothing provides python38-atomicwrites needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch
DEBUG util.py:443:    - nothing provides python38-attrs needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch
DEBUG util.py:443:    - nothing provides python38-more-itertools needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch
DEBUG util.py:443:    - nothing provides python38-packaging needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch
DEBUG util.py:443:    - nothing provides python38-wcwidth needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch
DEBUG util.py:443:    - nothing provides python38-pluggy >= 0.12 needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch
DEBUG util.py:443:    - nothing provides python38-py >= 1.5.0 needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch
DEBUG util.py:443:   Problem 2: conflicting requests
DEBUG util.py:443:    - nothing provides python38-urllib3 >= 1.21.1 needed by python38-requests-2.22.0-9.module_el8.4.0+647+0ba99ce8.noarch

https://koji.fedoraproject.org/koji/taskinfo?taskID=87497860

These are other packages in the python38-devel module except for python38-urllib3 which is part of the python38 module. But it seems is can see python38-pytest from python38-devel and python38-requests from python38.

One possible solution for ensuring the mtime values are all the same: https://github.com/fedora-modularity/GrobiSplitter/pull/7

Glad to see some progress here. Thanks everyone.

It is definitely 100% not fixed yet. :)

I fixed python38-pytest as a test... we have not yet decided on the best fix, not deployed it. ;)

I have worked with @sgallagh and @kevin to put in some fixes to the script. I believe I have it 'working' for at least some test parts. Please test

This appears to have been fixed for epel8 but not epel8-next. Has the script been run for the c8s mirror yet?

DEBUG util.py:443:  No matches found for the following disable plugin patterns: local, spacewalk, versionlock
DEBUG util.py:443:  Error: 
DEBUG util.py:443:   Problem: conflicting requests
DEBUG util.py:443:    - nothing provides python38-atomicwrites needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch
DEBUG util.py:443:    - nothing provides python38-attrs needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch
DEBUG util.py:443:    - nothing provides python38-more-itertools needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch
DEBUG util.py:443:    - nothing provides python38-packaging needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch
DEBUG util.py:443:    - nothing provides python38-wcwidth needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch
DEBUG util.py:443:    - nothing provides python38-pluggy >= 0.12 needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch
DEBUG util.py:443:    - nothing provides python38-py >= 1.5.0 needed by python38-pytest-4.6.6-3.module_el8.5.0+742+dbad1979.noarch
DEBUG util.py:445:  (try to add '--skip-broken' to skip uninstallable packages)
DEBUG util.py:596:  Child return code was: 1
DEBUG util.py:168:  kill orphans

https://kojipkgs.fedoraproject.org//work/tasks/5102/87565102/root.log

I don't know how the CentOS side runs. Looking at the directories.. the last time it successfully ran to completion was 2 years ago.

drwxr-xr-x. 5 root root 4096 2022-05-26 10:16 2022-05-26T10/
drwxr-xr-x. 5 root root 4096 2022-05-27 10:16 2022-05-27T10/
drwxr-xr-x. 3 root root 4096 2022-05-28 10:10 2022-05-28T10/
drwxr-xr-x. 5 root root 4096 2020-12-08 19:48 latest/
lrwxrwxrwx. 1 root root   13 2022-05-27 10:21 staged -> 2022-05-27T10/

but that seems wrong so I don't know where koji is looking.

@kevin @carlgeorge @tdawson This needs to be looked at by someone who is more fulltime on EPEL and koji configs. I would be bothering you enough to tell me what to fix :dizzy_face:

but that seems wrong so I don't know where koji is looking.

Koji is indeed pulling from https://infrastructure.fedoraproject.org/repo/centos/stream8-kojitarget/latest/$arch/CS-8-001/. See https://koji.fedoraproject.org/koji/externalrepoinfo?extrepoID=47.

latest is not a symlink, but the individual CS-8-001 directories (e.g. in https://infrastructure.fedoraproject.org/repo/centos/stream8-kojitarget/latest/x86_64/) are from yesterday. I believe this is how the mirror was configured before your script changes. It looks like the updated cents8-split.sh either failed or didn't run, because the files in the aforementioned directory have varying mtimes.

It also looks like populating https://infrastructure.fedoraproject.org/repo/centos/stream8-kojitarget/2022-05-28T10/ failed part way through; only aarch64 is present.

I will rerun the cent8-split.sh . OK the updated splitter code does not like stream.
Edited a month ago by smooge 
# /usr/local/bin/splitter.py --debug --action hardlink --target CS-8-002 /mnt/fedora/app/fi-repo/centos/centos-8-stream/AppStream/aarch64/os/ --only-defaults
/usr/lib64/python3.6/site-packages/hawkey/__init__.py:348: DeprecationWarning: The class hawkey.Repo is deprecated. Please use dnf.repo.Repo instead. The class will be removed on 2019-12-31.
  warnings.warn(msg, DeprecationWarning)
Traceback (most recent call last):
  File "/usr/local/bin/splitter.py", line 524, in <module>
    main()
  File "/usr/local/bin/splitter.py", line 505, in main
    repos = parse_repository(args.repository)
  File "/usr/local/bin/splitter.py", line 479, in parse_repository
    mod = _parse_repository_modular(repo_info, package_sack)
  File "/usr/local/bin/splitter.py", line 153, in _parse_repository_modular
    idx = _get_modulemd(repo_info=repo_info)
  File "/usr/local/bin/splitter.py", line 87, in _get_modulemd
    mmdcts = gzf.read().decode('utf-8')
  File "/usr/lib64/python3.6/gzip.py", line 276, in read
    return self._buffer.read(size)
  File "/usr/lib64/python3.6/gzip.py", line 463, in read
    if not self._read_gzip_header():
  File "/usr/lib64/python3.6/gzip.py", line 411, in _read_gzip_header
    raise OSError('Not a gzipped file (%r)' % magic)
OSError: Not a gzipped file (b'\xfd7')

Here is what is in the repodata that crashes it

# ls -l /mnt/fedora/app/fi-repo/centos/centos-8-stream/AppStream/aarch64/os/repodata/
total 38144
-rw-rw-r--. 1 1004 1005  5645664 May 26 02:41 1182694afc27bfd77140350529cb80221ada17472090ed8c98eeae472c0370c4-primary.sqlite.xz
-rw-rw-r--. 1 1004 1005  8398900 May 26 02:41 30d38eb4c34ecaaef0043c2a3e2af53076ce80b4033faba7db0913c637e4a920-filelists.sqlite.xz
-rw-rw-r--. 1 1004 1005   100576 May 26 02:40 3c2db7d0904b792773d68a1a90f01be660518deee2eb761ee5092cab73c75d90-modules.yaml.xz
-rw-rw-r--. 1 1004 1005  2963100 May 26 02:41 85cbffe91205dbe0275bbdf548a9b49dc5b20d6d5e7a00624775c2c2897d2885-primary.xml.gz
-rw-rw-r--. 1 1004 1005 16300172 May 26 02:41 8e289556ce1b1a80f6bc9d5e91481e530ad249c8665241e2090cb914ccd5dfcf-filelists.xml.gz
-rw-rw-r--. 1 1004 1005  2433351 May 26 02:41 9619d89f0feefc5bf204db0f5a7df87952775a6b4e15453a03fb9326da25d16d-other.xml.gz
-rw-rw-r--. 1 1004 1005    73504 May 26 02:41 a6b041d34e4ece8cd8a309880990ec8cc9d1e984b1db94bbc706ac41a09c9bed-comps-AppStream.aarch64.xml.xz
-rw-rw-r--. 1 1004 1005   426685 May 26 02:40 b5deb7cdeaa11606ba9342bdfaec8a91f8544b95bb6431239e2fd5f1013cf849-comps-AppStream.aarch64.xml
-rw-rw-r--. 1 1004 1005  2497664 May 26 02:41 dc7728ea2bd631fce3922364191368bf826888e8b800c9c8ccef3d7cee198d6a-other.sqlite.xz
-rw-rw-r--. 1 1004 1005     4474 May 26 02:41 repomd.xml
-rw-r--r--. 1 1004 1005      811 May 26 12:57 repomd.xml.asc

RHEL instead ships those as

-rw-r--r--. 1 root sysadmin-main 172315034 May 28 14:19 10d070ba7d692c744106499e5b55399a09b4b839286a4862203db690d1bbf270-other.xml.gz
-rw-r--r--. 1 root sysadmin-main  32175077 May 28 14:19 1a64a2be0011c7191cbc4fc3f3e28fa08ba762d0c7f9e707f4e22547ff6542aa-filelists.xml.gz
-rw-r--r--. 1 root sysadmin-main 158871797 May 28 14:19 2bfe77b233df8da805ef2a8f12294f3809aad3cac044b4107aaef847d42ae4c6-other.sqlite.bz2
-rw-r--r--. 1 root sysadmin-main    423545 May 28 14:19 377ac3c40be6e75c3d2a3177865b863ec3c8c941c0707aadc9276e6c22639b53-modules.yaml.gz
-rw-r--r--. 1 root sysadmin-main   2289476 May 28 14:19 588619576b18b1f6d5b58b4ee272c39c5ef812e9e2808417cb802a8f222767d7-updateinfo.xml.gz
-rw-r--r--. 1 root sysadmin-main  10567053 May 28 14:19 595ffedcc88831b2f511bb65cb9380fe5501acf6577d62804272658fb567bd27-primary.xml.gz
-rw-r--r--. 1 root sysadmin-main  34034214 May 28 14:19 86ed0793bb27fa8917551d78ee0d82af40e9b84b686e05923da9c79dc8cae1e5-filelists.sqlite.bz2
-rw-r--r--. 1 root sysadmin-main  17724576 May 28 14:19 a80fc44247f301e7c56ccf8f25573ee2e1b38123169dc4b1d2fc7b0205ffc278-primary.sqlite.bz2
-rw-r--r--. 1 root sysadmin-main      2171 May 28 14:19 c281b795-db8a-40e4-bb51-f5823d0a0d4e
-rw-r--r--. 1 root sysadmin-main    500319 May 28 14:19 d5da7c8e34701c9709259f8bdb19199fe63a5e1b30df83140afc5ec3d8aa2be4-comps.xml
-rw-r--r--. 1 root sysadmin-main      4631 May 28 14:19 repomd.xml

so splitter.py needs to gain some smarts to determine how the file is compressed, then uncompress it in the way which will work. So currently the cs8 is broken on creating any new repos.

I have done a 'temp' fix by using the last good one and touch -r timestamp all the files then rerun createrepo. The splitter.py code will need a patch.

ok I think I fixed it to work with the two different file formats. It seems to be running correctly now for c8s

ok I think I fixed it to work with the two different file formats. It seems to be running correctly now for c8s

I can confirm! Both my epel8 and epel8-next test builds with python38-pytest succeeded.

Thank you all!

Where do we stand on this? Last I recall there were some issues and things were reverted. Is that still the case? My epel8 scratch build fails to find python38-attrs which should be in python38-devel.

Where do we stand on this? Last I recall there were some issues and things were reverted. Is that still the case?

It should be fixed now. As part of this change, grobisplitter was updated to the latest upstream version, but that caused problems, so it was reverted back to our downstream modified version. The changes that were made to fix the python38-devel content being dropped by koji remained intact.

My epel8 scratch build fails to find python38-attrs which should be in python38-devel.

I just submitted a scratch build that depends on python38-attrs and it worked fine.
Edited 3 days ago by gotmax23

Sorry, my bad. I forgot to change %py3_dist to %py38_dist. Should we close this now, or are we waiting for anything else at this point?

I think we can close this bug.

FYI, %py38_dist doesn't exist. %py3_dist will work properly once the epel-rpm-macros maintainers merge my PR.

Edited 3 days ago by gotmax23

Yeah, all the python38 module rpms should be available now... so, closing this.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
3 days ago

Login to comment on this ticket.

Metadata
None

medium-trouble medium-gain ops

None
None
None
None
None
None
Boards 1
Ops Status: Backlog
Powered by Pagure 5.13.3
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.