#50771 1.4.2.5 doesn't compile due to error ModuleNotFoundError: No module named 'pkg_resources.extern'
Closed: wontfix 4 years ago by vashirov. Opened 4 years ago by svenstaro.

Issue Description

I'm compiling 1.4.2.5 and I get

Traceback (most recent call last):
  File "setup.py", line 32, in <module>
    setup(
  File "/usr/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/lib/python3.8/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/lib/python3.8/site-packages/build_manpages/build_manpages.py", line 85, in run
    self.run_command(DEFAULT_CMD_NAME)
  File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/lib/python3.8/site-packages/build_manpages/build_manpages.py", line 74, in run
    parser = get_parser(data['import_type'], data['import_from'], data['objname'], data['objtype'])
  File "/usr/lib/python3.8/site-packages/build_manpages/build_manpage.py", line 43, in get_parser
    return get_parser_from_file(import_from, objname, objtype)
  File "/usr/lib/python3.8/site-packages/build_manpages/build_manpage.py", line 37, in get_parser_from_file
    filedict = run_path(filename)
  File "/usr/lib/python3.8/runpy.py", line 262, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/usr/lib/python3.8/runpy.py", line 95, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.8/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "cli/dsconf", line 18, in <module>
    from lib389._constants import DSRC_HOME
  File "/build/389-ds-base/src/389-ds-base-1.4.2.5/src/lib389/lib389/__init__.py", line 58, in <module>
    from lib389._entry import Entry
  File "/build/389-ds-base/src/389-ds-base-1.4.2.5/src/lib389/lib389/_entry.py", line 19, in <module>
    from lib389.utils import (ensure_str, ensure_bytes, ensure_list_bytes, display_log_data)
  File "/build/389-ds-base/src/389-ds-base-1.4.2.5/src/lib389/lib389/utils.py", line 43, in <module>
    from pkg_resources.extern.packaging.version import LegacyVersion
ModuleNotFoundError: No module named 'pkg_resources.extern'

Package Version and Platform

1.4.2.5 on Arch Linux

Steps to reproduce

Compile using

  ./configure \
    --prefix=/usr \
    --sysconfdir=/etc \
    --sbindir=/usr/bin \
    --localstatedir=/var \
    --libexecdir=/usr/lib/${pkgbase} \
    --with-tmpfiles-d=/usr/lib/tmpfiles.d \
    --with-systemd \
    --with-systemdsystemunitdir=/usr/lib/systemd/system \
    --with-systemdsystemconfdir=/etc/systemd/system \
    --with-journald \
    --enable-autobind \
    --enable-cmocka \
    --with-openldap \
    --enable-rust
  make

  cd src/lib389
  python setup.py build

Actual results

ModuleNotFoundError: No module named 'pkg_resources.extern'

Expected results

Clean compile. 1.4.2.4 worked just fine except for #50737.


I'd like to note that setuptools, which seems to be the Python package that is reported to be missing here, is installed in the system.

Part of the problem seems to be the way python-setuptools is packaged in Arch:
https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/python-setuptools#n24
All vendored packages are removed and replaced with their distro equivalents.

But we rely on the way python-setuptools is packaged by upstream and in Fedora/RHEL/OpenSUSE/SLES -- together with the vendored packages:
http://rpmfind.net/linux/RPM/fedora/devel/rawhide/x86_64/p/python3-setuptools-41.6.0-1.fc32.noarch.html
http://rpmfind.net/linux/RPM/opensuse/tumbleweed/noarch/python3-setuptools-41.2.0-2.1.noarch.html

Why we're using pkg_resources.extern.packaging.version instead of packaging.version in the first place? Because unfortunately python3-packaging is not present in RHEL8 default repos (it's in CodeReady Linux Builder only). This is another part of the problem.

So the short term fix would be to patch lib389 in Arch to not use pkg_resources.extern and depend on python-packaging instead.
As for the long term solution -- I'm open to suggestions. Perhaps we can import python-packaging into lib389 as a vendored package and drop python-setuptools requirement.

Metadata Update from @vashirov:
- Custom field origin adjusted to None
- Custom field reviewstatus adjusted to None

4 years ago

Why are you relying on private undocumented API? Use pkg_resources.parse_version().

Why are you relying on private undocumented API? Use pkg_resources.parse_version().

Because it returns different objects for different version strings:
https://github.com/pypa/setuptools/blob/1d03fdc94c3676a5b675ec7d818d48c6a772fb49/pkg_resources/__init__.py#L127

And comparison doesn't work correctly.

>>> from pkg_resources import parse_version
>>> x = parse_version('1.4.2.3')
>>> y = parse_version('1.4.2.3.20191112git7a7bc7872')
>>> x
<Version('1.4.2.3')>
>>> y
<LegacyVersion('1.4.2.3.20191112git7a7bc7872')>
>>> x < y 
False
>>> 

If you need to handle inconsistent version schemas, then instead of vendoring a library you can simply:

try:
    from packaging.version import LegacyVersion
except:
    # if python3-packaging is not available, pkg_resources likely has a copy,
    # so try that and hope for the best
    from pkg_resources.extern.packaging.version import LegacyVersion

Although it seems like it should not be that hard to get a core part of the python ecosystem into RHEL, especially as a new package that doesn't break anything.

So, given that upstream of setuptools and every major distribution except for Arch has packaging bundled with setuptools, I'd switch try and except bodies.
For reference:
Debian: https://packages.debian.org/sid/all/pypy-setuptools/filelist
Fedora: http://rpmfind.net/linux/RPM/fedora/devel/rawhide/x86_64/p/python3-setuptools-41.6.0-1.fc32.noarch.html
OpenSUSE: http://rpmfind.net/linux/RPM/opensuse/tumbleweed/noarch/python3-setuptools-41.2.0-2.1.noarch.html
Gentoo: https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/setuptools/setuptools-9999.ebuild (no monkeying with the vendored packages)

Although it seems like it should not be that hard to get a core part of the python ecosystem into RHEL, especially as a new package that doesn't break anything.

It's very hard to get a new package in RHEL. Moreover, python3-packaging is present in CodeReady Linux Builder repository, but it's useless for us there, because we need to have it AppStream.
Every package in RHEL has dev, qe, rel-eng and support resources attached to it. So to add a new package requires a really good business justification. "Doesn't break anything" is not good enough, unfortunately.

That would work, I guess. However, I still think it would make more sense semantically to first try the "correct" way (that uses the official packaging module), and then fall back on what is essentially (as I understand it) a workaround for missing distribution packages in RHEL.

Note that setuptools upstream officially considers it wrong to depend on pkg_resources.extern, and warns that it may be arbitrarily dropped:

https://github.com/pypa/setuptools/blob/master/CHANGES.rst#v3600

At such a point that a mechanism exists that allows build tools to have dependencies, Setuptools will adopt it.

This was in followup to an actual stage of setuptools' development where the .extern vendored dependencies were, in fact, removed from setuptools and replaced by install_requires dependencies on separately provided global modules.

It seems wise to in general try to get to a stage where you're depending on packaging everywhere, and importing it without try/except hacks.

That would work, I guess. However, I still think it would make more sense semantically to first try the "correct" way (that uses the official packaging module), and then fall back on what is essentially (as I understand it) a workaround for missing distribution packages in RHEL.

I agree with you, that it makes more sense semantically, but in practice we have to work with what we have in RHEL/Fedora.

Note that setuptools upstream officially considers it wrong to depend on pkg_resources.extern, and warns that it may be arbitrarily dropped:

I don't like to use undocumented APIs either. We'll cross this bridge when we come to it.

It seems wise to in general try to get to a stage where you're depending on packaging everywhere, and importing it without try/except hacks.

I went with using packaging initially (7a7bc78, aa9600e). But because this dependency is missing in RHEL8 (and it's a runtime one), I had to use a workaround.

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/3826

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix (was: Fixed)

3 years ago

Login to comment on this ticket.

Metadata
Related Pull Requests