#671 python2-pulp-common uses other project's namespace
Closed: nothingtodo 3 years ago Opened 3 years ago by apevec.

Found out during package review:

python2-pulp-common is a subpackage of existing Fedora package:

https://pypi.python.org/pypi/PuLP is taken by the "LP modeler written in python"

PyPI is case-insensitive but it is also lowercase Python module: "import pulp" to use PuLP

I assume neither of the upstream projects could be renamed at this point, both are in wide use for different use-cases, so request FPC to allow usage of Conflict: as per https://fedoraproject.org/wiki/Packaging:Conflicts#Library_Name_Conflicts

Even if you think it's unlikely, you should still try asking nicely. You never know if they're amenable or not unless there have been previous failed attempts.

I am not sure that simply adding Conflicts: will help. What does the python automatic dependency generation do with these packages? If it generates the same thing then one package absolutely must change, even if that introduces a deviance from upstream.

autodeps are free of conflicts, python2-pulp-common provides python2.7dist(pulp-common)
while python2-PuLP under review provides python2.7dist(pulp)

It conflicts only in /usr/lib/python2.7/site-packages/pulp/ dir (and corresponding init.py) which really should be pulp_common in pulpproject.org (but even that and other pulp_ names are not registered on PyPI!) to match actual egginfo from pulpproject.org.

Since they missed to registerd on PyPI, pulpproject.org upstream shoudl change,
dradez has emailed pulp RPM maintainer for assistance and I've now opened an issue in the upstream tracker: https://pulp.plan.io/issues/2531

I am the team lead for the Pulp project that lives at pulpproject.org and is currently in Fedora. We are heavily invested in using the "pulp" python namespace, and I don't see that it would be practical for us to change. But we are committed to being in Fedora and are very interested in finding a resolution for this issue.

For clarity in this discussion, I'd like to use terminology from PEP 420.


It is important to keep in mind the difference between python "distribution" names and the python "package" names they contain. It is true that our project does not currently have a distribution with the name "pulp". Our distributions have names (as seen in various setup.py files) such as "pulp-common", "pulp-server", etc. Several such distributions collectively utilize the python namespace "pulp" via pkgutil.extend_path. Each such "distribution" contains a "package" named "pulp", and pkgutil.extend_path allows their subpackages to be cumulative. Eventually we may utilize PEP 420 to accomplish this.

With that in mind, it appears that the collision of projects Pulp and PuLP is between "package" names, whereas PyPI is a registry of "distribution" names.

While only one project can register a given distribution name, that does not make any guarantees about package name use. And in the case of a project using the "namespace package" concept, it can be awkward and/or difficult to have a distribution name matching the package name. So while I appreciate that the PuLP project owns the "distribution" name "pulp" on PyPI, I do not see that as an exclusive right to the package name.

But obviously "package" name collisions are problematic and best avoided when possible. Since it is far too late to avoid this particular collision, using the "Conflicts" method seems like a reasonable approach for facilitating co-existence in Fedora.

As a next step, we will look into registering all of our distribution names on PyPI.

(Note: not a current FPC member)

Although it is correct to say that pypi registers distribution names rather than package names (I've just recently seen that the attrs distribution name on pypi is using the attr package name, for instance :-( registering on pypi stakes a claim to both the distribution name and package name in the eyes of other python developers. Trying to split hairs here will only alienate people.

The last time this came up was with the two mock packages. That one took many years of on-again off-again discussion with neither side willing to budge. It kept the mock library registered on pypi out of Fedora for too long. In the end, we (the mock package that we worked on in Fedora) resolved it by renaming the python package name to mockbuild. Since the only consumer of mock as a library were other packages created by the mock team, this was doable and made people in the upstream world happy with us.

In that case, we really had no choice (although it took us years to face that fact). Our mock had been created first but we had not registered on pypi, our usages were all "internal" (meaning, among our own projects, not in end-user projects), and the registered on pypi mock package was wildly popular. We stood no chance.

In this case, it looks like the registered on PyPI pulp package is not as popular as the mock package was. However, googling for "import pulp" and "from pulp" import only turns up references to their software, not to the pulp software that's presently in Fedora. That leads me to believe the pulp python package is mainly an internal import of the packages built by the same team whereas the other package is being used by other people unrelated to the library's authors. This does not leave us with a clear winner.

The best thing to do here is first and foremost to talk: the software authors (or failing that , the Fedora packagers) need to talk to the other upstream. See if you can wheedle the other upstream to change their name or be reluctantly convinced that it's the currently in Fedora package that should rename to be a good citizen. (Approaching both upstreams is supposed to be done before coming to the FPC, by the way ;-).

If this fails, then think about what the currently-in-Fedora's usage of the pulp library is. If it's internal, then there's a couple options open to it:
rename. This is a single operation in a refactoring browser. It's hard to give up something you think of as yours and also hard to do so when the only reasons are external pressures but it's the kind of sacrifice adults make to show that they've learned how to work well together.
move to a private directory. Moving the common library out of %{site_packages}/pulp and into, for instance, %{_datadir}/pulp_common/pulp will also fix the conflict. The pulp applications would then set PYTHONPATH (if driven via a shell script) or sys.path (if driven via python file) to include that directory before importing something from the pulp namespace.

If the already-in-Fedora pulp is used as a public library (things outside of the pulp project are using the pulp library), then talking it out between the two upstreams until one agrees to rename is really the best option. Having libraries that can't both be installed means that the ecosystem of packages which depend on those libraries also can't be parallel installed. This can be a recipe for huge user frustration. It's best to do a lot of front-end work to try to resolve it than to end up in that situation down the road.

Having both upstreams talk is a good idea. Without a resolution in upstream, packagers and users will have headaches and frustration. I'm a developer with the in-Fedora Pulp (pulpproject.org), and we will reach out to the other project to discuss options soon. While that happens, if the FPC wants to make a decision on the allowing of Conflicts in the meantime, I think that is OK. If you have other advice/ideas, or if you want to hear what the upstream resolution of the naming conflict is before making a decision here, let us know and we can share the outcome. Thank you for encouraging our projects to communicate. When we communicate with respect and candor everyone wins.

So was there any progress here? Basically involving any of the Fedora committees is the last resort when the upstreams and package maintainers can't work something out, and we're happy to do stay completely out of the way and do absolutely nothing.

For my personal opinion: Conflicts: is an absolutely terrible solution. It's basically just giving up and hoping that end users will never, ever want to install both on the same system. To suggest it implies that you know that at least one of the projects will never be a part of any significant dependency chain, which implies that renaming (even if it's just a patch carried by Fedora) is actually not that difficult and so you might as well just go ahead and do it.

Metadata Update from @tibbs:
- Issue close_status updated to: None
- Issue tagged with: needinfo

3 years ago

We are trying to discuss how to resolve the package conflict between upstreams, but I don't have an update on a resolution yet. To resolve the conflict, at least one project will have to release a major update to change the Python package name, which would be a backwards incompatible change upstream. This makes any resolution take time. Given a longer possible timeline, I'm not sure what the requester or committee should do about this request. Once there is a planned change, we will post back here.

Thanks for the update. It's good to see that it's at least being discussed.

It hurts nothing except those with OCD to have this ticket around, though we may periodically ping it to see if there's been any progress.

Pulp (http://pulpproject.org) and PuLP (https://github.com/coin-or/pulp) have had some direct discussion about resolving this conflict upstream. The resolution reached is that we, Pulp (pulpproject.org), will use a different, non-conflicting Python module name starting with our next major release 3.0.0. Additionally the namespace we use will be one that we have registered on PyPI so that our Python namespace is clearly one that we have rights to. This should resolve this issue permanently in upstream.

Pulp 3.0.0 is currently in development so this has come at a good time. It will take some time (months) to release this version so I'm not sure what the committee would like to do regarding this request in the near-term.

If there are any other questions or requests related to this for us to handle upstream please let us know.

If a temporary use of Conflicts: is really needed then I would support that. I'm not sure exactly what happens when it's not there. Not having looked at all of the packages involved, I would guess that you get a file conflict if you try to install both, which would contradict our goal of having conflicting packages explicitly declare their conflicts.

Metadata Update from @tibbs:
- Issue untagged with: needinfo
- Issue tagged with: meeting

3 years ago

We discussed this at this weeks meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2017-04-13/fpc.2017-04-13-16.00.txt):

  • 671 python2-pulp-common uses other project's namespace (geppetto,
  • This seems complete now, thanks for resolving this upstream. And as
    tibbs said if you want to conflict in the short term, that's fine,
    just add explicit conflicts. (geppetto, 16:13:50)

Metadata Update from @james:
- Issue close_status updated to: nothingtodo
- Issue status updated to: Closed (was: Open)

3 years ago

Login to comment on this ticket.