#558 Application/Library distinction and package splitting
Closed: accepted 2 years ago Opened 3 years ago by rkuska.

From https://fedoraproject.org/wiki/Packaging:Python#Naming

If executables are to be shipped for both python 2 and python 3:

* Both python 2 and python 3 variants must provide symlinks with a '-X' and '-X.Y' suffix (python runtime major version, or python runtime major.minor version), unless upstream already provides appropriately versioned executables without the dash.    
* The unversioned executable must be the python2 version.

Yet the example specfile has install macros in reverse order:

{{{
%install

Must do the python2 install first because the scripts in /usr/bin are

overwritten with every setup.py install, and in general we want the

python3 version to be the default.

%py2_install
%py3_install
}}}

and therefore python-exec is packaged under python3-example subpackage.

This should be fixed by reversing the incorrect order of python install macros and having
python-exec file under python2-example subpackage, also python-exec-X.Y files should be generated and packaged in the example specfile under both python subpackages.


There are two classes of packages which provide Python scripts or executables:
1. those where the script doesn't care which version of Python is used (dnf, pytables' ptdump, copr-cli, etc.)
2. those where both versions need to be packaged, because they provide python-version-specific functionality (py.test, idle, cython, python itself, easy_install, etc.)

The rules are completely different for class 2 and class 1. For class 2 the rules specify that both binaries should be available with distinguishing suffixes (-2.7, -3.4, etc.), and that the unsuffixed version point to the -2.7 implementation. This is true for even for python itself, and python→python2→python2.7. For class 1, only one version is packaged, and according to the new guidelines it should be the python3 version if possible.

Now, the guidelines quoted describe how to do what's right for class 1. This is reasonable because class 1 packages are more common (just guessing here, but I expect this to be true, because class 2 is essentially things for Python program development, and all "applications" should fall into class 1). Also, the rules for naming of executables in class 2 usually require some hand-tuning of installation anyway (to provide the correct -2, -2.x, -3, -3.x suffixed links), so just running the installers in either order will probably not be enough anyway.

I think it should maybe be clarified that the rules in https://fedoraproject.org/wiki/Packaging:Python#Example_common_spec_file apply to class 1. packages. Something should be added to paragraph 3 there.

Much more common are python modules therefore example spec should cover those (as it was previously - python-setuptools, but using the example spec as setuptools is kind of complicated specfile to begin with).

If the example spec file represents applications (class 1 as you described) then I don't understand why it is built with both python versions as most of applications don't need to be built for both python versions unless they provide some sort of bindings in which case I would expect the package would look like this:

dnf - main package with executable running on python3\
python(2)-dnf - python2 bindings for dnf\
python3-dnf - python3 bindings for dnf\

(replace dnf with example)

Much more common are python modules therefore example spec should cover those
It does. For python modules you want to install both, and the order does not matter. So current example covers modules well.

It does not, if module is shipped with binary (as in example spec) then the order matter -> example doesn't cover modules.

To explain my reasoning more:

Actually there are 3 types of packages you can encounter in Python:

  1. Modules - modules are used within python code through imports, therefore both python2 and python3 subpackage is needed (if upstream supports both)
  2. Applications - application is used through binary executable, user doesn't care which python is it built with as both provide same experience (devassistant) so we only build it once (with default python - python3 in Fedora23) and we don't use prefix python- when naming package. Also such application may provide bindings (dnf), in that case (and if application supports both python interprets) we want to package look like this - dnf (main application with executable running on python3), python2-dnf (python2 bindings subpackage), python3-dnf (python3 bindings subpackage), please note that main package (dnf) is not prefixed with python-.[[br]] This leads us to following executables in package (in Fedora23)- example (running on python3), example-3.4, example-3
  3. Applications which behaves differently based on interpret used - packages (pytest, pip, virtualenv) which are shipping executable which behaves differently on python3 and python2 and therefore both executables are needed, also the package may be used as module sometime (pytest) in this case python3 version should be installed first so python2 install overwrite unprefixed executable and also package should be prefixed with python-.[[br]] This leads us to following executables - example (running on python2), example-X.Y and example-X (for each python interpret)

Currently, the example provided in packaging guidelines is a mix of all 3 types which I find confusing.

Replying to [comment:6 rkuska]:

  1. Applications which behaves differently based on interpret used - packages (pytest, pip, virtualenv) which are shipping executable which behaves differently on python3 and python2 and therefore both executables are needed, also the package may be used as module sometime (pytest) in this case python3 version should be installed first so python2 install overwrite unprefixed executable and also package should be prefixed with python-.[[br]] This leads us to following executables - example (running on python2), example-X.Y and example-X (for each python interpret)

A very good summary. I'd just like to add that the ''reason'' for having unversioned executables point to Python 2 for case 3 is that these should be aligned with what {{{/usr/bin/python}}} is. And since the upstream recommendation is that {{{/usr/bin/python}}} points to Python 2 [1], it makes sense that all unversioned executables stay in line with that to not confuse users.

[1] https://www.python.org/dev/peps/pep-0394/

Note the question of whether to subpackage had come up on the mailing list this week as well: https://lists.fedoraproject.org/pipermail/packaging/2015-August/010930.html

I think subpackaging to split applications and libraries is more general than python so we should address it globally. (The mailing list specifically calls out nodejs in addition to python).

We discussed this at this weeks meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2015-08-20/fpc.2015-08-20-16.00.txt):

...action was that tibbs would add some more examples and make it clearer. Moving it to needinfo.

I finally pasted my notes into the wiki: https://fedoraproject.org/wiki/User:Tibbs/AppsVsLibs

I'm not at all happy with this; for one, it's way too verbose. It also hasn't really gotten any wiki formatting treatment, but it should at least be readable. Shouldn't be too tough to polish up assuming there's agreement on the actual content there.

I would like to know if this covers the nodejs case mentioned above as I'm not really familiar with that issue.

Changing the ticket description to something more relevant to what's currently being discussed.

Is there any progress? Example spec in Python packaging guidelines is still unfixed.

Since the problem has been unresolved for months it came up again: https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/PXIFWMIQXXOEGFZSCWFWWEOPW35OJXB6/

Could I ask what has been the hold up? If it'd be welcomed, I'll gladly amend the existing example myself so you could possibly use it.

As fate would have it, I was asked to write a Python RPM Porting Guide [0] aiming to thoroughly detail the entire process of porting an RPM package to Python 3.

In pursuit of that, I and Miro Hroncok have also written a new Example spec file [1] that follows all the standing packaging guidelines, and can be readily substituted for the current Example spec file [2] in the Packaging Guidelines for Python—which is faulty on several counts as discussed above.

In addition, we have identified 4 distinct types of packages with regards to porting to Python 3 [3], as well as tried to cover a plethora of common pitfalls when porting, as the process is not always straight forward. Therefore if it was at all possible, I believe including a link to the Python RPM Porting Guide [0] might be beneficial to packagers in their porting efforts.

[0] http://python-rpm-porting.readthedocs.io

[1] http://python-rpm-porting.readthedocs.io/en/latest/tools.html#ported-rpm-spec-file

[2] https://fedoraproject.org/wiki/Packaging:Python#Example_common_spec_file

[3] http://python-rpm-porting.readthedocs.io/en/latest/#what-type-of-software-are-you-packaging

Replying to [comment:14 rkuska]:

Is there any progress? Example spec in Python packaging guidelines is still unfixed.

tibbs answered in the python-devel thread:

Nobody has submitted a draft and that ticket isn't particularly urgent.

Guess the ball is back in python-maint's court now!

We discussed this at this weeks meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2016-06-30/fpc.2016-06-30-16.00.txt):

This is rough and I haven't had time to format it, but:

Libraries and Applications

Many language- or domain-specific guidelines refer to "libraries",
"modules", "plug-ins" or other terms specific to the language or
domain ("libraries"). This is specifically important to package
naming. Some applications may include libraries, and some libraries
may include applications, so the distinction is not always clear.

Library or Application?

  • If a package includes only files, libraries, or executables which
    are intended to be imported or loaded into other code, it is a
    library and MUST be packaged as such. (XXX Reference to
    application-specific naming.)

  • If a package includes only executables and libraries used internally
    by that those executables, it is an application and MUST be
    packaged as such. (XXX Need reference to naming guidelines for
    applications, which we don't specifically have.)

  • If the primary purpose of a package is to provide executables to be
    run by users, but also includes libraries which may be imported by
    other code, then it is considered a library and in general SHOULD be
    packaged as an application. However, see Mixed Use Packages below.

  • If the primary purpose of a package is to provide libraries intended
    to be imported or loaded into other code, it is considered a library
    and MUST be packaged as such. However, see Mixed Use Packages below.

It is left to the packager to determine the primary purpose of a
package. Often times upstream will already have done this with their
choice of naming and that choice SHOULD be followed by the Fedora
packager.

Mixed Use Packages

Many packages, regardless of their primary purpose, include both
applications and libraries. There are a few options:

1) Simply package as an application or a library and just include the
libraries or executables in that package.

2) Do #1 but also add a Provides: for the name of an application or
library according to the regular packaging guidelines.

3) Split out into one or more subpackages the executables or
libraries, following the relevant guidelines.

If the executables have significant dependencies which the libraries
do not, then #3 MUST be followed. Otherwise #2 or #3 SHOULD be
followed, but note that in some cases, RPM itself may automatically
add the necessary Provides: and these MUST not be duplicated. (XXX
This whole section is unwieldy.)

If the primary purpose of a package is to provide executables to be run by users, but also includes libraries which may be imported by other code, then it is considered a '''library''' and in general SHOULD be packaged as an '''application'''.

I think this is a typo.

Concerning the example spec file: We'd need to replace the current one with 2 or 3 different example spec files according to the above categories. If the spec files can be made expandable (like the Macros table) it shouldn't be too long.

Proposals for the example spec files:
Application: http://python-rpm-porting.readthedocs.io/en/latest/applications.html#ported-rpm-spec-file
Module: http://python-rpm-porting.readthedocs.io/en/latest/modules.html#ported-rpm-spec-file
* Mixed: http://python-rpm-porting.readthedocs.io/en/latest/application-modules.html#ported-rpm-spec-file


I would also like to propose that a link to the Python RPM Porting Guide be included: http://python-rpm-porting.readthedocs.io/

It guides any packager completely through the process of porting their Python 2 package to Python 3, including help in determining the type of the package, and providing examples.

I've formatted the draft and fixed the thinko torsava pointed out. It's at https://fedoraproject.org/wiki/User:Tibbs/AppVsLib

We discussed this at this weeks meeting (https://meetbot-raw.fedoraproject.org/fedora-meeting-1/2016-09-22/fpc.2016-09-22-16.00.txt):

  • Ticket #558 Application/Library distinction and package splitting
    (geppetto, 16:07:42)
  • LINK: https://fedoraproject.org/wiki/User:Tibbs/AppVsLib
    (geppetto, 16:08:53)
  • ACTION: Application/Library distinction and package splitting, from
    tibbs (+1:6, 0:0, -1:0) (geppetto, 17:10:03)

Metadata Update from @torsava:
- Issue assigned to tibbs

2 years ago

Somehow I never actually wrote this into the guidelines.

Metadata Update from @tibbs:
- Issue close_status updated to: None
- Issue tagged with: writeup

2 years ago

Written up. Announcement text:

A new section was added to the main guideline page with information on the differences between libraries and how packages which fall into both categories should be packaged.
https://fedoraproject.org/wiki/Packaging:Guidelines#Libraries_and_Applications
https://pagure.io/packaging-committee/issue/558

Metadata Update from @tibbs:
- Issue untagged with: writeup
- Issue tagged with: announce

2 years ago

Metadata Update from @tibbs:
- Issue untagged with: announce
- Issue close_status updated to: accepted
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata