From f5723f9e0ba645eb417ad77aef1386cb1c5dff05 Mon Sep 17 00:00:00 2001 From: Several members of Fedora Python SIG Date: Jul 20 2021 10:48:30 +0000 Subject: [PATCH 1/7] Add new Python packaging guidelines --- diff --git a/guidelines/modules/ROOT/nav.adoc b/guidelines/modules/ROOT/nav.adoc index fde1f1a..ca22a25 100644 --- a/guidelines/modules/ROOT/nav.adoc +++ b/guidelines/modules/ROOT/nav.adoc @@ -57,7 +57,8 @@ ** xref:Perl.adoc[Perl] ** xref:PHP.adoc[PHP] ** xref:Python.adoc[Python] -*** xref:Python_Appendix.adoc[Python_Appendix] +*** xref:Python_201x.adoc[Python (201x-era)] +*** xref:Python_Appendix.adoc[Python Appendix] ** xref:R.adoc[R] ** xref:Ruby.adoc[Ruby] ** xref:Rust.adoc[Rust] diff --git a/guidelines/modules/ROOT/pages/Naming.adoc b/guidelines/modules/ROOT/pages/Naming.adoc index 268349e..3b7768e 100644 --- a/guidelines/modules/ROOT/pages/Naming.adoc +++ b/guidelines/modules/ROOT/pages/Naming.adoc @@ -414,47 +414,29 @@ For details on the PHP naming scheme, see Packaging/PHP#NamingScheme . === Python modules -Python packaging is complicated by the fact -that there are two partially incompatible language versions in use, -here called Python2 and Python3. -However, the common case is for one source package to produce -a binary package for each Python version, -and this results in a naming convention where -the source package does not generally produce a binary package of the same name. +Naming of Python modules is fully covered +in the xref:Python.adoc#_naming[Naming section] +of the Python Packaging Guidelines. +In short: The package name SHOULD reflect the upstream name of the Python module, and SHOULD generally take into account the name of the module used when importing it in Python scripts. This name will be prefixed depending on the type of the package. -Note that when a module that has a dot in its name, the -<> about changing "." to "-" applies. +* A built (i.e. non-SRPM) package for a _Python library_ +MUST be named with the prefix `+python3-+`. +* A source package containing primarily a _Python library_ +MUST be named with the prefix `+python-+`. -=== Python source package naming +The character `pass:[+]` is reserved for +xref:Python.adoc#_extras[Extras]. -Source packages for Python modules SHOULD be named using the `+python-+` prefix. - -'''Examples: ''' - -.... -python-psycopg (python module named psycopg) -python-PyQt4 (python module named PyQt4) -.... - -However, it does occur that two separately maintained modules -have the same name but are targeted against different versions of Python. -In this case, source package for the software targeted at Python2 -SHOULD take a `+python2-+` prefix, -and the source package for the Python3 version -SHOULD have a name beginning with `+python3-+`. ==== Python2 binary package naming -Python2 binary packages MUST be named using a `+python2-+` prefix. - -==== Python3 binary package naming - -Python3 binary packages MUST be named with a prefix of `+python3-+`. +Python 2 binary packages MUST be named using a `+python2-+` prefix. +Note that Python 2 is deprecated in Fedora and requires an explicit exception. === R modules diff --git a/guidelines/modules/ROOT/pages/Python.adoc b/guidelines/modules/ROOT/pages/Python.adoc index 0e9dcd9..4107a96 100644 --- a/guidelines/modules/ROOT/pages/Python.adoc +++ b/guidelines/modules/ROOT/pages/Python.adoc @@ -1,526 +1,1739 @@ = Python Packaging Guidelines -:last-reviewed: 2020-08-05 +:last-reviewed: 2021-06-04 :toc: -== Python Version Support +//// + +Additional style notes: + +- Normative points are at the beginnings of sections +and include highlighted keywords like **MUST** or **SHOULD**. +So, if a reader only wants the rules, +they can skip remainders of sections +(discussion, examples, clarifications, ...) +without missing context. + +- Some sections have additional IDs +(see https://docs.asciidoctor.org/asciidoc/latest/sections/custom-ids/ ) +to avoid breaking links to removed/renamed sections. + +- In text, macros appear as Literal Monospace with a leading percent sign. +Use curly braces for value-like macros, but not command-like macros +(for example: `%{version}` but `%autosetup` or `%check`). + +- The first mention of a macro generally links to the documentation. +The link includes some non-monospace text, e.g. "the `+%{version}+` macro". +(In the current Fedora docs style, links aren't visible in monospace text). + +- The documentation for a macro has the macro's name as ID. + +//// + + +This version of Python Packaging Guidelines +is in effect since 2021 +and represents a major rewrite and paradigm shift. +Not all packages are updated to reflect this. +Older guidelines are still being kept up to date, +and existing packages *MAY* use them instead of this document: + +* xref:Python_201x.adoc[“201x-era” Python packaging guidelines] +(Packages using these usually use the +<> +or <> +or call `+setup.py install+`.) + +* xref:Python_Appendix.adoc#_python_2_packages[Python 2 appendix] +Note that Python 2 packages require a FESCo exception. + +//// +NOTE: A summary of changes from the older guidelines +is available at https://hackmd.io/@python-maint/rJmQQc4DP +//// + +NOTE: These guidelines only support Fedora 31+. +For older releases (such as in EPEL 8), +consult the xref:Python.adoc[201x-era guidelines]. + +The two <> below +apply to all software in Fedora that uses Python at build- or run-time. + +The rest of the Guidelines apply to packages that ship code +that can be imported with Python’s `+import+` statement. +Specifically, that is all packages +that install files under `+/usr/lib*/python*/+`. + +Except for the two “Distro-wide guidelines”, +these Guidelines do not apply to simple one-file scripts or utilities, +especially if these are included with software +not written in Python. +However, if an application (e.g. CLI tool, script or GUI app) +needs a more complex Python library, +the library *SHOULD* be packaged as an importable library +under these guidelines. + +A major goal for Python packaging in Fedora is to +_harmonize with the wider Python ecosystem_, +that is, +the https://pypa.io[Python Packaging Authority] (PyPA) standards +and the https://pypi.org/[Python Package Index] (PyPI). +Packagers *SHOULD* be prepared to get involved with upstream projects +to establish best practices as outlined here. +We wish to improve both Fedora and the wider Python ecosystem. + +NOTE: Some build tools (like CMake or autotools) +may not work with the latest PyPA standards yet. +(For example, they might generate `+.egg-info+` directories +rather than `+.dist-info+`.) +While this document's normative points (MUST/SHOULD) are tool-agnostic, +many of the practical tips and helper macros will not be applicable. +If this affects you, consider contacting the +https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/[Python SIG] +for guidance +and/or following the xref:Python_201x.adoc[older guidelines] +for the time being. + +NOTE: Fedora’s Python SIG not only develops these guidelines, +but it’s also involved in PyPA standards and Python packaging best practices. +Check out https://fedoraproject.org/wiki/SIGs/Python[the wiki] or +https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/[mailing list] +if you need help or wish to help out. + +== Distro-wide guidelines + +=== BuildRequire python3-devel + +*Every* package that uses Python (at runtime and/or build time) +and/or installs Python modules +*MUST* explicitly include `+BuildRequires: python3-devel+` +in its `+.spec+` file, +even if Python is not actually invoked during build time. + +If the package uses an alternate Python interpreter instead of `+python3+` +(e.g. `+pypy+`, `+jython+`, `+python2.7+`), +it *MAY* instead require the corresponding `+*-devel+` package. + +The `+*-devel+` package brings in relevant RPM macros. +It may also enable automated or manual checks: for example, +Python maintainers use this requirement to list packages +that use Python in some way +and might be affected by planned changes. + +=== Mandatory macros + +The following macros *MUST* be used where applicable. + +The expansions in parentheses are provided only as reference/examples. + +The macros are defined for you in all supported Fedora and EPEL +versions. + +// Keep the list synced with "Macro Reference" below + +* `+%{python3}+` (`+/usr/bin/python3+`): +The Python interpreter. +For example, this macro should be +used for invoking Python from a `+spec+` file script, +passed to `+configure+` scripts to select a Python executable, +or used as `+%{python3} -m pip+` to run a Python-based tool. ++ +If the packaged software invokes Python at _run time_ +(as opposed to running Python to build/test it), +it might be necessary to pass flags to `+%{python3}+` +to isolate it from user-installed packages. +See <> for details. ++ +* `+%{python3_version}+` (e.g. `+3.9+`, `+3.10+`): +Version of the Python interpreter. ++ +* `+%{python3_version_nodots}+` (e.g. `+39+`, `+310+`): +Version of the Python interpreter without the dot. +* `+%{python3_sitelib}+` (e.g. `+/usr/lib/python3.9/site-packages+`): +Where pure-Python modules are installed. ++ +* `+%{python3_sitearch}+` (e.g. `+/usr/lib64/python3.9/site-packages+`): +Where Python extension modules (native code, e.g. compiled from C) are +installed. + +The rest of this document uses these macros, +along with `+%{_bindir}+` (`+/usr/bin/+`), +instead of the raw path names. + +=== Python implementation support[[_multiple_python_runtimes]] + +Fedora primarily targets _CPython_, +the reference implementation of the Python language. +We generally use “Python” to mean CPython. + +Alternate implementations like `+pypy+` are available, +but currently lack comprehensive tooling and guidelines for packaging. +When targetting these, there are no hard rules +(except the general Fedora packaging guidelines). +But please try to abide by the _spirit_ of these guidelines. +When in doubt, consider consulting the Python SIG. + +=== Python version support + +Fedora packages *MUST NOT* depend on other versions +of the CPython interpreter +than the current `+python3+`. + +In Fedora, Python libraries are packaged for a single version of Python, +called `+python3+`. +For example, in Fedora 32, `+python3+` is Python 3.8. + +In the past, there were multiple Python stacks, +e.g. `+python3.7+` and `+python2.7+`, +installable together on the same machine. +That is also the case in some projects that build _on top_ of Fedora, +like RHEL, EPEL and CentOS. +Fedora might re-introduce parallell-installable stacks in the future +(for example if a switch to a new Python version needs a transition period, +or if enough interested maintainers somehow appear). + +Fedora does include alternate interpreter versions, +e.g. `+python2.7+` or `+python3.5+`, +but these are meant only for developers that need to test upstream code. +Bug and security fixes for these interpreters only cover this use case. +Packages such as `+pip+` or `+tox+`, +which enable setting up isolated environments +and installing third-party packages into them, +*MAY*, as an exception to the rule above, use these interpreters +as long as this is coordinated with the maintainers +of the relevant Python interpreter. -In Fedora we have multiple Python runtimes, -one for each supported major Python release. -At this point that's one for python3.x and one for python2.7. -However the Python 2 stack will be removed from Fedora and is xref:deprecating-packages.adoc[deprecated]. -Upstream support for the python2 interpreter officially ends in 2020. -If a piece of software supports python3, -it MUST be packaged for python3. -Software using python2 MUST NOT be newly packaged into Fedora without FESCo exception. +== Naming -For guidelines on maintaining already existing python2 packages, see the xref:Python_Appendix.adoc[appendix]. +Python packages have several different names, +which should be kept in sync but will sometimes differ +for historical or practical reasons. +They are: -== Multiple Python Runtimes +* the Fedora _source package name_ (or _component name_, `+%{name}+`), +* the Fedora _built RPM name_, +* the _project name_ used on https://pypi.org/[PyPI] +or by https://pip.pypa.io[pip], and +* the _importable module name_ used in Python +(a single package may have multiple importable modules). -On Fedora `/usr/bin/python` is, if it is installed, a symbolic link to `/usr/bin/python3`. -It was a symbolic link to `/usr/bin/python2` on previous releases. +Some examples (both good and worse): -Packages in Fedora MUST NOT use `/usr/bin/python`. Instead packages for Python 3 MUST use `/usr/bin/python3` (even if upstream supports both Python 2 and 3). As a result of that `/usr/bin/python` (as well as `/usr/bin/env python` and similar) MUST NOT be used in shebang lines or as a dependency of a package. All uses of unversioned python executables in shebang lines will fail the build. -These shebangs MUST be fixed (for example by using the `+%py3_shebang_fix+` macro in the spec file). -If it is necessary to disable the checks, please see the information in xref:index#_shebang_lines[Shebang lines]. +[cols=",,,",options="header",] +|=== +|Fedora component |Built RPM |Project name |Importable module +|`+python-requests+` |`+python3-requests+` |`+requests+` |`+requests+` +|`+python-django+` |`+python3-django+` |`+Django+` |`+django+` +|`+PyYAML+` |`+python3-pyyaml+` |`+pyyaml+` |`+yaml+` +|`+python-ldap+` |`+python3-ldap+` |`+python-ldap+` |`+ldap+`, `+ldif+`, etc. +|`+python-pillow+` |`+python3-pillow+` |`+pillow+` |`+PIL+` +|=== -All Python runtimes have a virtual provide for `+python(abi) = $MAJOR.$MINOR+`. For example, the Python 3.7 runtime package has: +Elsewhere in this text, +the metavariables `+SRPMNAME+`, `+RPMNAME+`, `+PROJECTNAME+`, `+MODNAME+` +refer to these names, respectively. + +=== Canonical project name + +Most of these names are case-sensitive machine-friendly identifiers, +but the _project name_ has human-friendly semantics: +it is case-insensitive +and treats some sets of characters (like `+._-+`) specially. +For automated use, +it needs to be normalized to a canonical format +used by Python tools and services such as setuptools, pip and PyPI. +For example, the canonical name of the `+Django+` project +is `+django+` (in lowercase). +This normalization is defined in +https://www.python.org/dev/peps/pep-0503/#normalized-names[PEP 503], +and <> implements it +for Fedora packaging. + +Elsewhere in this text, +the metavariable `+DISTNAME+` refers to the canonical form of the project name. + +Note that in some places, the original, +non-normalized project name must be used. +For example, +the <> and the `+%autosetup+` macro +need `+Django+`, not `+django+`. + +=== Name limitations + +The character `pass:[+]` in names of built (i.e. non-SRPM) packages +that include `+.dist-info+` or `+.egg-info+` directories +is reserved for <> and *MUST NOT* be used for any other purpose. + +As an exception, `pass:[+]` characters *MAY* appear at the _end_ of such names. + +The `pass:[+]` character triggers +the automatic dependency generator for extras. + +Replace any `pass:[+]` signs in the upstream name with `+-+`. +Omit `pass:[+]` signs on the beginning of the name. +Consider adding `Provides` for the original name with `pass:[+]` characters +to make the package easier to find for users. + +=== Library naming + +A built (i.e. non-SRPM) package for a _Python library_ +*MUST* be named with the prefix `+python3-+`. +A source package containing primarily a _Python library_ +*MUST* be named with the prefix `+python-+`. + +The Fedora package’s name *SHOULD* contain +the <>. +If possible, the project name *SHOULD* be the same +as the name of the main importable module, +with underscores (`+_+`) replaced by dashes (`+-+`). + +If the importable module name and the project name do not match, +users frequently end up confused. +In this case, packagers *SHOULD* ensure that upstream is aware of the problem +and (especially for new packages where renaming is feasible) +strive to get the package renamed. +The Python SIG is available for assistance. + +A _Python library_ is a package meant to be imported in Python, +such as with `+import requests+`. +Tools like _Ansible_ or _IDLE_, whose code is importable +but not primarily meant to be imported from other software, +are not considered libraries in this sense. +So, this section does not apply for them. +(See the +xref:index#_libraries_and_applications[general Libraries and Applications guidelines] +for general guidance.) + +The Fedora component (source package) name for a library +should be formed by taking the _canonical project name_ +and prepending `+python-+` if it does not already start with `+python-+`. +This may leads to conflicts +(e.g. between https://pypi.org/project/bugzilla/[bugzilla] +and https://pypi.org/project/python-bugzilla/[python-bugzilla]). +In that case, ensure upstream is aware of the potentially confusing naming +and apply best judgment. + +=== Application naming + +Packages that primarily provide applications, services +or any kind of executables *SHOULD* be named +according to the general xref:Naming.adoc[Fedora naming guidelines] +(e.g. `+ansible+`). + +Consider adding a virtual provide according to <> above +(e.g. `+python3-PROJECTNAME+`), +if it would help users find the package. -.... -$ rpm -q --provides python3 | grep abi -python(abi) = 3.7 -.... +== Files to include -Python modules using these runtimes should have a corresponding "Requires" line on the Python runtime that they are used with. This is done automatically for files below `+/usr/lib[^/]*/python${PYVER}+` +=== Source files and bytecode cache[[_byte_compiling]] -Mirroring the policy for regular packages, the Python-version-specific subpackages of your package MUST NOT be removed in a release branch of Fedora. +Packages *MUST* include the source file (`+*.py+`) +*AND* the bytecode cache (`+*.pyc+`) for each pure-Python importable module. +The source files *MUST* be included in the same package as the bytecode cache. -== Naming +Scripts that are not importable +(typically ones in `+%{_bindir}+` or `+%{_libexecdir}+`) +*SHOULD NOT* be byte-compiled. + +The cache files are found in a `+__pycache__+` directory +and have an interpreter-dependent suffix like `+.cpython-39.pyc+`. + +The cache is not necessary to run the software, +but if it is not found, Python will try to create it when a module is imported. +If this succeeds, the file is not tracked by RPM +and it will linger on the system after uninstallation. +If it does not succeed, users can get spurious SELinux AVC denials in the logs. + +Normally, byte compilation (generating the cache files) +is done for you by the `+brp-python-bytecompile+` +xref:index.adoc#_brp_buildroot_policy_scripts[BRP script], +which runs automatically +after the `+%install+` section of the spec file has been processed. +It byte-compiles any `+.py+` files that it finds +in `+%{python3_sitelib}+` or `+%{python3_sitearch}+`. + +You must include these files of your package +(i.e. in the `+%files+` section). + +If the code is in a subdirectory (importable package), +include the entire directory: + +[source,spec] +---- +%files +%{python3_sitelib}/foo/ +---- + +Adding the trailing slash is best practice for directories. + +However, this cannot be used for top-level modules (those directly in +e.g. `+%{python3_sitelib}+`), because both `+%{python3_sitelib}+` and +`+%{python3_sitelib}/__pycache__/+` are owned by Python itself. Here, the +`+%pycached+` macro can help. It expands to the given `+*.py+` source file +and its corresponding cache file(s). For example: + +[source,spec] +---- +%files +%pycached %{python3_sitelib}/foo.py +---- + +expands roughly to: + +[source,spec] +---- +%files +%{python3_sitelib}/foo.py +%{python3_sitelib}/__pycache__/foo.cpython-3X{,.opt-?}.pyc +---- + +==== Manual byte compilation[[manual-bytecompilation]] + +If you need to bytecompile stuff +outside of `+%{python3_sitelib}+`/`+%{python3_sitearch}+`, +use the <>. + +For example, +if your software adds `+%{_datadir}/mypackage+` to Python’s import path +and imports package `+foo+` from there, you will need to compile `+foo+` with: + +[source,spec] +---- +%py_byte_compile %{python3} %{buildroot}%{_datadir}/mypackage/foo/ +---- + +=== Dist-info metadata + +Each Python package *MUST* include _Package Distribution Metadata_ +conforming to https://packaging.python.org/specifications/[PyPA specifications] +(specifically, +https://packaging.python.org/specifications/recording-installed-packages/[Recording installed distributons]). + +The metadata *SHOULD* be included +in the same subpackage as the main importable module, +if there is one. + +This applies to libraries (e.g. `+python-requests+`) +as well as tools (e.g. `+ansible+`). + +When software is split into several subpackages, +it is OK to only ship metadata in one built RPM. +In this case, consider working with upstream +to also split the upstream project. + +The metadata takes the form of a `+.dist-info+` directory +installed in `+%{python3_sitelib}+` or `+%{python3_sitearch}+`, +and contains information that tools like +https://docs.python.org/3/library/importlib.metadata.html[`+importlib.metadata+`] +use to introspect installed libraries. + +For example, a project named `+MyLib+` with importable package `+mylib+` +could be packaged with: -The source package for a Python library MUST be named with the `python-` prefix. A built package however must include the Python major version in the name, using the `python3-` prefix. This is accomplished by adding a subpackage. See example below. +[source,spec] +---- +%files -p python3-mylib +%{python3_sitelib}/mylib/ +%{python3_sitelib}/MyLib-%{version}.dist-info/ +%doc README.md +%license LICENSE.txt +---- -This rule does not apply to applications. +Note that some older tools instead put metadata in an `+.egg-info+` directory, +or even a single file. +This won’t happen if you use the `+%pyproject_wheel+` macro. +If your package uses a build system +that generates an `+.egg-info+` directory or file, +please contact Python SIG. -The character `+` in names of built packages (i.e. non-SRPM) -that include `.dist-info` or `.egg-info` directories is reserved for <> and MUST NOT be used for any other purpose. -The `+` character triggers the automatic dependency generator for extras. -Replace any `+` signs in the upstream name with `-`, -or omit them when at the beginning of the name. -As an exception, `+` characters are permitted at the _end_ of the name. +As an exception, +the Python standard library *MAY* ship without this metadata. -== Dependencies +=== Explicit lists -Packages building for Python 3 will need `BuildRequires: python3-devel`. -Most of them will also need `BuildRequires: python3-setuptools`. -When in doubt, inspect the `setup.py` file for `setuptools` import. +Packages *MUST NOT* own shared directories owned by Python itself, +such as the top-level `+__pycache__+` directories +(`+%{python3_sitelib}/__pycache__+`, `+%{python3_sitearch}/__pycache__+`). -Packages MUST NOT have dependencies (either build-time or runtime) on packages named with the unversioned `python-` prefix. Dependencies on Python packages instead MUST use names beginning with `python3-`. +Packagers *SHOULD NOT* simply glob everything under a shared directory. -=== Automatically generated dependencies +In particular, the following *SHOULD NOT* be used in `+%files+`: -Packages MAY use the automatic Python dependency generator. This generator uses upstream egg/dist metadata (such as https://python-packaging.readthedocs.io/en/latest/dependencies.html[setuptool's install_requires]) to determine what the package should depend on. The generator parses the installed metadata from `+/usr/lib(64)?/pythonX.Y/site-packages/[^/]+++.(egg|dist)-info/requires.txt+`, so it will not work with software that uses plain https://docs.python.org/3/distutils/[distutils]. +* `+%{python3_sitelib}/*+` +* `+%{python3_sitearch}/*+` +* `+%{python_sitelib}/*+` +* `+%{python_sitearch}/*+` +* `+%{_bindir}/*+` +* `+%pyproject_save_files '*'+` +* `+%pyproject_save_files +auto+` + +This rule serves as a check against common mistakes +which are otherwise hard to detect. +It does limit some possibilities for automation. + +The most common mistakes this rule prevents are: + +* installing a test suite system-wide as an importable module named `+test+`, +which would then conflict with other such packages, and +* upstream adding commands in `+%{_bindir}/*+` – +you should always check such changes for +https://docs.fedoraproject.org/en-US/packaging-guidelines/Conflicts/#_common_conflicting_files_cases_and_solutions[conflicts] +(especially with non-Python software), +and keep the list of such files explicit and auditable. + +== PyPI parity + +Every Python package in Fedora *SHOULD* also be available +on https://pypi.org[the Python Package Index] (PyPI). + +The command `+pip install PROJECTNAME+` *MUST* +install the same package (possibly in a different version), +install nothing, +or fail with a reasonable error message. + +If this is not the case, +the packager *SHOULD* contact upstream about this. +The goal is to get the project name registered or blocked on PyPI, +or to otherwise ensure the rule is followed. + +If your package is not or cannot be published on PyPI, you can: + +* Ask upstram to publish it +* If you wish: publish it to PyPI yourself and maintain it +* Ask mailto:python-devel@lists.fedoraproject.org[Python SIG] +to _block_ the name on PyPI for you +* Email mailto:admin@pypi.org[PyPI admins] to block the name for you, +giving the project name and explaining the situation +(for example: the package cannot currently be installed via `+pip+`). +You can ask questions and discuss the process at the +https://discuss.python.org/t/block-names/4045[Python Discourse]. + +NOTE: Project names that were in Fedora but not on PyPI +when these guidelines were proposed are _blocked_ from being uploaded to PyPI. +This prevents potential trolls from taking them, +but it also blocks legitimate owners. +If your package is affected, contact the Python SIG or +https://github.com/pypa/pypi-support/issues/new?labels=PEP+541&template=pep541-request.md&title=PEP+541+Request%3A+PROJECT_NAME[file a PyPA issue] +and mention `+@encukou+`. + +If your package’s project name conflicts with a different package on PyPI, +change the project name. +As painful as it is, +we need to use a single global namespace across the Python ecosystem. +Software that is not written specifically for Fedora already expects +that project names use the PyPI namespace: +for example, if a third-party library identifies a dependency by name, +we don’t want that dependency satisfied by an unrelated Fedora package. + +As always, +xref:index.adoc#_general_exception_policy[specific exceptions] +can be granted by the Packaging Committee. + +== Provides and requirements[[_provides]] + +=== Provides for importable modules[[_the_py_provides_macro]] + +For any module intended to be used in Python 3 with `+import MODNAME+`, +the package that includes it *SHOULD* provide `+python3-MODNAME+`, +with underscores (`+_+`) replaced by dashes (`+-+`). + +This is of course always the case if the package is named `+python3-MODNAME+`. +If the subpackage has some other name, +then add `+%py_provides python3-MODNAME+` explicitly. +See the following section to learn about `+%py_provides+`. + +[#Automatic-unversioned-provides] +=== Automatic python- and python3.X- provides -This generates run time requires in the form of `+pythonX.Ydist(foo)+`. If the generated dependencies are not accurate, additional ones can still be added manually. To remove some, a packager MAY modify upstream-provided metadata (usually specified in the `setup.py` file) in the `+%prep+` section of the specfile or fall back to xref:AutoProvidesAndRequiresFiltering.adoc[filtering] those dependencies. +For any `+FOO+`, +a package that provides `+python3-FOO+` *SHOULD* use `+%py_provides+` +or an automatic generator +to also provide `+python-FOO+` +and `+python3.X-FOO+`, where `+X+` is the minor version of the interpreter. -The packager MUST inspect the generated requires for correctness. All dependencies MUST be resolvable within the targeted Fedora version. +The provide *SHOULD NOT* be added manually: +if a generator or macro is not used, +do not add the `+python-FOO+` / `+python3.X-FOO+` provides at all. -As an example, the upstream notebook package has (as of version 5.6.0): +On Fedora 33+, this is done automatically for package names by a generator. +If absolutely necessary, the generator can be disabled +by undefining <<__pythonname_provides,the `+%__pythonname_provides+` macro>>. -[source,python] +On releases older than Fedora 33, +or (for technical reasons) for packages without files, +or for provides that aren’t package names, +the generator will not work. +For these cases, the following invocation will provide `+python3-FOO+`, +`+python-FOO+` and `+python3.X-FOO+`: + +[source,spec] ---- -install_requires = [ -    'jinja2', -    'tornado>=4', -    'pyzmq>=17', -    'ipython_genutils', -    'traitlets>=4.2.1', -    'jupyter_core>=4.4.0', -    'jupyter_client>=5.2.0', -    'nbformat', -    'nbconvert', -    'ipykernel', -    'Send2Trash', -    'terminado>=0.8.1', -    'prometheus_client' -], +%py_provides python3-FOO ---- -And the resulting dependencies: +Using the generator or macro is important, because the specific form of +the provide may change in the future. + +=== Machine-readable provides[[_automatic_provides_with_a_standardized_name]] + +Every Python package *MUST* provide `+python3dist(DISTNAME)+` +*and* `+python3.Xdist(DISTNAME)+`, +where `+X+` is the minor version of the interpreter +and `+DISTNAME+` is the <> +corresponding to the <>. +For example, `+python3-django+` would provide +`+python3dist(django)+` and `+python3.9dist(django)+`. + +This is generated automatically from the dist-info metadata. +The provide *SHOULD NOT* be added manually: +if the generator fails to add it, the metadata *MUST* be fixed. + +These _Provides_ are used for automatically generated _Requires_. + +If absolutely necessary, the automatic generator can be disabled by using the +<>. +Consider discussing your use case with the Python SIG if you need to do this. + +=== Dependencies + +As mentioned above, +each Python package *MUST* explicitly BuildRequire `+python3-devel+`. + +Packages *MUST NOT* have dependencies (either build-time or runtime) +with the unversioned prefix `+python-+` +if the corresponding `+python3-+` dependency can be used instead. + +Packages *SHOULD NOT* have explicit dependencies +(either build-time or runtime) +with a minor-version prefix such as `+python3.8-+` or `+python3.8dist(+`. +Such dependencies *SHOULD* instead be automatically generated +or a macro should be used to get the version. + +Packages *SHOULD NOT* have an explicit runtime dependency on `+python3+`. + +Instead of depending on `+python3+`, +packges have an automatic dependency on `+python(abi) = 3.X+` +when they install files to `+%{python3_sitelib}+` or `+%{python3_sitearch}+`, +or they have an automatic dependency on `+/usr/bin/python3+` +if they have executable Python scripts, +or they have an automatic dependency on `+libpython3.X.so.1.0()+` +if they embed Python. + +These rules help ensure a smooth upgrade path +when `+python3+` is updated in new versions of Fedora. -.... -python3.7dist(ipykernel) -python3.7dist(ipython-genutils) -python3.7dist(jinja2) -python3.7dist(jupyter-client) >= 5.2 -python3.7dist(jupyter-core) >= 4.4 -python3.7dist(nbconvert) -python3.7dist(nbformat) -python3.7dist(prometheus-client) -python3.7dist(pyzmq) >= 17 -python3.7dist(send2trash) -python3.7dist(terminado) >= 0.8.1 -python3.7dist(tornado) >= 4 -python3.7dist(traitlets) >= 4.2.1 -.... +=== Automatically generated dependencies[[_requires_and_buildrequires_with_standardized_names]] -Note that any `+.0+` suffixes are removed from version numbers to match the behavior of Python tools. -(https://www.python.org/dev/peps/pep-0440/#final-releases[PEP 440] specifies that `+X.Y+` and `+X.Y.0+` are treated as equal.) +Packages *MUST* use the automatic Python run-time dependency generator. -This generator is enabled by default in Fedora. If a packager wishes to explicitly opt out of the generator because the upstream metadata are not applicable, a packager SHOULD opt out explicitly by adding: +Packages *SHOULD* use the opt-in build-dependency generator if possible. -`+%{?python_disable_dependency_generator}+` +The packager *MUST* inspect the generated requires for correctness. +All dependencies *MUST* be resolvable within the targeted Fedora version. -Although this statement can be used anywhere in the spec, we recommend putting it just before the main package's `+%description+` declaration. +Any necessary changes *MUST* be done by patches +or modifying the source (e.g. with `+sed+`), +rather than disabling the generator. +The resulting change *SHOULD* be offered to upstream. +As an exception, xref:AutoProvidesAndRequiresFiltering.adoc[filtering] +*MAY* be used for temporary workarounds +and xref:index.adoc#bootstrapping[bootstrapping]. -=== Python Extras +Dependencies covered by the generators *SHOULD NOT* +be repeated in the `+.spec+` file. +(For example, if the generator finds a `+requests+` dependency, +then `+Requires: python3-requests+` is redundant.) -https://www.python.org/dev/peps/pep-0508/#extras[Python extras] are a way for Python projects to declare that extra dependencies are required for additional functionality. +The automatically generated requirements +are in the form `+python3.Xdist(DISTNAME)+`, +potentially augmented with version requirements or combined together +with https://rpm.org/user_doc/boolean_dependencies.html[rich dependencies]. +Any `+.0+` suffixes are removed from version numbers +to match the behavior of Python tools. +(https://www.python.org/dev/peps/pep-0440/[PEP 440] specifies +that `+X.Y+` and `+X.Y.0+` are treated as equal.) -For example, `requests` has several standard dependencies (e.g. `urllib3`). +Note that the generators only cover Python packages. +Other dependencies, often C libraries like `+openssl-devel+`, +must be specified in the `+.spec+` file manually. + +Where the requirements are specified in the source +depends on each project’s build system and preferences. +Common locations are `+pyproject.toml+`, `+setup.py+`, `+setup.cfg+`, +`+config.toml+`. + +==== Run-time dependency generator + +The automatic runtime dependency generator uses package metadata +(as recorded in installed `+*.dist-info+` directories) +to determine what the package depends on. + +In an emergency, you can opt-out from running the requires generator by +<> +to the package (usually, just before the main package’s `+%description+`). + +==== Build-time dependency generator + +The opt-in (but strongly recommended) build-time dependency generator +gathers information from +https://www.python.org/dev/peps/pep-0517/#source-trees[`+pyproject.toml+` build-system information] +(with fallback to `+setuptools+`) plus a standardized +https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-wheel[build-system hook] +to gather further requirements. +See <> +for more details. + +Note that with the `+-r+` flag, the generator will include run-time +requirements in BuildRequires. This is useful for running tests and for +checking that the dependencies are available in Fedora. + +=== Test dependencies + +See the <> section. + +=== Extras[[_python_extras]] + +Python extras are a way for Python projects +to declare that extra dependencies are required for additional functionality. + +For example, +`+requests+` has several standard dependencies (e.g. `+urllib3+`). But it also declares an _extra_ named `+requests[security]+`, -which lists additional dependencies (e.g. `cryptography`). -Unlike RPM subpackages, -extras can only specify additional dependencies, +which lists additional dependencies (e.g. `+cryptography+`). +Unlike RPM subpackages, extras can only specify additional dependencies, not additional files. The main package will work if the optional dependency is not installed, but it might have limited functionality. Python tools treat extras as virtual packages. -For example, -if a user runs `+pip install requests[security]+`, +For example, if a user runs `+pip install 'requests[security]'+`, or installs a project that depends on `+requests[security]+`, -both `requests` and `cryptography` will be installed. +both `+requests+` and `+cryptography+` will be installed. -Starting with Fedora 33, -extras are usually provided by packages with no files. +In Fedora, extras are usually provided by packages with no files. Instead of square brackets, -Fedora package names conventionally use the `+` character to separate the package name and the _extra_ name, -e.g. the package would be named `python3-requests+security`. -The plus sign is valid in RPM package names, -but not in Python canonical project names nor in extras identifiers. - -Python packages SHOULD have `Provides` for all extras the upstream project specifies, -except those that are not useful for other packages -(for example build/development requirements, -commonly named `dev`, `doc` or `test`). +Fedora package names conventionally use the `pass:[+]` character +(which is valid in RPM package names, +but not in Python canonical project names nor in extras identifiers). -A package that provides a Python extra MUST provide `+python3dist(…[…])+` and `+python3.Xdist(…[…])+`, -for example, `+python3.9dist(requests[security])+`. -These requirements SHOULD be generated using the automatic dependency generator. +==== Handling extras -A package that provides a Python extra MUST require the extra's main package with exact NEVR. +Python packages *SHOULD* have Provides +for all extras the upstream project specifies, except: -A subpackage that primarily provides one Python extra SHOULD be named by appending `+` and the extra name to the main package name. -For example, `++python3-requests+security++`. - -The most straightforward way to provide an extra is with a dedicated subpackage containing no files (a "metapackage"). -This case can be automated with the `+%python_extras_subpkg+` macro. - -Alternative approach: +* those that are not useful for other packages +(for example build/development requirements, +commonly named `+dev+`, `+doc+` or `+test+`), and +* those that have requirements that are not packaged in Fedora. + +A package that provides a Python extra +*MUST* provide `+python3dist(DISTNAME[EXTRA])+` +*and* `+python3.Xdist(DISTNAME[EXTRA])+`, +where `+X+` is the minor version of the interpreter, +`+DISTNAME+` is the <>, +and `+EXTRA+` is the name of a single extra. +For example, `+python3.9dist(requests[security])+`. +These requirements *SHOULD* be generated +using the automatic dependency generator. + +A package that provides a Python extra +*MUST* require the extra’s main package with exact NEVR. + +A subpackage that primarily provides one Python extra *SHOULD* be named +by appending `pass:[+]` and the extra name to the main package name. +For example, `+python3-requests+security+`. + +The most straightforward way to provide an extra +is with a dedicated subpackage containing no files (a “metapackage”). +This case can be automated with +the <> +or the <>. + +This is not the only way: when some extra is always useful in a distro, it can be provided by the main package; when several extras are related, they may be provided by a single subpackage. -However, -having one dedicated subpackage per extra -allows you to use the automatic dependency generator to ensure that the extras' requirements will stay in sync with upstream. -If you create a dedicated subpackage and want it to be always/usually installed, you MAY Require/Recommend/Suggest it from the main package. +However, having one dedicated subpackage per extra +allows you to use the automatic dependency generator +to ensure that the extras’ requirements will stay in sync with upstream. +If you create a dedicated subpackage +and want it to be always/usually installed, +you can _Require_/_Recommend_/_Suggest_ it from the main package. The dependency generator for extras activates if the following holds: -- The package must contain the `.egg-info`/`.dist-info` directory, usually as `+%ghost+`. -- The package name must end with `+EXTRA` (where `EXTRA` is the extra name). +* The package name must end with `++EXTRA+` +(where `+EXTRA+` is the extra name). +* The package must contain the `+.dist-info+` directory, usually as `+%ghost+`. -As an example, -the extra subpackage for `+requests[security]+` can be specified using the `+%python_extras_subpkg+` convenience macro as follows. -The macro takes the main package name and name(s) of the extra(s) as well as path to the `.egg-info` or `.dist-info` directory: +===== Example and convenience macros +The extra subpackage for `+setuptools_scm[toml]+` can be specified +using the `+%pyproject_extras_subpkg+` convenience macro as follows. +The macro takes the main package name and name(s) of the extra(s): + +[source,spec] ---- -%{?python_extras_subpkg:%python_extras_subpkg -n python3-requests -i %{python3_sitelib}/*.egg-info security} +%pyproject_extras_subpkg -n python3-setuptools_scm toml ---- -For this case, -the extras dependency generator will read upstream metadata from the `.egg-info` directory. -If it finds that the `security` extra has a dependency on `cryptography`, -it will generate `+Requires: python3.Xdist(cryptography)+`, -`+Provides: python3dist(requests[security])+` -(and the corresponding `+python3.Xdist+` variant). +If not using `+%pyproject_install+`, +you will instead need to use `+%python_extras_subpkg+` +and pass a path to the `+dist-info+` directory: -If you need additional features that the `+%python_extras_subpkg+` macro does not cover, +[source,spec] +---- +%python_extras_subpkg -n python3-setuptools_scm -i %{python3_sitelib}/*.dist-info toml +---- + +For this case, +the extras dependency generator will read upstream metadata +from the `+.dist-info+` directory. +If it finds that the extra requires on `+toml+`, +it will generate `+Requires: python3.Xdist(toml)+` +and `+Provides: python3dist(setuptools-scm[toml])+` +(and the corresponding `+python3.Xdist+` provide). + +If you need additional features +that the `+*_extras_subpkg+` macros do not cover, you will need to write the subpackage sections manually. Such features can be, for example: -- Obsoleting/providing other names (e.g. obsoleted extras packages) -- Manual strong or weak dependencies on other (possibly non-Python) packages -- Including files excluded from the main package (if such files only make sense with the extra and the base package does not fail without them) +* Obsoleting/providing other names (e.g. obsoleted extras packages) +* Manual strong or weak dependencies on other +(possibly non-Python) packages As an example of what you need to write in these cases, -the `+%python_extras_subpkg+` macro invocation above expands to the following: +both of the `+*_extras_subpkg+` macro invocations above +expand to the following: +[source,spec] ---- -%package -n python3-requests+security -Summary: Metapackage for python3-requests: security extras -Requires: python3-requests = %{?epoch:%{epoch}:}%{version}-%{release} -%description -n python3-requests+security -This is a metapackage bringing in security extras requires for python3-requests. +%package -n python3-setuptools_scm+toml +Summary: Metapackage for python3-setuptools_scm: toml extra +Requires: python3-setuptools_scm = %{?epoch:%{epoch}:}%{version}-%{release} + +%description -n python3-setuptools_scm+toml +This is a metapackage bringing in toml extra requires for python3-setuptools_scm. It contains no code, just makes sure the dependencies are installed. -%files -n python3-requests+security -%ghost %{python3_sitelib}/*.egg-info +%files -n python3-setuptools_scm+toml +%ghost %{python3_sitelib}/*.dist-info ---- -Note that the dependency generator does not add a dependency on the main package +Note that the dependency generator +does not add a dependency on the main package (the `+Requires: python3-setuptools_scm = ...+` above). If you are not using the `+%python_extras_subpkg+` macro, you need to add it manually. -NOTE: The `+%python_extras_subpkg+` can take multiple extras names to generate multiple packages. -For more options, see the https://fedoraproject.org/wiki/Changes/PythonExtras[change proposal which introduced this]. +==== Removing extras -== Provides +If an existing extra is removed from an upstream project, +the Fedora maintainer *SHOULD* try to convince upstream to re-introduce it +(with an empty list of dependencies). +If that fails, the extra *SHOULD* be Obsoleted +from either the main package or another extras subpackage. -For any module `foo` intended to be used in Python 3 with `import foo`, -the package that includes it *should* provide `python3-foo`. -This is of course always the case if the subpackage is named `python3-foo` -(as in the examples below). -If the subpackage has some other name, -then `Provides: python3-foo` should be added explicitly -(via `+%py_provides python3-foo+`, see below). +Note that removing extras is discouraged in +https://setuptools.readthedocs.io/en/latest/userguide/dependency_management.html#optional-dependencies[setuptools documentation] +(see the final paragraph of the linked section). + +==== Automatic Requires for extras + +The automatic <> +will generate Requires on `+python3.Xdist(DISTNAME[EXTRA])+` +from upstream `+Requires-Dist+` metadata. + +If the required package does not yet provide metadata for the extra, +contact the Fedora maintainer to add it. + +In an emergency, you can define +the <<_python_no_extras_requires,`+%_python_no_extras_requires+` macro>> +to avoid automatically generating _all_ extras requirements. + +== Interpreter invocation -=== The %py_provides macro +=== Shebangs -All packages that provide `+python3-...+` (for any `+...+`) SHOULD also provide `+python-...+` and `+python3.X-...+`. -Starting from Fedora 33, -most of the Python packages named `+python3-...+` will provide such names automatically via the dependency generator in `/usr/lib/rpm/fileattrs/pythonname.attr`. +Shebang lines to invoke Python *MUST* use `+%{python3}+` as the interpreter. -Any manually added virtual provides of `+python3-...+` SHOULD be done via the `+%py_provides+` macro. +Shebang lines to invoke Python *SHOULD* be `+#!%{python3} -%{py3_shebang_flags}+` +and they *MAY* include extra flags. -Instead of: +If the default flags from <> +are not desirable, +packages *SHOULD* explicitly redefine the macro to remove them. +Using `+#!%{python3}+` (`+#!/usr/bin/python3+`) +rather than e.g. `+#!/usr/bin/env python+` +ensures that the system-wide Python interpreter is used to run the code, +even if the user modifies `+$PATH+` (e.g. by activating a virtual environment). + +By default, `+-%{py3_shebang_flags}+` expands to `+-s+`, +which means _don’t add user site directory to `+sys.path+`._ +That ensures the user’s Python packages +(e.g. installed by `+pip install --user+`, +or just placed in the current directory) +don’t interfere with the RPM installed software. +Sometimes, such content is desirable, such as with plugins. +Undefining <> +or redefining it to a set of flags without `+s+`, +rather than not using the macro at all, +ensures that existing or future automation won’t add the flag. + +The <> +automatically changes all Python shebangs +in `+%{buildroot}%{_bindir}/*+` to use `+%{python3}+` +and add contents of the <> +to the existing flags. +If you’re not using that macro +or you need to change a shebang in a different directory, +you can use <> as follows: + +[source,spec] ---- -Provides: python3-pkg_resources = %{version}-%{release} +%py3_shebang_fix SCRIPTNAME … ---- -Do: +=== Invokable Python modules + +Every executable `+TOOL+` +for which the current version of Python matters +*SHOULD* also be invokable by `+python3 -m TOOL+`. + +If the software doesn’t provide this functionality, +packagers *SHOULD* ask the upstream to add it. + +This applies to tools that +modify the current Python environment (like installing or querying packages), +use Python for configuration, +or use Python to run plugins. +It does not apply to tools like GIMP or Bash +which support plugins in multiple languages +and/or have other means to specify the interpreter. + +For example, `+pip+` can be invoked as `+python3 -m pip+`. + +This allows users to accurately specify +the Python version used to run the software. +This convention works across different environments +that might not always set `+$PATH+` or install scripts consistently. + +== Using Cython[[_packages_using_cython]] + +Tightening the +xref:what-can-be-packaged.adoc/#_pregenerated_code[general Fedora policy], +packages *MUST NOT* use files pre-generated by Cython. +These *MUST* be deleted in `+%prep+` and regenerated during the build. + +As an exception, these sources *MAY* be used temporarily to prevent +build time circular dependencies by following the +xref:index.adoc#bootstrapping[bootstrapping guidelines]. +Generated files (the ones that must be deleted) +have a generic `+.c+` or `+.cpp+` extension. +Cython source files (which should stay) +usually have the `+.pyx+` or `+.pxd+` extension. + +Cython is a popular tool for writing extension modules for Python. +If compiles a Python-like language to C, which is then fed to the C compiler. +Historically, Cython was hard to use upstream as a build-time dependency. +Many projects include pre-generated C files in source +distributions to avoid users from needing to install the tool. + +Cython uses CPython’s fast-changing internal API for performance reasons. +For a new release of Python, +Cython generally needs to be updated and the C files regenerated. +In Fedora, this is frequently needed +before upstreams release re-generated sources +(e.g. for Alpha versins of Python). +Since we do not have a problem with build-time dependencies, +we always want to run the Cython step. + +For example, `+PyYAML+` removes a generated C file with: + +[source,spec] ---- -%py_provides python3-pkg_resources +rm -rf ext/_yaml.c ---- -Optionally, -supply a custom `epoch:version-release` as a second argument to `+%py_provides+`. +For another example, in `+python-lxml+` all C files are generated with Cython, +which allows removing them with: -On releases older than Fedora 33, -or (for technical limitations) for packages without files, -it is necessary to use `+%py_provides+` even for package names: +[source,spec] +---- +# Remove pregenerated Cython C sources +find -type f -name '*.c' -print -delete +---- + +Some upstreams mix generated and hand-written C files. +In such cases a grep like this one from `+scipy+` helps +(but might not be entirely future proof): +[source,spec] ---- -%package -n python3-%{srcname} -Summary: %{summary} -%py_provides python3-%{srcname} +# Remove pregenerated Cython C sources +rm $(grep -rl '/\* Generated by Cython') ---- -Packagers SHOULD try to remove explicit `+%py_provides+` calls for package names, -but MAY preserve them if they aim for compatibility with older releases or packages without files. +== Tests -NOTE: Historically, there was `+%python_provide+` macro with similar but different semantics. -It still works for compatibility reasons but it is deprecated and SHOULD NOT be used -and packagers SHOULD replace is with appropriate `+%py_provides+` call. +=== Running tests -== Automatic Provides with a standardized name +If a test suite exists upstream, +it *SHOULD* be run in the `+%check+` section. +If that is not possible with reasonable effort, +at least a basic smoke test (such as importing the packaged module) +*MUST* be run in `+%check+`. -When building a Python package, RPM looks for `+.dist-info+` and `+.egg-info+` files or directories in the `+%files+` sections of all packages. If one or more are found, RPM parses them to find the *standardized name* (i.e. dist name, name on PyPI) of the packaged software, and then automatically creates two `+Provides:+` tags in the following format: +You *MAY* exclude specific failing tests. +You *MUST NOT* disable the entire testsuite +or ignore its result to solve a build failure. -.... -Provides: python3.Ydist(CANONICAL_STANDARDIZED_NAME) -Provides: python3dist(CANONICAL_STANDARDIZED_NAME) -.... +As an exception, +you *MAY* disable tests with an appropriate `+%if+` conditional +(e.g. http://rpm.org/user_doc/conditional_builds.html[bcond]) +when xref:index.adoc#bootstrapping[bootstrapping]. -The `+3.Y+` is the Python version used (usually 3.6 and higher), and between the parentheses is the name of the software in a *canonical format* used by Python tools and services such as setuptools, pip and PyPI. The canonical name is obtained by switching the standardized name to lower case and converting all runs of non-alphanumeric characters to single “-” characters. Example: “The $$$ Tree” becomes “the-tree”. +Most errors in Python happen at run-time, +so tests are extremely important to root out issues, +especially when mass rebuilds are required. -=== Requires and BuildRequires with standardized names +Common reasons for skipping tests in `+%check+` include requiring +network access, +dependencies not packaged in Fedora, +and/or specialized hardware or resources. -These Provides tags can be used to list Requires and BuildRequires of a package using the _standardized names_ (i.e. dist name, name on PyPI) of Python modules. To make it easier, you can use the `+%{py3_dist}+` macro that accept one or more parameters: the _standardized name(s)_ of the desired Python software. It will convert the name(s) to the _canonical format_ and create the proper `+python3dist(...)+` tag(s). -In addition, you can use the `+%{py_dist_name}+` macro that simply transforms any _standardized name_ to the _canonical format_. +==== Tox -For example: +A popular testing tool, and one which is well integrated in Fedora, +is `+tox+`. +Upstream, it is commonly used to test against multiple Python versions. +In a Fedora package, BuildRequire test dependencies +(see _Test dependencies_ below) +and run `+tox+` with: -.... -BuildRequires: %{py3_dist PyMySQL} >= 0.7.5 -# => BuildRequires: python3dist(pymysql) >= 0.7.5 +[source,spec] +---- +%tox +---- -Requires: %{py3_dist virtualenv pyPEG2} -# => Requires: python3dist(virtualenv) python3dist(pypeg2) +This sets up the environment +(`+$PATH+`, `+$PYTHONPATH+`, `+$TOX_TESTENV_PASSENV+`) +and instructs `+tox+` to use the current environment rather than create new ones. +For more options, see <>. -%{py_dist_name 0-._.-._.-._.-._.-._.-._.-0} -# => 0-0 -.... +==== pytest -== Source Files from PyPI +When upstream doesn’t use `+tox+`, +the tests need to be run directly +depending on upstream choice of a test runner. +A popular runner is `+pytest+`, which can be invoked using `+%pytest+`. -When packaging software which is available from PyPI, you can make use of the `+%pypi_source+` macro. This macro accepts from zero to three arguments and evaluates to an appropriate URL for the source file on PyPI. The arguments are: +Use positional arguments to specify the test directory. +See `+python3 -m pytest --help+` for how to select tests. +For example, if network-related tests are marked “network”, +you might use `+-m+` to deselect them: -1. The name of the PyPI project. Defaults to `+%srcname+` if defined, or to `+%pypi_name+` if defined, or to `+%name+` (the package name). -2. The version of the PyPI project. Defaults to `+%version+` (the package version) with any `~` characters removed (used for alpha/beta/dev versions in RPM version but not in Python package version). -3. The file extension to use. Defaults to `tar.gz`. +[source,spec] +---- +%pytest -m "not network" +---- -In most cases it is not necessary to specify any arguments. +The `+%pytest+` macro sets several environment variables +appropriate for `+%check+`: -== Macros +* Locations in the buildroot are added to `+$PATH+` and `+$PYTHONPATH+`. +* `+$PYTHONDONTWRITEBYTECODE+` is set to avoid +writing pytest-specific cache files to buildroot +* If unset, `+$CFLAGS+` and `+$LDFLAGS+` are set to match the build flags -The following macros are defined for you in all supported Fedora and EPEL releases: +=== Test dependencies -[%header,cols=3*] -|=== -|Macro -|Expanded path -|Notes +One part of the Python packaging ecosystem that is still not standardized +is specifying test dependencies (and development dependencies in general). -|`+%{__python}+` -|(Error) -|Don't use this macro without redefining it. Defining it changes the meaning of other "unversioned" Python macros such as `+%{python}+` or `+%{python_sitelib}+`. +A good, common way for upstreams to specify test dependencies +is using an <> like `+[test]+`, `+[testing]+` or `+[dev]+`. +In this case, upstream’s instructions to install test dependencies +might look like `+$ pip install -e.[test]+`. +Projects using `+tox+` usually specify test dependencies +in a `+tox+`-specific format: +a https://tox.readthedocs.io/en/latest/config.html#conf-requires[requires] +key in the configuration. -|`+%{__python3}+` -|`+/usr/bin/python3+` -|Python 3 interpreter. Redefining this macro changes all the `+%{python3...}+` macros. +These two forms are handled by +the <>. -|`+%{python3}+` -|`+%{__python3}+` -|Python 3 interpreter. Use this macro in spec files. +If upstream does not use either form, +list test dependencies as manual _BuildRequires_ in the `+spec+` file, +for example: -|`+%py_provides+` -|(Lua script) -|See <> for detailed explanation. +[source,spec] +---- +# Test dependencies: +BuildRequires: python3dist(pytest) +---- -|`+%{python3_sitelib}+` -|`+/usr/lib/python3.X/site-packages+` -|Where pure python3 modules are installed. +If you need to do this, +consider asking upstream to add a `+[test]+` extra. -|`+%{python3_sitearch}+` -|`+/usr/lib64/python3.X/site-packages+` on 64bit architectures (e.g. x86_64) and `+/usr/lib/python3.X/site-packages+` on 32bit. -|Where python3 extension modules (e.g. C compiled) are installed. +=== Linters -|`+%{py_byte_compile}+` -|(script) -|See xref:Python_Appendix.adoc#manual-bytecompilation[byte-compiling] section for usage. +In `+%check+`, packages *SHOULD NOT* run “linters”: +code style checkers, +test coverage checkers +and other tools that check code quality rather than functionality. -|`+%{python3_version}+` -|`+3.X+` -|Python 3 version. Useful when running programs with Python version in filename, such as `+nosetests-%{python3_version}+`. +Tools like `+black+`, `+pylint+`, `+flake8+`, or `+mypy+` +are often “opinionated” and their “opinions” change frequently enough +that they are nuisance in Fedora, +where the linter is not pinned to an exact version. +Furthermore, some of these tools take a long time +to adapt to new Python versions, +preventing early testing with Aplha and Beta releases of Python. +And they are just not needed: wrongly formatted code is not important enough +for the Fedora packager to bug the upstream about it. +Making such an issue break a package build is entirely unreasonable. -|`+%{python3_version_nodots}+` -|`+3X+` -|Python 3 version without dots. Useful when listing files explicitly in %files section, such as `+%{python3_sitearch}/foo/_speedups.cpython-%{python3_version_nodots}*.so+` +Linters _do_ make sense in upstream CI. But not in Fedora. -|`+%{python3_platform}+` -|`+linux-x86_64+` on x86_64 -|The platform name used in Python, useful for specifying `+$PYTHONPATH+`, such as `+PYTHONPATH=build/lib.%{python3_platform}-%{python3_version}+` +If a linter is used, disable it and remove the dependency on it. +If that is not easy, talk to upstream about making it easy +(for example with a configuration option or a separate `+tox+` environment). -|`+%{python3_ext_suffix}+` -|`+.cpython-3X-x86_64-linux-gnu.so+` on x86_64 -|The usual suffix of Python extension modules, useful when listing files. Note that extension modules can alternatively have a simple `+.so+` suffix as well, depending on how they are built. +For packages that contain such linters, use them at runtime or extend them, +you will usually need to run the linter in `+%check+`. +Run it to test functionality, not code quality of the packaged software. -|`+%py3_build+` -|`+%{__python3} setup.py build …+` -|See `+%py3_install+` for passing arguments to `+setup.py build+` or directly to `+setup.py+`. +== Source files from PyPI -|`+%py3_install+` -|`+%{__python3} setup.py install --skip-build …+` -|Various flags are passed to `+setup.py install+`, see `/usr/lib/rpm/macros.d/macros.python3` for details and similar macros. To add extra flags/arguments to `+setup.py install+`, separate them with `+--+`, for example: `+%py3_install -- --install-scripts %{_libexecdir}+`. To pass custom command line arguments directly to `+setup.py+`, define `+%py_setup_args+`. +Packages *MAY* use sources from PyPI. -|`+%__pytest+` -|`+/usr/bin/pytest+` -|The `pytest` command used in `+%pytest+`. Don't use this macro directly, but feel free to redefine it for usage in `+%pytest+` if desired. +However, packages *SHOULD NOT* use an archive that omits test suites, +licenses and/or documentation present in other source archives. -|`+%pytest+` -|`+PATH=… PYTHONPATH=… … %{__pytest}+` -|Various environment variables are set to ensure the packaged version is tested. Use this macro instead of direct `pytest` calls in `+%check+`. Pass additional argument as if passed to `pytest`, e.g. `+%pytest -m "not network"+` to deselect tests marked as `network`. +For example, as of this writing `+pip+` provides a +https://pypi.org/project/pip/#files[source tarball (“sdist”)] +which omits the relatively large `+tests+` and `+docs+` directories +present in https://github.com/pypa/pip[the source on GitHub]. +In this case, the tarball from GitHub should be used. +(See the xref:SourceURL#_git_tags[Git tags] section +of Fedora SourceURL guidelines.) -|`+%py3_check_import …+` -|`+PATH=… PYTHONPATH=… … %{__python3} -c 'import …'+` -|Various environment variables are set to ensure the packaged version is tested. Use this macro in `+%check+` to test public Python modules are importable if running upstream tests suite is not feasible. Pass module names as positional arguments separated by spaces or commas. +When using sources from PyPI, +you can use the <> +to generate the proper URL. -|`+%{py_dist_name}+` -|(Lua script) -|Given a standardized name (i.e. dist name, name on PyPI) of Python software, it will convert it to a canonical format. See <> for more information. +== Example spec file[[_example_python_spec_file]] -|`+%{py3_dist}+` -|(Lua script) -|Given a standardized name (i.e. dist name, name on PyPI) of Python software, it will convert it to a canonical format, and evaluates to `+python3dist(CANONICAL_NAME)+`, which is useful when listing dependencies. See <> for more information. +The following is a viable spec file +for a Python library called `+Pello+` +that follows packaging best practices. -|`+%{pypi_source}+` -|(Lua script) -|Evaluates to the appropriate URL for the package. See above for more information. +Note that the project name `+Pello+` <> +to the lowercase `+pello+`. +The example spec shows where each variant is typically used. -|`+%pycached ….py+` -|(Lua script) -|Given a Python file, lists the file and the files with its bytecode cache. See <> for more information. +The project has an <> `+color+`, +which enables colorized output when installed. +Since the required dependency is quite minimal +and color improves the user experience, +the extra is Recommended from the main package. -|`+%{py3_shebang_flags}+` -|`s` -|The default set of flags for Python shebangs. Redefine or undefine this to change the set. Used by `+%py3_shebang_fix+`. +[source,spec] +---- +Name: python-pello +Version: 1.0.2 +Release: 1%{?dist} +Summary: Example Python library -|`+%py3_shebang_fix …+` -|(Python script) -|Given paths for Python files or directories with them, it changes Python shebangs to `+#! %{__python3}+`, preserves any existing flags (if found) and adds flags defined in `+%{py3_shebang_flags}+` (if not already present). +License: MIT +URL: https://github.com/fedora-python/Pello +Source0: %{url}/archive/v%{version}/Pello-%{version}.tar.gz -|=== +BuildArch: noarch +BuildRequires: python3-devel -During `+%install+` or when listing `+%files+` you can use the `+%{python3_sitearch}+` and `+%{python3_sitelib}+` macros to specify where the installed modules are to be found. For instance: +%global _description %{expand: +A python module which provides a convenient example. +This description provides some details.} -.... -%files -# A pure python3 module -%{python3_sitelib}/foomodule/ -# A compiled python3 extension module -%{python3_sitearch}/barmodule/ -.... +%description %_description -Use of the macros has several benefits: +%package -n python3-pello +Summary: %{summary} +Recommends: python3-pello+color -* It ensures that the packages are installed correctly on multilib architectures. -* Using these macros instead of hardcoding the directory in the specfile ensures your spec remains compatible with the installed Python version even if the directory structure changes radically (for instance, if `+python3_sitelib+` moves into `+%{_datadir}+`). +%description -n python3-pello %_description -== Packages using Cython -A great amount of extension modules for Python -(Python modules written in a compiled language such as C or C++) -are written using the https://cython.org/[Cython] language and compiler. +%pyproject_extras_subpkg -n python3-pello color -Majority of such packages contains the generated C (or C++) sources in the source tarball. +%prep +%autosetup -p1 -n Pello-%{version} -Tightening the xref:what-can-be-packaged.adoc#_pregenerated_code[general Fedora policy], -packages MUST NOT use pre-generated Cython sources. -They MUST be deleted in `%prep` and regenerated during the build. -Any exception to this rule should be considered a xref:index.adoc#bootstrapping[bootstrapping]. +%generate_buildrequires +%pyproject_buildrequires -t -== Files to include -When packaging Python modules, several types of files are included: +%build +%pyproject_wheel -* +*.py+ source files because they are used when generating tracebacks. -* +*.pyc+ byte compiled files. -** Python will try to create them at runtime if they don't exist which leads to spurious SELinux AVC denials in the logs. -** If the system administrator invokes Python with -OO, they will be created with no docstrings. This can break some programs. -* +*.egg-info+ or +*.dist-info+ files or directories. If these are generated by the module's build scripts they must be included in the package because they might be needed by other applications and modules at runtime. -The source files MUST be included in the same package as the byte compiled versions. +%install +%pyproject_install -Packagers SHOULD NOT simply glob everything under the sitelib or sitearch directories. The following SHOULD NOT be used: +# Here, "pello" is the name of the importable module. +%pyproject_save_files pello -* `+%{python3_sitelib}/*+` -* `+%{python3_sitearch}/*+` -* `+%{python_sitelib}/*+` -* `+%{python_sitearch}/*+` -And packages MUST NOT include the top-level `+__pycache__+` directory (see below). +%check +%tox -== Byte compiling -Python will automatically try to byte compile files when it runs in order to speed up startup the next time it is run. These files are saved in files with the extension of .pyc (compiled Python). These files will be located inside a directory named `+__pycache__+`. +# Note that there is no %%files section for +# the unversioned python module, python-pello. -The .pyc files contain byte code that is portable across OSes. If you do not include them in your packages, Python will try (and generally fail) to create them when the user runs the program. If the system administrator runs the program, then the files will be successfully written, causing stray .pyc files which will not be removed when the package is removed. To prevent that the byte compiled files need to be compiled and included in the `+%files+` section. Normally, byte compilation is done for you by the `+brp-python-bytecompile+` script. This script runs after the `+%install+` section of the spec file has been processed and byte compiles any .py files that it finds in `+%{python3_sitelib}+` or `+%{python3_sitearch}+` (this recompilation puts the proper filesystem paths into the modules otherwise tracebacks would include the `+%{buildroot}+` in them). +# For python3-pello, %%{pyproject_files} handles code files, but +# executables, documentation and licenses must be listed in the spec file: -You must include the .pyc files in your package. If the build process creates a `+__pycache__+` directory in a subdirectory of `+%{python3_sitearch}+` or `+%{python3_sitelib}+`, you must also include all items in the `+__pycache__+` directory. You MUST NOT include the directories `+%{python3_sitearch}/__pycache__+` or `+%{python3_sitelib}/__pycache__+` because they are already owned by the python3-libs package. +%files -n python3-pello -f %{pyproject_files} +%doc README.md +%license LICENSE.txt +%{_bindir}/pello_greeting -All that you need to do is include the files in the `+%files+` section (replacing `+%{python3_sitelib}+` with the appropriate macro for your package): -.... -%files -%{python3_sitelib}/foo/ -.... +%changelog +---- -or, if the Python code installs directly into `+%{python3_sitelib}+`, use the `+%pycached+` macro to include the bytecode cache files: +== Empty spec file -.... -%files -%pycached %{python3_sitelib}/foo.py -.... +The following is an unfinished spec file template to copy, paste and edit. -That evaluates roughly to +[source,spec] +---- +Name: python-... +Version: ... +Release: 0%{?dist} +Summary: ... + +License: ... +URL: https://... +Source0: %{url}/archive/v%{version}/...-%{version}.tar.gz / %{pypi_source ...} + +BuildArch: noarch / BuildRequires: gcc +BuildRequires: python3-devel + +%global _description %{expand: +...} + +%description %_description + +%package -n python3-... +Summary: %{summary} + +%description -n python3-... %_description + + +%prep +%autosetup -p1 -n ...-%{version} + + +%generate_buildrequires +%pyproject_buildrequires -rx... / -t + + +%build +%pyproject_wheel -.... -%files -%{python3_sitelib}/foo.py -%{python3_sitelib}/__pycache__/foo.cpython-%{python3_version_nodots}{,.opt-?}.pyc -.... -NOTE: The `+%pycached+` macro only supports Python 3.5+, -so for older Python versions -(such as 3.4 in EPEL 6 or 7), -you need to list the files manually. +%install +%pyproject_install +%pyproject_save_files ... -NOTE: In case you need to use other macros with the `+%pycached+` macro, -such as `+%exclude+` or `+%ghost+`, -pass the other macro as part of the argument to `+%pycached+`. -For example: -`+%pycached %exclude /path/to/foo.py+` -Using the macros in wrong order would only apply `+%exclude+` to the first entry that `+%pycached+` generates. -=== Manual byte compilation +%check +%tox / %pytest -For more details on the internals of byte compilation, please see xref:Python_Appendix.adoc#manual-bytecompilation[the appendix]. -== Example Python spec file +%files -n python3-... -f %{pyproject_files} +%doc README.* +%license LICENSE* +%{_bindir}/... -The following is a very simple spec file for a Python module. -.python-example.spec -[source] +%changelog ---- -include::{examplesdir}/python-example.spec[] + +== Macro Reference[[_macros]] + +This section documents macros that are available +to help with Python packaging. +The expansions in parentheses are provided only as reference/examples. + +See the <> section above for: + +* `+%{python3}+` (`+/usr/bin/python3+`) +* `+%{python3_version}+` (e.g. `+3.9+`) +* `+%{python3_version_nodots}+` (e.g. `+39+`) +* `+%{python3_sitelib}+` (e.g. `+/usr/lib/python3.9/site-packages+`) +* `+%{python3_sitearch}+` (e.g. `+/usr/lib64/python3.9/site-packages+`) + +=== Shebang macros + +[#py3_shebang_flags] +* `+%{py3_shebang_flags}+` (`+s+`) ++ +Flags for `+%{python3}+` to use in shebangs. +Redefine this macro to use a different set of flags. +Undefine it to use no flags. See <> for details. + +[#py3_shebang_fix] +* `+%py3_shebang_fix PATHS+` (`+pathfix.py ... PATHS+`) ++ +A macro to fix shebangs in specified `+PATHS+`. +Only shebnags that already have `+python+` in them are changed. +If a directory is given, all `+.py+` files in it are fixed, recursively. +(So, if you need to fix shebangs in files not named `+*.py+`, +you need to list each file separately or use a Shell glob, +such as `+%{buildroot}%{_libexecdir}/mytool/*+`.) +Existing flags are preserved and `+%{py3_shebang_flags}+` are added +(if not there already). ++ +For example, +`+#! /usr/bin/env python+` will be changed to `+#! /usr/bin/python3 -s+` +and `+#! /usr/bin/python -u+` will be changed to `+#! /usr/bin/python3 -su+`. ++ +This macro is called automatically +by `+%pyproject_install+` on `+%{buildroot}%{_bindir}/*+`. + +=== Convenience macros + +[#pypi_source] +* `+%{pypi_source PROJECTNAME [VERSION [EXT]]}+` +(e.g. `+https://.../Django-3.0.5.tar.gz+`) ++ +Evaluates to the appropriate URL for source archive hosted on PyPI. +Accepts the project name and up to two optional arguments: ++ +-- +** The version of the PyPI project. +Defaults to `+%version+` (the package version) with any `+~+` removed. +** The file extension to use. Defaults to `+tar.gz+`. +-- ++ +In most cases it is not necessary to specify those two arguments. ++ +For backward compatibility, the first argument is technically optional +as well, but omitting it is deprecated. (It defaults to `+%srcname+` if +defined, or to `+%pypi_name+` if defined, or to `+%name+`.) + +[#python3_platform] +* `+%{python3_platform}+` (e.g. `+linux-x86_64+`) ++ +The platform name. +Used in some Python build systems. +This corresponds to +https://docs.python.org/3/library/sysconfig.html#sysconfig.get_platform[`+sysconfig.get_platform()+`]. + +[#python3_ext_suffix] +* `+%{python3_ext_suffix}+` (e.g. `+.cpython-39-x86_64-linux-gnu.so+`) ++ +Filename extension for Python extension modules. +This corresponds to the `+EXT_SUFFIX+` +https://docs.python.org/3/library/sysconfig.html[sysconfig] variable. + +[#python3_platform_triplet] +* `+%{python3_platform_triplet}+` (e.g. `+x86_64-linux-gnu+`) ++ +A string identifying the architecture/platform. +This corresponds to the `+MULTIARCH+` +https://docs.python.org/3/library/sysconfig.html[sysconfig] variable. + +=== Build macros + +The “pyproject macros” are most useful +for packaging Python projects that use the `+pyproject.toml+` file +defined in https://www.python.org/dev/peps/pep-0518/[PEP 518] +and https://www.python.org/dev/peps/pep-0517/[PEP 517], +which specifies the package’s build dependencies +(including the build system, such as `+setuptools+`, `+flit+` or `+poetry+`). + +If `+pyproject.toml+` is not found, +the macros automatically fall backs to using `+setuptools+` +with configuration in `+setup.cfg+`/`+setup.py+`. + +A full tutorial and discussion for the macros is available in the macros’ +https://src.fedoraproject.org/rpms/pyproject-rpm-macros/[README]. + +[#pyproject_buildrequires] +* `+%pyproject_buildrequires+` ++ +Generate BuildRequires for the package. +Used in the `+%generate_buildrequires+` section of the `+spec+` file. +The macro has these options: ++ +** `+-r+`: Include run-time requirements (commonly needed for `+%check+`). +** `+-x EXTRA+`: Include dependencies given by the given <>. +Implies `+-r+`. +** `+-t+`: Include dependencies for the default _tox_ environment. +Implies `+-r+`. +** `+-e ENV+`: Include dependencies for the given _tox_ environment, +and save the `+ENV+` name as `+%{toxenv}+`. +Implies `+-r+`. +Multiple comma separated values can be given, for example: ++ +[source,spec] +---- +%pyproject_buildrequires -e %{toxenv}-unit,%{toxenv}-integration +---- + +[#pyproject_wheel] +* `+%pyproject_wheel+` ++ +Build the package. +Commonly, this is the only macro needed in the `+%build+` section. ++ +This macro needs BuildRequires generated by `+%pyproject_buildrequires+`. + +[#pyproject_install] +* `+%pyproject_install+` ++ +Install the package built by `+%pyproject_wheel+`. +Calls `+%py3_shebang_fix %{_buildroot}%{_bindir}/*+`. ++ +This macro needs BuildRequires generated by `+%pyproject_buildrequires+`. + +[#pyproject_save_files] +* `+%pyproject_save_files MODNAME …+` ++ +Generate a list of files corresponding to the given importable module(s) +and save it as `+%{pyproject_files}+`. ++ +Note that README and license files are not included. +Also, while the macro allows including executable and other files +(using the `+auto` flag), +this feature *MUST NOT* be used in Fedora. ++ +The `+MODNAME+` may be a glob pattern, +which should be specific to your package. +To prevent Shell from expanding the globs, put them in `+''+`, +e.g. `+%pyproject_save_files '*pytest'+`. +As mentioned in the <> section, +expressions like `+%pyproject_save_files '*'+` are not acceptable. + +[#pyproject_files] +* `+%{pyproject_files}+` ++ +Path of the file written by `+%pyproject_save_files+`, to be used as: ++ +[source,spec] +---- +%files -n python3-DISTNAME -f %{pyproject_files} ---- -== Reviewer checklist +=== Test macros -The following briefly summarizes the guidelines for reviewers to go over: +[#tox] +* `+%tox+` ++ +Run tests using `+tox+`. ++ +Different environments may be specified with `+-e+`, for example: ++ +[source,spec] +---- +%check +%tox %{?with_integration_tests:-e %{toxenv},%{toxenv}-integration} +---- ++ +Flags for the `+tox+` command can be specified after `+--+`: ++ +[source,spec] +---- + %tox -- --parallel 0 +---- ++ +Additional arguments for the test runner may be specified after another `+--+`: ++ +[source,spec] +---- + %tox -- --parallel 0 -- --verbose tests/* +---- + +[#toxenv] +* `+%{toxenv}+` ++ +The _tox_ environment(s) used by the `+%tox+` macro. +Multiple environments are separated by commas. +Can be overridden manually or with `+%pyproject_buildrequires -t ENV1,ENV2+`. + +[#default_toxenv] +* `+%{default_toxenv}+` (e.g. `+py39+`) ++ +The system-wide default value of `+%{toxenv}+`. + +[#pytest] +* `+%pytest+` ++ +Run `+%__pytest+` with environment variables appropriate for tests in `%check`. +See <> for details. + +[#__pytest] +* `+%__pytest+` (`+/usr/bin/pytest+`) ++ +The command that `+%pytest+` uses. May be redefined. + +=== Extras macros + +[#pyproject_extras_subpkg] +* `+%pyproject_extras_subpkg+` ++ +Generates a simple subpackage for a Python extra. +See <> for more information. ++ +This macro needs to be used with `+%pyproject_install+` +(use `+%python_extras_subpkg+` in other cases). ++ +Required arguments: ++ +-- +** `+-n+`: name of the “base” package (e.g. `+python3-requests+`) +** Positional arguments: the extra name(s). +Multiple metapackages are generated when multiple names are provided. +-- ++ +The macro also takes `+-i+`/`+-f+`/`+-F+` arguments +for `+%python_extras_subpkg+` below, +but if they are not given, a filelist written by `+%pyproject_install+` is used. ++ +This macro generates all the subpackage definition sections +(`+%package+` including the `+Summary+` and `+Requires+` on the base package, +`+%description+` and, by default, `+%files+`). +Hence, it cannot be extended with custom _Provides_/_Obsoletes_/_Requires_/etc. +This macro is designed to fit only the most common uses. +For more complicated uses, construct the subpackage manually +as shown in the <> section. ++ +The `+%files+` section is last. +It can be continued to add files that only make sense with the extra +and the base package does not fail without them. +For example, the following macro will +package the extra `+cli+` for the project `+a-cool-tool+` +and include an `+a-cool-tool+` command: ++ +[source,spec] +---- +%pyproject_extras_subpkg -n a-cool-tool cli +%{_bindir}/a-cool-tool +---- ++ +Due to technical limitations, +the macro never generates requirements on the arched +`+BASE_PACKAGE%{?_isa} = %{?epoch:%{epoch}:}%{version}-%{release}+`. +It only adds +`+Requires: BASE_PACKAGE = %{?epoch:%{epoch}:}%{version}-%{release})+` +because a macro cannot reliably detect if the subpackage is arched or not. +So far, this has not been a problem in practice. + +[#python_extras_subpkg] +* `+%python_extras_subpkg+` ++ +Generates a simple subpackage for a Python extra. +See <> for more information. +Takes these arguments: ++ +-- +** `+-n+`: name of the “base” package (e.g. `+python3-requests+`) +** `+-i+`: the `+%files %ghost+` path (glob) to the `+.dist-info+` directory +** Positional arguments specify the extra name(s) +— multiple metapackages are generated when multiple names are provided. +** `+-f+`: Relative path to the filelist for this metapackage +(which should contain the `+%files %ghost+` path (glob) +to the the metadata directory). +Conflicts with `+-i+` and `+-F+`. +** `+-F+`: Skip the %files section entirely +(if the packager wants to construct it manually). +Conflicts with `+-i+` and `+-f+`. +-- ++ +As with `+%pyproject_extras_subpkg+`: ++ +** This macro generates all the subpackage definition sections, +with only `+%files+` being customizable. +For more complicated uses, construct the subpackage manually +as shown in the <> section. +** It never generates requirements on the arched +`+BASE_PACKAGE%{?_isa} = %{?epoch:%{epoch}:}%{version}-%{release}+`. + +=== Manual generation + +The following macros are available for cases where +automatic generation is turned off. +They can also be useful for handling files in non-standard locations +where the generators don’t look. + +[#pycached] +* `+%pycached MODNAME.py+` ++ +Given a Python file, lists the file and the files with its bytecode +cache. See _Source files and bytecode cache_ for more information. + +[#py_provides] +* `+%py_provides python3-MODNAME+` ++ +Generates `+Provides+` for `+python3-MODNAME+`, `+python3.X-MODNAME+` +and `+python-MODNAME+`. +See <> for more details. + +[#py_byte_compile] +* `+%py_byte_compile INTERPRETER PATH+` ++ +Byte-compile a Python file into a `+__pycache__/*.pyc+`. ++ +If the `+PATH+` argument is a directory, +the macro will recursively byte compile all `+*.py+` files in the directory. +(So, if you need to compile files not named `+*.py+`, +you need to use the macro on each file separately.) ++ +The `+INTERPRETER+` determines the compiled file name’s suffix +and the magic number embedded in the file. +These muct match the interpreter that will import the file. +Usually, the `+INTERPRETER+` should be set to `+%{python3}+`. +If you are compiling for a non-default interpreter, +use that interpreter instead and add a `+BuildRequires+` line for it. + +[#py_dist_name] +* `+%{py_dist_name PROJECTNAME}+` ++ +Given a _project name_ (e.g. `+PyYAML+`) it will convert it +to the canonical format (e.g. `+pyyaml+`). +See <> for more information. + +[#py3_dist] +* `+%{py3_dist PROJECTNAME …}+` ++ +Given one or more _project names_, +it will convert them to the canonical format +and evaluate to `+python3dist(DISTNAME)+`, +which is useful when listing dependencies. +See <> for more information. + +=== System Settings + +The following macros can be redefined for special use cases. + +[#__python] +* `+%{__python}+` (`+/usr/bin/python+`) ++ +Defining this macro sets the meaning of all “unversioned” Python macros +such as `+%{python}+` or `+%{python_sitelib}+`. +Don’t use these macros without redefining `+%{__python}+`. + +[#__python3] +* `+%{__python3}+` (`+/usr/bin/python3+`) ++ +The python 3 interpreter. +Redefining this macro changes all the `+%{python3...}+` macros, +e.g. `+%{python3}+` or `+%{python3_sitelib}+`. + +[#python3_pkgversion] +* `+%{python3_pkgversion}+` (`+3+`) ++ +Distro-wide Python version, i.e. the `+3+` in `+python3+`. +Projects that build on top of Fedora might define it to e.g. `+3.9+` +to try allowing multiple Python stacks installable in parallel. +Packages in Fedora *MAY* use it +(e.g. in package names: `+python%{python3_pkgversion}-requests+`), +but *MUST NOT* redefine it. + +=== Comparing Python versions + +When comparing Python versions +(e.g. to ask: is `+%{python3_version}+` greater than 3.8?), +using naïve `+%if %{python3_version} > 3.8+` +or `+%if "%{python3_version}" > "3.8"+` is not possible, +because the comparison is performed alphabetically on strings. +Hence it is true that `+"3.10" < "3.8"+` (which is not desired). + +Starting with RPM 4.16 (Fedora 33), +it is possible to explicitly compare version literals by using the `+v+` prefix, +similar to the Python string prefixes: + +[source,spec] +---- +%if v"%{python3_version}" > v"3.8" +... +%endif +---- + +As a workaround for compatibility with older RPM releases, +`+%{python3_version_nodots}+` can be compared as an integers: + +[source,spec] +---- +%if %{python3_version_nodots} > 38 +... +%endif +---- -* *Must*: Python modules must be built from source. They cannot simply drop an egg or whl from upstream into the proper directory. (See xref:what-can-be-packaged.adoc#prebuilt-binaries-or-libraries[prebuilt binaries Guidelines] for details). -* *Must*: Python modules must not download any dependencies during the build process. -* *Must*: When building a compat package, it must install using easy_install -m so it won't conflict with the main package. -* *Must*: When building multiple versions (for a compat package) one of the packages must contain a default version that is usable via "import MODULE" with no prior setup. -* *Should*: Additional `+python3-...+` provides should be accomplished via a `+%py_provides+` call. -* *Should*: A package which is used by another package via an egg interface should provide egg info. +This will work with Python 3.10 (310 > 39), +but eventually break with Python 4.0 (40 < 310). + +=== Disabling automation + +The following macros can turn off Python-specific automation. + +Consider contacting the Python SIG if you need to do this. + +[#python_disable_dependency_generator] +* `+%{?python_disable_dependency_generator}+` ++ +Disables the automatic dependency generator. +See <> for details. + +[#__pythonname_provides] +* `+%undefine __pythonname_provides+` ++ +Disables automatic generation of unversioned/versioned provides for package names, +e.g. `+python-FOO+` and `+python3.9-FOO+` for `+python3-foo+`. +See <> for more details. + +[#_python_no_extras_requires] +* `+%global _python_no_extras_requires 1+` ++ +If defined, <> will not be generated. + +=== Deprecated Macros + +The following macros are deprecated. +See the xref:Python[201x-era Python Packaging guidelines] +for how some of them were used. + +[#py3_build] +* `+%py3_build+` +[#py3_build_wheel] +* `+%py3_build_wheel+` +[#py3_build_egg] +* `+%py3_build_egg+` +[#py3_install] +* `+%py3_install+` +[#py3_install_wheel] +* `+%py3_install_wheel+` +[#py3_install_egg] +* `+%py3_install_egg+` +[#py3dir] +* `+%py3dir+` +[#py3_other_build] +* `+%py3_other_build+` +[#py3_other_install] +* `+%py3_other_install+` +[#python_provide] +* `+%python_provide+` +. diff --git a/guidelines/modules/ROOT/pages/Python_201x.adoc b/guidelines/modules/ROOT/pages/Python_201x.adoc new file mode 100644 index 0000000..4969164 --- /dev/null +++ b/guidelines/modules/ROOT/pages/Python_201x.adoc @@ -0,0 +1,533 @@ += Python Packaging Guidelines (201x-era) +:last-reviewed: 2020-08-05 +:toc: + +[IMPORTANT] +==== +These guidelines are being replaced by xref:Python.adoc[a newer version]. + +You are free to use either version of the Python Packaging Guidelines. +==== + +== Python Version Support + +In Fedora we have multiple Python runtimes, +one for each supported major Python release. +At this point that's one for python3.x and one for python2.7. +However the Python 2 stack will be removed from Fedora and is xref:deprecating-packages.adoc[deprecated]. +Upstream support for the python2 interpreter officially ends in 2020. +If a piece of software supports python3, +it MUST be packaged for python3. +Software using python2 MUST NOT be newly packaged into Fedora without FESCo exception. + +For guidelines on maintaining already existing python2 packages, see the xref:Python_Appendix.adoc[appendix]. + +== Multiple Python Runtimes + +On Fedora `/usr/bin/python` is, if it is installed, a symbolic link to `/usr/bin/python3`. +It was a symbolic link to `/usr/bin/python2` on previous releases. + +Packages in Fedora MUST NOT use `/usr/bin/python`. Instead packages for Python 3 MUST use `/usr/bin/python3` (even if upstream supports both Python 2 and 3). As a result of that `/usr/bin/python` (as well as `/usr/bin/env python` and similar) MUST NOT be used in shebang lines or as a dependency of a package. All uses of unversioned python executables in shebang lines will fail the build. +These shebangs MUST be fixed (for example by using the `+%py3_shebang_fix+` macro in the spec file). +If it is necessary to disable the checks, please see the information in xref:index#_shebang_lines[Shebang lines]. + +All Python runtimes have a virtual provide for `+python(abi) = $MAJOR.$MINOR+`. For example, the Python 3.7 runtime package has: + +.... +$ rpm -q --provides python3 | grep abi +python(abi) = 3.7 +.... + +Python modules using these runtimes should have a corresponding "Requires" line on the Python runtime that they are used with. This is done automatically for files below `+/usr/lib[^/]*/python${PYVER}+` + +Mirroring the policy for regular packages, the Python-version-specific subpackages of your package MUST NOT be removed in a release branch of Fedora. + +== Naming + +The source package for a Python library MUST be named with the `python-` prefix. A built package however must include the Python major version in the name, using the `python3-` prefix. This is accomplished by adding a subpackage. See example below. + +This rule does not apply to applications. + +The character `+` in names of built packages (i.e. non-SRPM) +that include `.dist-info` or `.egg-info` directories is reserved for <> and MUST NOT be used for any other purpose. +The `+` character triggers the automatic dependency generator for extras. +Replace any `+` signs in the upstream name with `-`, +or omit them when at the beginning of the name. +As an exception, `+` characters are permitted at the _end_ of the name. + +== Dependencies + +Packages building for Python 3 will need `BuildRequires: python3-devel`. +Most of them will also need `BuildRequires: python3-setuptools`. +When in doubt, inspect the `setup.py` file for `setuptools` import. + +Packages MUST NOT have dependencies (either build-time or runtime) on packages named with the unversioned `python-` prefix. Dependencies on Python packages instead MUST use names beginning with `python3-`. + +=== Automatically generated dependencies + +Packages MAY use the automatic Python dependency generator. This generator uses upstream egg/dist metadata (such as https://python-packaging.readthedocs.io/en/latest/dependencies.html[setuptool's install_requires]) to determine what the package should depend on. The generator parses the installed metadata from `+/usr/lib(64)?/pythonX.Y/site-packages/[^/]+++.(egg|dist)-info/requires.txt+`, so it will not work with software that uses plain https://docs.python.org/3/distutils/[distutils]. + +This generates run time requires in the form of `+pythonX.Ydist(foo)+`. If the generated dependencies are not accurate, additional ones can still be added manually. To remove some, a packager MAY modify upstream-provided metadata (usually specified in the `setup.py` file) in the `+%prep+` section of the specfile or fall back to xref:AutoProvidesAndRequiresFiltering.adoc[filtering] those dependencies. + +The packager MUST inspect the generated requires for correctness. All dependencies MUST be resolvable within the targeted Fedora version. + +As an example, the upstream notebook package has (as of version 5.6.0): + +[source,python] +---- +install_requires = [ +    'jinja2', +    'tornado>=4', +    'pyzmq>=17', +    'ipython_genutils', +    'traitlets>=4.2.1', +    'jupyter_core>=4.4.0', +    'jupyter_client>=5.2.0', +    'nbformat', +    'nbconvert', +    'ipykernel', +    'Send2Trash', +    'terminado>=0.8.1', +    'prometheus_client' +], +---- + +And the resulting dependencies: + +.... +python3.7dist(ipykernel) +python3.7dist(ipython-genutils) +python3.7dist(jinja2) +python3.7dist(jupyter-client) >= 5.2 +python3.7dist(jupyter-core) >= 4.4 +python3.7dist(nbconvert) +python3.7dist(nbformat) +python3.7dist(prometheus-client) +python3.7dist(pyzmq) >= 17 +python3.7dist(send2trash) +python3.7dist(terminado) >= 0.8.1 +python3.7dist(tornado) >= 4 +python3.7dist(traitlets) >= 4.2.1 +.... + +Note that any `+.0+` suffixes are removed from version numbers to match the behavior of Python tools. +(https://www.python.org/dev/peps/pep-0440/#final-releases[PEP 440] specifies that `+X.Y+` and `+X.Y.0+` are treated as equal.) + +This generator is enabled by default in Fedora. If a packager wishes to explicitly opt out of the generator because the upstream metadata are not applicable, a packager SHOULD opt out explicitly by adding: + +`+%{?python_disable_dependency_generator}+` + +Although this statement can be used anywhere in the spec, we recommend putting it just before the main package's `+%description+` declaration. + +=== Python Extras + +https://www.python.org/dev/peps/pep-0508/#extras[Python extras] are a way for Python projects to declare that extra dependencies are required for additional functionality. + +For example, `requests` has several standard dependencies (e.g. `urllib3`). +But it also declares an _extra_ named `+requests[security]+`, +which lists additional dependencies (e.g. `cryptography`). +Unlike RPM subpackages, +extras can only specify additional dependencies, +not additional files. +The main package will work if the optional dependency is not installed, +but it might have limited functionality. + +Python tools treat extras as virtual packages. +For example, +if a user runs `+pip install requests[security]+`, +or installs a project that depends on `+requests[security]+`, +both `requests` and `cryptography` will be installed. + +Starting with Fedora 33, +extras are usually provided by packages with no files. +Instead of square brackets, +Fedora package names conventionally use the `+` character to separate the package name and the _extra_ name, +e.g. the package would be named `python3-requests+security`. +The plus sign is valid in RPM package names, +but not in Python canonical project names nor in extras identifiers. + +Python packages SHOULD have `Provides` for all extras the upstream project specifies, +except those that are not useful for other packages +(for example build/development requirements, +commonly named `dev`, `doc` or `test`). + +A package that provides a Python extra MUST provide `+python3dist(…[…])+` and `+python3.Xdist(…[…])+`, +for example, `+python3.9dist(requests[security])+`. +These requirements SHOULD be generated using the automatic dependency generator. + +A package that provides a Python extra MUST require the extra's main package with exact NEVR. + +A subpackage that primarily provides one Python extra SHOULD be named by appending `+` and the extra name to the main package name. +For example, `++python3-requests+security++`. + +The most straightforward way to provide an extra is with a dedicated subpackage containing no files (a "metapackage"). +This case can be automated with the `+%python_extras_subpkg+` macro. + +Alternative approach: +when some extra is always useful in a distro, +it can be provided by the main package; +when several extras are related, +they may be provided by a single subpackage. +However, +having one dedicated subpackage per extra +allows you to use the automatic dependency generator to ensure that the extras' requirements will stay in sync with upstream. +If you create a dedicated subpackage and want it to be always/usually installed, you MAY Require/Recommend/Suggest it from the main package. + +The dependency generator for extras activates if the following holds: + +- The package must contain the `.egg-info`/`.dist-info` directory, usually as `+%ghost+`. +- The package name must end with `+EXTRA` (where `EXTRA` is the extra name). + +As an example, +the extra subpackage for `+requests[security]+` can be specified using the `+%python_extras_subpkg+` convenience macro as follows. +The macro takes the main package name and name(s) of the extra(s) as well as path to the `.egg-info` or `.dist-info` directory: + +---- +%{?python_extras_subpkg:%python_extras_subpkg -n python3-requests -i %{python3_sitelib}/*.egg-info security} +---- + +For this case, +the extras dependency generator will read upstream metadata from the `.egg-info` directory. +If it finds that the `security` extra has a dependency on `cryptography`, +it will generate `+Requires: python3.Xdist(cryptography)+`, +`+Provides: python3dist(requests[security])+` +(and the corresponding `+python3.Xdist+` variant). + +If you need additional features that the `+%python_extras_subpkg+` macro does not cover, +you will need to write the subpackage sections manually. +Such features can be, for example: + +- Obsoleting/providing other names (e.g. obsoleted extras packages) +- Manual strong or weak dependencies on other (possibly non-Python) packages +- Including files excluded from the main package (if such files only make sense with the extra and the base package does not fail without them) + +As an example of what you need to write in these cases, +the `+%python_extras_subpkg+` macro invocation above expands to the following: + +---- +%package -n python3-requests+security +Summary: Metapackage for python3-requests: security extras +Requires: python3-requests = %{?epoch:%{epoch}:}%{version}-%{release} +%description -n python3-requests+security +This is a metapackage bringing in security extras requires for python3-requests. +It contains no code, just makes sure the dependencies are installed. + +%files -n python3-requests+security +%ghost %{python3_sitelib}/*.egg-info +---- + +Note that the dependency generator does not add a dependency on the main package +(the `+Requires: python3-setuptools_scm = ...+` above). +If you are not using the `+%python_extras_subpkg+` macro, +you need to add it manually. + +NOTE: The `+%python_extras_subpkg+` can take multiple extras names to generate multiple packages. +For more options, see the https://fedoraproject.org/wiki/Changes/PythonExtras[change proposal which introduced this]. + +== Provides + +For any module `foo` intended to be used in Python 3 with `import foo`, +the package that includes it *should* provide `python3-foo`. +This is of course always the case if the subpackage is named `python3-foo` +(as in the examples below). +If the subpackage has some other name, +then `Provides: python3-foo` should be added explicitly +(via `+%py_provides python3-foo+`, see below). + +=== The %py_provides macro + +All packages that provide `+python3-...+` (for any `+...+`) SHOULD also provide `+python-...+` and `+python3.X-...+`. +Starting from Fedora 33, +most of the Python packages named `+python3-...+` will provide such names automatically via the dependency generator in `/usr/lib/rpm/fileattrs/pythonname.attr`. + +Any manually added virtual provides of `+python3-...+` SHOULD be done via the `+%py_provides+` macro. + +Instead of: + +---- +Provides: python3-pkg_resources = %{version}-%{release} +---- + +Do: + +---- +%py_provides python3-pkg_resources +---- + +Optionally, +supply a custom `epoch:version-release` as a second argument to `+%py_provides+`. + +On releases older than Fedora 33, +or (for technical limitations) for packages without files, +it is necessary to use `+%py_provides+` even for package names: + +---- +%package -n python3-%{srcname} +Summary: %{summary} +%py_provides python3-%{srcname} +---- + +Packagers SHOULD try to remove explicit `+%py_provides+` calls for package names, +but MAY preserve them if they aim for compatibility with older releases or packages without files. + +NOTE: Historically, there was `+%python_provide+` macro with similar but different semantics. +It still works for compatibility reasons but it is deprecated and SHOULD NOT be used +and packagers SHOULD replace is with appropriate `+%py_provides+` call. + +== Automatic Provides with a standardized name + +When building a Python package, RPM looks for `+.dist-info+` and `+.egg-info+` files or directories in the `+%files+` sections of all packages. If one or more are found, RPM parses them to find the *standardized name* (i.e. dist name, name on PyPI) of the packaged software, and then automatically creates two `+Provides:+` tags in the following format: + +.... +Provides: python3.Ydist(CANONICAL_STANDARDIZED_NAME) +Provides: python3dist(CANONICAL_STANDARDIZED_NAME) +.... + +The `+3.Y+` is the Python version used (usually 3.6 and higher), and between the parentheses is the name of the software in a *canonical format* used by Python tools and services such as setuptools, pip and PyPI. The canonical name is obtained by switching the standardized name to lower case and converting all runs of non-alphanumeric characters to single “-” characters. Example: “The $$$ Tree” becomes “the-tree”. + +=== Requires and BuildRequires with standardized names + +These Provides tags can be used to list Requires and BuildRequires of a package using the _standardized names_ (i.e. dist name, name on PyPI) of Python modules. To make it easier, you can use the `+%{py3_dist}+` macro that accept one or more parameters: the _standardized name(s)_ of the desired Python software. It will convert the name(s) to the _canonical format_ and create the proper `+python3dist(...)+` tag(s). + +In addition, you can use the `+%{py_dist_name}+` macro that simply transforms any _standardized name_ to the _canonical format_. + +For example: + +.... +BuildRequires: %{py3_dist PyMySQL} >= 0.7.5 +# => BuildRequires: python3dist(pymysql) >= 0.7.5 + +Requires: %{py3_dist virtualenv pyPEG2} +# => Requires: python3dist(virtualenv) python3dist(pypeg2) + +%{py_dist_name 0-._.-._.-._.-._.-._.-._.-0} +# => 0-0 +.... + +== Source Files from PyPI + +When packaging software which is available from PyPI, you can make use of the `+%pypi_source+` macro. This macro accepts from zero to three arguments and evaluates to an appropriate URL for the source file on PyPI. The arguments are: + +1. The name of the PyPI project. Defaults to `+%srcname+` if defined, or to `+%pypi_name+` if defined, or to `+%name+` (the package name). +2. The version of the PyPI project. Defaults to `+%version+` (the package version) with any `~` characters removed (used for alpha/beta/dev versions in RPM version but not in Python package version). +3. The file extension to use. Defaults to `tar.gz`. + +In most cases it is not necessary to specify any arguments. + +== Macros + +The following macros are defined for you in all supported Fedora and EPEL releases: + +[%header,cols=3*] +|=== +|Macro +|Expanded path +|Notes + +|`+%{__python}+` +|(Error) +|Don't use this macro without redefining it. Defining it changes the meaning of other "unversioned" Python macros such as `+%{python}+` or `+%{python_sitelib}+`. + + +|`+%{__python3}+` +|`+/usr/bin/python3+` +|Python 3 interpreter. Redefining this macro changes all the `+%{python3...}+` macros. + +|`+%{python3}+` +|`+%{__python3}+` +|Python 3 interpreter. Use this macro in spec files. + +|`+%py_provides+` +|(Lua script) +|See <> for detailed explanation. + +|`+%{python3_sitelib}+` +|`+/usr/lib/python3.X/site-packages+` +|Where pure python3 modules are installed. + +|`+%{python3_sitearch}+` +|`+/usr/lib64/python3.X/site-packages+` on 64bit architectures (e.g. x86_64) and `+/usr/lib/python3.X/site-packages+` on 32bit. +|Where python3 extension modules (e.g. C compiled) are installed. + +|`+%{py_byte_compile}+` +|(script) +|See xref:Python_Appendix.adoc#manual-bytecompilation[byte-compiling] section for usage. + +|`+%{python3_version}+` +|`+3.X+` +|Python 3 version. Useful when running programs with Python version in filename, such as `+nosetests-%{python3_version}+`. + +|`+%{python3_version_nodots}+` +|`+3X+` +|Python 3 version without dots. Useful when listing files explicitly in %files section, such as `+%{python3_sitearch}/foo/_speedups.cpython-%{python3_version_nodots}*.so+` + +|`+%{python3_platform}+` +|`+linux-x86_64+` on x86_64 +|The platform name used in Python, useful for specifying `+$PYTHONPATH+`, such as `+PYTHONPATH=build/lib.%{python3_platform}-%{python3_version}+` + +|`+%{python3_ext_suffix}+` +|`+.cpython-3X-x86_64-linux-gnu.so+` on x86_64 +|The usual suffix of Python extension modules, useful when listing files. Note that extension modules can alternatively have a simple `+.so+` suffix as well, depending on how they are built. + +|`+%py3_build+` +|`+%{__python3} setup.py build …+` +|See `+%py3_install+` for passing arguments to `+setup.py build+` or directly to `+setup.py+`. + +|`+%py3_install+` +|`+%{__python3} setup.py install --skip-build …+` +|Various flags are passed to `+setup.py install+`, see `/usr/lib/rpm/macros.d/macros.python3` for details and similar macros. To add extra flags/arguments to `+setup.py install+`, separate them with `+--+`, for example: `+%py3_install -- --install-scripts %{_libexecdir}+`. To pass custom command line arguments directly to `+setup.py+`, define `+%py_setup_args+`. + +|`+%__pytest+` +|`+/usr/bin/pytest+` +|The `pytest` command used in `+%pytest+`. Don't use this macro directly, but feel free to redefine it for usage in `+%pytest+` if desired. + +|`+%pytest+` +|`+PATH=… PYTHONPATH=… … %{__pytest}+` +|Various environment variables are set to ensure the packaged version is tested. Use this macro instead of direct `pytest` calls in `+%check+`. Pass additional argument as if passed to `pytest`, e.g. `+%pytest -m "not network"+` to deselect tests marked as `network`. + +|`+%py3_check_import …+` +|`+PATH=… PYTHONPATH=… … %{__python3} -c 'import …'+` +|Various environment variables are set to ensure the packaged version is tested. Use this macro in `+%check+` to test public Python modules are importable if running upstream tests suite is not feasible. Pass module names as positional arguments separated by spaces or commas. + +|`+%{py_dist_name}+` +|(Lua script) +|Given a standardized name (i.e. dist name, name on PyPI) of Python software, it will convert it to a canonical format. See <> for more information. + +|`+%{py3_dist}+` +|(Lua script) +|Given a standardized name (i.e. dist name, name on PyPI) of Python software, it will convert it to a canonical format, and evaluates to `+python3dist(CANONICAL_NAME)+`, which is useful when listing dependencies. See <> for more information. + +|`+%{pypi_source}+` +|(Lua script) +|Evaluates to the appropriate URL for the package. See above for more information. + +|`+%pycached ….py+` +|(Lua script) +|Given a Python file, lists the file and the files with its bytecode cache. See <> for more information. + +|`+%{py3_shebang_flags}+` +|`s` +|The default set of flags for Python shebangs. Redefine or undefine this to change the set. Used by `+%py3_shebang_fix+`. + +|`+%py3_shebang_fix …+` +|(Python script) +|Given paths for Python files or directories with them, it changes Python shebangs to `+#! %{__python3}+`, preserves any existing flags (if found) and adds flags defined in `+%{py3_shebang_flags}+` (if not already present). + +|=== + +During `+%install+` or when listing `+%files+` you can use the `+%{python3_sitearch}+` and `+%{python3_sitelib}+` macros to specify where the installed modules are to be found. For instance: + +.... +%files +# A pure python3 module +%{python3_sitelib}/foomodule/ +# A compiled python3 extension module +%{python3_sitearch}/barmodule/ +.... + +Use of the macros has several benefits: + +* It ensures that the packages are installed correctly on multilib architectures. +* Using these macros instead of hardcoding the directory in the specfile ensures your spec remains compatible with the installed Python version even if the directory structure changes radically (for instance, if `+python3_sitelib+` moves into `+%{_datadir}+`). + +== Packages using Cython + +A great amount of extension modules for Python +(Python modules written in a compiled language such as C or C++) +are written using the https://cython.org/[Cython] language and compiler. + +Majority of such packages contains the generated C (or C++) sources in the source tarball. + +Tightening the xref:what-can-be-packaged.adoc#_pregenerated_code[general Fedora policy], +packages MUST NOT use pre-generated Cython sources. +They MUST be deleted in `%prep` and regenerated during the build. + +Any exception to this rule should be considered a xref:index.adoc#bootstrapping[bootstrapping]. + +== Files to include + +When packaging Python modules, several types of files are included: + +* +*.py+ source files because they are used when generating tracebacks. +* +*.pyc+ byte compiled files. +** Python will try to create them at runtime if they don't exist which leads to spurious SELinux AVC denials in the logs. +** If the system administrator invokes Python with -OO, they will be created with no docstrings. This can break some programs. +* +*.egg-info+ or +*.dist-info+ files or directories. If these are generated by the module's build scripts they must be included in the package because they might be needed by other applications and modules at runtime. + +The source files MUST be included in the same package as the byte compiled versions. + +Packagers SHOULD NOT simply glob everything under the sitelib or sitearch directories. The following SHOULD NOT be used: + +* `+%{python3_sitelib}/*+` +* `+%{python3_sitearch}/*+` +* `+%{python_sitelib}/*+` +* `+%{python_sitearch}/*+` + +And packages MUST NOT include the top-level `+__pycache__+` directory (see below). + +== Byte compiling + +Python will automatically try to byte compile files when it runs in order to speed up startup the next time it is run. These files are saved in files with the extension of .pyc (compiled Python). These files will be located inside a directory named `+__pycache__+`. + +The .pyc files contain byte code that is portable across OSes. If you do not include them in your packages, Python will try (and generally fail) to create them when the user runs the program. If the system administrator runs the program, then the files will be successfully written, causing stray .pyc files which will not be removed when the package is removed. To prevent that the byte compiled files need to be compiled and included in the `+%files+` section. Normally, byte compilation is done for you by the `+brp-python-bytecompile+` script. This script runs after the `+%install+` section of the spec file has been processed and byte compiles any .py files that it finds in `+%{python3_sitelib}+` or `+%{python3_sitearch}+` (this recompilation puts the proper filesystem paths into the modules otherwise tracebacks would include the `+%{buildroot}+` in them). + +You must include the .pyc files in your package. If the build process creates a `+__pycache__+` directory in a subdirectory of `+%{python3_sitearch}+` or `+%{python3_sitelib}+`, you must also include all items in the `+__pycache__+` directory. You MUST NOT include the directories `+%{python3_sitearch}/__pycache__+` or `+%{python3_sitelib}/__pycache__+` because they are already owned by the python3-libs package. + +All that you need to do is include the files in the `+%files+` section (replacing `+%{python3_sitelib}+` with the appropriate macro for your package): + +.... +%files +%{python3_sitelib}/foo/ +.... + +or, if the Python code installs directly into `+%{python3_sitelib}+`, use the `+%pycached+` macro to include the bytecode cache files: + +.... +%files +%pycached %{python3_sitelib}/foo.py +.... + +That evaluates roughly to + +.... +%files +%{python3_sitelib}/foo.py +%{python3_sitelib}/__pycache__/foo.cpython-%{python3_version_nodots}{,.opt-?}.pyc +.... + +NOTE: The `+%pycached+` macro only supports Python 3.5+, +so for older Python versions +(such as 3.4 in EPEL 6 or 7), +you need to list the files manually. + +NOTE: In case you need to use other macros with the `+%pycached+` macro, +such as `+%exclude+` or `+%ghost+`, +pass the other macro as part of the argument to `+%pycached+`. +For example: +`+%pycached %exclude /path/to/foo.py+` +Using the macros in wrong order would only apply `+%exclude+` to the first entry that `+%pycached+` generates. + +=== Manual byte compilation + +For more details on the internals of byte compilation, please see xref:Python_Appendix.adoc#manual-bytecompilation[the appendix]. + +== Example Python spec file + +The following is a very simple spec file for a Python module. + +.python-example.spec +[source] +---- +include::{examplesdir}/python-example.spec[] +---- + +== Reviewer checklist + +The following briefly summarizes the guidelines for reviewers to go over: + +* *Must*: Python modules must be built from source. They cannot simply drop an egg or whl from upstream into the proper directory. (See xref:what-can-be-packaged.adoc#prebuilt-binaries-or-libraries[prebuilt binaries Guidelines] for details). +* *Must*: Python modules must not download any dependencies during the build process. +* *Must*: When building a compat package, it must install using easy_install -m so it won't conflict with the main package. +* *Must*: When building multiple versions (for a compat package) one of the packages must contain a default version that is usable via "import MODULE" with no prior setup. +* *Should*: Additional `+python3-...+` provides should be accomplished via a `+%py_provides+` call. +* *Should*: A package which is used by another package via an egg interface should provide egg info. diff --git a/guidelines/modules/ROOT/pages/Python_Appendix.adoc b/guidelines/modules/ROOT/pages/Python_Appendix.adoc index 95e2948..3a8a4f8 100644 --- a/guidelines/modules/ROOT/pages/Python_Appendix.adoc +++ b/guidelines/modules/ROOT/pages/Python_Appendix.adoc @@ -7,7 +7,7 @@ Here are some additional Python-related guidelines, moved here in order to keep == Python 2 packages If your package needs to build for Python 2 only, -follow the xref:Python.adoc[regular Python guidelines], +follow the xref:_201xPython.adoc[201x-era Python guidelines], except substitute 3 for 2. For example instead of `+%{python3_sitearch}+`, use `+%{python2_sitearch}+`. @@ -346,6 +346,10 @@ You should usually follow upstream's directions on how to run `+2to3+` and build [#manual-bytecompilation] == Manual byte compilation +NOTE: This section only applies for the 201x-era guidelines. +In the new guidelines, see the +xref:Python.adoc#manual-bytecompilation[Manual byte compilation] section. + When byte compiling a .py file, python embeds a magic number in the byte compiled files that correspond to the runtime. Files in `+%{python?_sitelib}+` and `+%{python?_sitearch}+` MUST correspond to the runtime for which they were built. For instance, a pure Python module compiled for the 3.4 runtime MUST be below `+%{_usr}/lib/python3.4/site-packages+` The `+brp-python-bytecompile+` script tries to figure this out for you. From b1cf9816c4bed06a31b13d3561d8f24e65fc5d55 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Jul 20 2021 10:48:30 +0000 Subject: [PATCH 2/7] Document %py3_check_import --- diff --git a/guidelines/modules/ROOT/pages/Python.adoc b/guidelines/modules/ROOT/pages/Python.adoc index 4107a96..0683842 100644 --- a/guidelines/modules/ROOT/pages/Python.adoc +++ b/guidelines/modules/ROOT/pages/Python.adoc @@ -1006,6 +1006,9 @@ network access, dependencies not packaged in Fedora, and/or specialized hardware or resources. +In these cases, +you can use <> +to test that installed modules are importable. ==== Tox @@ -1259,7 +1262,7 @@ Summary: %{summary} %check -%tox / %pytest +%tox / %pytest / %py3_check_import ... %files -n python3-... -f %{pyproject_files} @@ -1490,6 +1493,18 @@ See <> for details. + The command that `+%pytest+` uses. May be redefined. +[#py3_check_import] +* `+%py3_check_import+` ++ +Imports all modules given as positional arguments +(separated by spaces or commas). +If running an upstream test suite is not feasible, +use this macro in `+%check+` to test that public Python modules are importable. ++ +The macro sets various environment variables +such as `+PATH+` and `+PYTHONPATH+` +to ensure the packaged versions of modules are imported. + === Extras macros [#pyproject_extras_subpkg] From 018c5f8090e3ea04ddf55e94aa908ab2fdba5bb0 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Jul 20 2021 10:48:30 +0000 Subject: [PATCH 3/7] Properly mark up anchors for deprecated macros --- diff --git a/guidelines/modules/ROOT/pages/Python.adoc b/guidelines/modules/ROOT/pages/Python.adoc index 0683842..5c51c9f 100644 --- a/guidelines/modules/ROOT/pages/Python.adoc +++ b/guidelines/modules/ROOT/pages/Python.adoc @@ -1731,24 +1731,13 @@ The following macros are deprecated. See the xref:Python[201x-era Python Packaging guidelines] for how some of them were used. -[#py3_build] -* `+%py3_build+` -[#py3_build_wheel] -* `+%py3_build_wheel+` -[#py3_build_egg] -* `+%py3_build_egg+` -[#py3_install] -* `+%py3_install+` -[#py3_install_wheel] -* `+%py3_install_wheel+` -[#py3_install_egg] -* `+%py3_install_egg+` -[#py3dir] -* `+%py3dir+` -[#py3_other_build] -* `+%py3_other_build+` -[#py3_other_install] -* `+%py3_other_install+` -[#python_provide] -* `+%python_provide+` -. +* [[py3_build]] `+%py3_build+` +* [[py3_build_wheel]] `+%py3_build_wheel+` +* [[py3_build_egg]] `+%py3_build_egg+` +* [[py3_install]] `+%py3_install+` +* [[py3_install_wheel]] `+%py3_install_wheel+` +* [[py3_install_egg]] `+%py3_install_egg+` +* [[py3dir]] `+%py3dir+` +* [[py3_other_build]] `+%py3_other_build+` +* [[py3_other_install]] `+%py3_other_install+` +* [[python_provide]] `+%python_provide+` From 33d2332bfa13079d0868b2f22f0a951f252f3e26 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Jul 20 2021 10:48:30 +0000 Subject: [PATCH 4/7] Drop references to old Fedora versions --- diff --git a/guidelines/modules/ROOT/pages/Python.adoc b/guidelines/modules/ROOT/pages/Python.adoc index 5c51c9f..248689e 100644 --- a/guidelines/modules/ROOT/pages/Python.adoc +++ b/guidelines/modules/ROOT/pages/Python.adoc @@ -51,7 +51,7 @@ NOTE: A summary of changes from the older guidelines is available at https://hackmd.io/@python-maint/rJmQQc4DP //// -NOTE: These guidelines only support Fedora 31+. +NOTE: These guidelines only support current Fedora releases. For older releases (such as in EPEL 8), consult the xref:Python.adoc[201x-era guidelines]. @@ -555,13 +555,12 @@ The provide *SHOULD NOT* be added manually: if a generator or macro is not used, do not add the `+python-FOO+` / `+python3.X-FOO+` provides at all. -On Fedora 33+, this is done automatically for package names by a generator. +This is done automatically for package names by a generator. If absolutely necessary, the generator can be disabled by undefining <<__pythonname_provides,the `+%__pythonname_provides+` macro>>. -On releases older than Fedora 33, +For provides that aren’t package names, or (for technical reasons) for packages without files, -or for provides that aren’t package names, the generator will not work. For these cases, the following invocation will provide `+python3-FOO+`, `+python-FOO+` and `+python3.X-FOO+`: @@ -1677,8 +1676,7 @@ or `+%if "%{python3_version}" > "3.8"+` is not possible, because the comparison is performed alphabetically on strings. Hence it is true that `+"3.10" < "3.8"+` (which is not desired). -Starting with RPM 4.16 (Fedora 33), -it is possible to explicitly compare version literals by using the `+v+` prefix, +It is possible to explicitly compare version literals by using the `+v+` prefix, similar to the Python string prefixes: [source,spec] @@ -1688,7 +1686,9 @@ similar to the Python string prefixes: %endif ---- -As a workaround for compatibility with older RPM releases, +[NOTE] +==== +As a workaround for compatibility with RPM releases before 4.16 (Fedora 33), `+%{python3_version_nodots}+` can be compared as an integers: [source,spec] @@ -1700,6 +1700,7 @@ As a workaround for compatibility with older RPM releases, This will work with Python 3.10 (310 > 39), but eventually break with Python 4.0 (40 < 310). +==== === Disabling automation From f60657c3221e6890c781533617e16f2dea465b66 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Jul 20 2021 10:48:30 +0000 Subject: [PATCH 5/7] Fix the opt-out mechanism for Machine-readable provides --- diff --git a/guidelines/modules/ROOT/pages/Python.adoc b/guidelines/modules/ROOT/pages/Python.adoc index 248689e..5cdd9f5 100644 --- a/guidelines/modules/ROOT/pages/Python.adoc +++ b/guidelines/modules/ROOT/pages/Python.adoc @@ -573,6 +573,7 @@ For these cases, the following invocation will provide `+python3-FOO+`, Using the generator or macro is important, because the specific form of the provide may change in the future. +[#Machine-readable-provides] === Machine-readable provides[[_automatic_provides_with_a_standardized_name]] Every Python package *MUST* provide `+python3dist(DISTNAME)+` @@ -589,8 +590,9 @@ if the generator fails to add it, the metadata *MUST* be fixed. These _Provides_ are used for automatically generated _Requires_. -If absolutely necessary, the automatic generator can be disabled by using the -<>. +If absolutely necessary, +the automatic generator can be disabled by undefining the +<<__pythondist_provides,`+%{?__pythondist_provides}+` macro>>. Consider discussing your use case with the Python SIG if you need to do this. === Dependencies @@ -1721,6 +1723,13 @@ Disables automatic generation of unversioned/versioned provides for package name e.g. `+python-FOO+` and `+python3.9-FOO+` for `+python3-foo+`. See <> for more details. +[#__pythondist_provides] +* `+%undefine __pythondist_provides+` ++ +Disables automatic generation of machine-readable Provides, +e.g. `+python3dist(foo)+`. +See <> for more details. + [#_python_no_extras_requires] * `+%global _python_no_extras_requires 1+` + From cdaf71d29225adcf339303bf9f036f828dee6aab Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Jul 20 2021 10:48:30 +0000 Subject: [PATCH 6/7] Fix broken links/anchors --- diff --git a/guidelines/modules/ROOT/pages/Python.adoc b/guidelines/modules/ROOT/pages/Python.adoc index 5cdd9f5..fe6092c 100644 --- a/guidelines/modules/ROOT/pages/Python.adoc +++ b/guidelines/modules/ROOT/pages/Python.adoc @@ -623,6 +623,7 @@ if they embed Python. These rules help ensure a smooth upgrade path when `+python3+` is updated in new versions of Fedora. +[#Automatically-generated-dependencies] === Automatically generated dependencies[[_requires_and_buildrequires_with_standardized_names]] Packages *MUST* use the automatic Python run-time dependency generator. @@ -692,6 +693,7 @@ checking that the dependencies are available in Fedora. See the <> section. +[#Extras] === Extras[[_python_extras]] Python extras are a way for Python projects @@ -1639,7 +1641,7 @@ Given one or more _project names_, it will convert them to the canonical format and evaluate to `+python3dist(DISTNAME)+`, which is useful when listing dependencies. -See <> for more information. +See <> for more information. === System Settings @@ -1714,7 +1716,7 @@ Consider contacting the Python SIG if you need to do this. * `+%{?python_disable_dependency_generator}+` + Disables the automatic dependency generator. -See <> for details. +See <> for details. [#__pythonname_provides] * `+%undefine __pythonname_provides+` From f38eb82de7c7ec50e9caf8c6293966e28f581ea0 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Jul 20 2021 10:48:30 +0000 Subject: [PATCH 7/7] %py*_extras_subpkg: Mention that positional args can be separated with commas --- diff --git a/guidelines/modules/ROOT/pages/Python.adoc b/guidelines/modules/ROOT/pages/Python.adoc index fe6092c..a29f06a 100644 --- a/guidelines/modules/ROOT/pages/Python.adoc +++ b/guidelines/modules/ROOT/pages/Python.adoc @@ -1523,7 +1523,7 @@ Required arguments: + -- ** `+-n+`: name of the “base” package (e.g. `+python3-requests+`) -** Positional arguments: the extra name(s). +** Positional arguments (separated by spaces or commas): the extra name(s). Multiple metapackages are generated when multiple names are provided. -- + @@ -1570,7 +1570,8 @@ Takes these arguments: -- ** `+-n+`: name of the “base” package (e.g. `+python3-requests+`) ** `+-i+`: the `+%files %ghost+` path (glob) to the `+.dist-info+` directory -** Positional arguments specify the extra name(s) +** Positional arguments (separated by spaces or commas) +specify the extra name(s) — multiple metapackages are generated when multiple names are provided. ** `+-f+`: Relative path to the filelist for this metapackage (which should contain the `+%files %ghost+` path (glob)