From 6ba255eecac6ba6bc4f74802e6c9c871577bd731 Mon Sep 17 00:00:00 2001 From: Marcel Plch Date: Aug 27 2020 16:20:03 +0000 Subject: [PATCH 1/10] Document order of %pycached and other macros Changed in https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/62 Co-authored-by: Miro Hrončok --- diff --git a/guidelines/modules/ROOT/pages/Python.adoc b/guidelines/modules/ROOT/pages/Python.adoc index c3d0795..a063cea 100644 --- a/guidelines/modules/ROOT/pages/Python.adoc +++ b/guidelines/modules/ROOT/pages/Python.adoc @@ -337,6 +337,13 @@ so for older Python versions (such as 3.4 in EPEL 6 or 7), you need to list the files manually. +NOTE: In case you need to use other macros with the `+%pycached+` macro, +such as `+%exclude+` or `+%ghost+`, +pass the other macro as part of the argument to `+%pycached+`. +For example: +`+%pycached %exclude /path/to/foo.py+` +Using the macros in wrong order would only apply `+%exclude+` to the first entry that `+%pycached+` generates. + === Manual byte compilation For more details on the internals of byte compilation, please see xref:Python_Appendix.adoc#manual-bytecompilation[the appendix]. From 9521a7aac4f9769472c033f33764fbd5624e1a4e Mon Sep 17 00:00:00 2001 From: Miro Hrončok Date: Aug 27 2020 16:20:03 +0000 Subject: [PATCH 2/10] Weaken the rule for python3-foo provides to reflect reality Fixes https://pagure.io/packaging-committee/issue/965 --- diff --git a/guidelines/modules/ROOT/pages/Python.adoc b/guidelines/modules/ROOT/pages/Python.adoc index a063cea..e4171d2 100644 --- a/guidelines/modules/ROOT/pages/Python.adoc +++ b/guidelines/modules/ROOT/pages/Python.adoc @@ -106,7 +106,13 @@ Although this statement can be used anywhere in the spec, we recommend putting i == Provides -Using a fictional module named "example", the subpackage containing the Python 3 version *must* provide `python3-example`. This is of course always the case if the subpackage is named `python3-example` (as in the examples below). If the subpackage has some other name then then `Provides: python3-example` must be added explicitly (but see the `+%python_provide+` macro below). +For any module `foo` intended to be used in Python 3 with `import foo`, +the package that includes it *should* provide `python3-foo`. +This is of course always the case if the subpackage is named `python3-foo` +(as in the examples below). +If the subpackage has some other name, +then `Provides: python3-foo` should be added explicitly +(along with `+%python_provide python3-foo+`, see below). === The %python_provide macro From a4c32e4b876d5b6c767cde554c152b3a6d282537 Mon Sep 17 00:00:00 2001 From: Miro Hrončok Date: Aug 27 2020 16:20:03 +0000 Subject: [PATCH 3/10] Replace %python_provide with %py_provides --- diff --git a/guidelines/modules/ROOT/pages/Python.adoc b/guidelines/modules/ROOT/pages/Python.adoc index e4171d2..33dceb6 100644 --- a/guidelines/modules/ROOT/pages/Python.adoc +++ b/guidelines/modules/ROOT/pages/Python.adoc @@ -112,34 +112,47 @@ This is of course always the case if the subpackage is named `python3-foo` (as in the examples below). If the subpackage has some other name, then `Provides: python3-foo` should be added explicitly -(along with `+%python_provide python3-foo+`, see below). +(via `+%py_provides python3-foo+`, see below). -=== The %python_provide macro +=== The %py_provides macro -All packages that provide `+python3-...+` (for any `+...+`) SHOULD use the `+%python_provide+` macro with the provide, for example: +All packages that provide `+python3-...+` (for any `+...+`) SHOULD also provide `+python-...+` and `+python3.X-...+`. +Starting from Fedora 33, +most of the Python packages named `+python3-...+` will provide such names automatically via the dependency generator in `/usr/lib/rpm/fileattrs/pythonname.attr`. + +Any manually added virtual provides of `+python3-...+` SHOULD be done via the `+%py_provides+` macro. + +Instead of: ---- Provides: python3-pkg_resources = %{version}-%{release} -%{?python_provide:%python_provide python3-pkg_resources} ---- -In Fedora 31 and further, it adds a virtual provide of `+python-...+` for Python 3 libraries. -For previous releases, it used to add the virtual provide for Python 2 libraries. -Since Fedora 33, it also adds virtual provide of `+python3X-...+` (for example `+python39-...+`). +Do: + +---- +%py_provides python3-pkg_resources +---- + +Optionally, +supply a custom `epoch-version-release` as a second argument to `+%py_provides+`. -On Fedora 33 or newer it is no longer needed to use `+%python_provide+` for package names: +On releases older than Fedora 33, +or (for technical limitations) for packages without files, +it is necessary to use `+%py_provides+` even for package names: ---- %package -n python3-%{srcname} Summary: %{summary} -%{?python_provide:%python_provide python3-%{srcname}} +%py_provides python3-%{srcname} ---- -Such provides are now added automatically. -(The only exception are meta-packages without files, for technical limitations.) -Packagers SHOULD try to remove explicit `+%python_provide+` calls for package names, -but MAY preserve them if they aim for compatibility with older releases. +Packagers SHOULD try to remove explicit `+%py_provides+` calls for package names, +but MAY preserve them if they aim for compatibility with older releases or packages without files. +NOTE: Historically, there was `+%python_provide+` macro with similar but different semantics. +It still works for compatibility reasons but it is deprecated and SHOULD NOT be used +and packagers SHOULD replace is with appropriate `+%py_provides+` call. == Automatic Provides with a standardized name @@ -204,9 +217,9 @@ The following macros are defined for you in all supported Fedora and EPEL releas |`+%{__python3}+` |Python 3 interpreter. Use this macro in spec files. -|`+%{python_provide}+` +|`+%py_provides+` |(Lua script) -|See <> for detailed explanation. +|See <> for detailed explanation. |`+%{python3_sitelib}+` |`+/usr/lib/python3.X/site-packages+` @@ -376,5 +389,5 @@ The following briefly summarizes the guidelines for reviewers to go over: * *Must*: Python modules must not download any dependencies during the build process. * *Must*: When building a compat package, it must install using easy_install -m so it won't conflict with the main package. * *Must*: When building multiple versions (for a compat package) one of the packages must contain a default version that is usable via "import MODULE" with no prior setup. -* *Should*: Additional `+python3-...+` provides should be accompanied by a `+%python_provide+` call. +* *Should*: Additional `+python3-...+` provides should be accomplished via a `+%py_provides+` call. * *Should*: A package which is used by another package via an egg interface should provide egg info. From 20f4482312d1d077341585a722d1578541a07a92 Mon Sep 17 00:00:00 2001 From: Miro Hrončok Date: Aug 27 2020 16:20:03 +0000 Subject: [PATCH 4/10] %pypi_source now strips ~ from %version by default --- diff --git a/guidelines/modules/ROOT/pages/Python.adoc b/guidelines/modules/ROOT/pages/Python.adoc index 33dceb6..f619db9 100644 --- a/guidelines/modules/ROOT/pages/Python.adoc +++ b/guidelines/modules/ROOT/pages/Python.adoc @@ -189,7 +189,7 @@ Requires: %{py3_dist virtualenv pyPEG2} When packaging software which is available from PyPI, you can make use of the `+%pypi_source+` macro. This macro accepts from zero to three arguments and evaluates to an appropriate URL for the source file on PyPI. The arguments are: 1. The name of the PyPI project. Defaults to `+%srcname+` if defined, or to `+%pypi_name+` if defined, or to `+%name+` (the package name). -2. The version of the PyPI project. Defaults to `+%version+` (the package version). +2. The version of the PyPI project. Defaults to `+%version+` (the package version) with any `~` characters removed (used for alpha/beta/dev versions in RPM version but not in Python package version). 3. The file extension to use. Defaults to `tar.gz`. In most cases it is not necessary to specify any arguments. From e3f958f068627ea3e801a3762fb1414462e5bfde Mon Sep 17 00:00:00 2001 From: Miro Hrončok Date: Aug 27 2020 16:20:03 +0000 Subject: [PATCH 5/10] Document %pytest --- diff --git a/guidelines/modules/ROOT/pages/Python.adoc b/guidelines/modules/ROOT/pages/Python.adoc index f619db9..f0679bf 100644 --- a/guidelines/modules/ROOT/pages/Python.adoc +++ b/guidelines/modules/ROOT/pages/Python.adoc @@ -253,6 +253,14 @@ The following macros are defined for you in all supported Fedora and EPEL releas |`+%{__python3} setup.py install --skip-build …+` |Various flags are passed to `+setup.py install+`, see `/usr/lib/rpm/macros.d/macros.python3` for details and similar macros. To add extra flags/arguments to `+setup.py install+`, separate them with `+--+`, for example: `+%py3_install -- --install-scripts %{_libexecdir}+`. To pass custom command line arguments directly to `+setup.py+`, define `+%py_setup_args+`. +|`+%__pytest+` +|`+/usr/bin/pytest+` +|The `pytest` command used in `+%pytest+`. Don't use this macro directly, but feel free to redefine it for usage in `+%pytest+` if desired. + +|`+%pytest+` +|`+PATH=… PYTHONPATH=… … %{__pytest}+` +|Various environment variables are set to ensure the packaged version is tested. Use this macro instead of direct `pytest` calls in `+%check+`. Pass additional argument as if passed to `pytest`, e.g. `+%pytest -m "not network"+` to deselect tests marked as `network`. + |`+%{py_dist_name}+` |(Lua script) |Given a standardized name (i.e. dist name, name on PyPI) of Python software, it will convert it to a canonical format. See <> for more information. From 673bcf95ee9b243fe171c0cf8044a525c690ae4c Mon Sep 17 00:00:00 2001 From: Miro Hrončok Date: Aug 27 2020 16:20:04 +0000 Subject: [PATCH 6/10] Document %py3_shebang_fix --- diff --git a/guidelines/modules/ROOT/pages/Python.adoc b/guidelines/modules/ROOT/pages/Python.adoc index f0679bf..0bb1de1 100644 --- a/guidelines/modules/ROOT/pages/Python.adoc +++ b/guidelines/modules/ROOT/pages/Python.adoc @@ -21,7 +21,8 @@ Since Fedora 31 `/usr/bin/python` is, if it is installed, a symbolic link to `/u It was a symbolic link to `/usr/bin/python2` on previous releases. Packages in Fedora MUST NOT use `/usr/bin/python`. Instead packages for Python 3 MUST use `/usr/bin/python3` (even if upstream supports both Python 2 and 3). As a result of that `/usr/bin/python` (as well as `/usr/bin/env python` and similar) MUST NOT be used in shebang lines or as a dependency of a package. All uses of unversioned python executables in shebang lines will fail the build. -These shebangs MUST be fixed. If it is necessary to disable the checks, please see the information in xref:index#_shebang_lines[Shebang lines]. +These shebangs MUST be fixed (for example by using the `+%py3_shebang_fix+` macro in the spec file). +If it is necessary to disable the checks, please see the information in xref:index#_shebang_lines[Shebang lines]. All Python runtimes have a virtual provide for `+python(abi) = $MAJOR.$MINOR+`. For example, the Python 3.7 runtime package has: @@ -277,6 +278,14 @@ The following macros are defined for you in all supported Fedora and EPEL releas |(Lua script) |Given a Python file, lists the file and the files with its bytecode cache. See <> for more information. +|`+%{py3_shebang_flags}+` +|`s` +|The default set of flags for Python shebangs. Redefine or undefine this to change the set. Used by `+%py3_shebang_fix+`. + +|`+%py3_shebang_fix …+` +|(Python script) +|Given paths for Python files or directories with them, it changes Python shebangs to `+#! %{__python3}+`, preserves any existing flags (if found) and adds flags defined in `+%{py3_shebang_flags}+` (if not already present). + |=== During `+%install+` or when listing `+%files+` you can use the `+%{python3_sitearch}+` and `+%{python3_sitelib}+` macros to specify where the installed modules are to be found. For instance: From fcba721610d19c9e60671c91561cb305aab6865c Mon Sep 17 00:00:00 2001 From: Miro Hrončok Date: Aug 27 2020 16:20:04 +0000 Subject: [PATCH 7/10] %__python is error https://fedoraproject.org/wiki/Changes/PythonMacroError --- diff --git a/guidelines/modules/ROOT/pages/Python.adoc b/guidelines/modules/ROOT/pages/Python.adoc index 0bb1de1..66b97a4 100644 --- a/guidelines/modules/ROOT/pages/Python.adoc +++ b/guidelines/modules/ROOT/pages/Python.adoc @@ -206,7 +206,7 @@ The following macros are defined for you in all supported Fedora and EPEL releas |Notes |`+%{__python}+` -|`+/usr/bin/python+` (for backwards compatibility) +|(Error) |Don't use this macro without redefining it. Defining it changes the meaning of other "unversioned" Python macros such as `+%{python}+` or `+%{python_sitelib}+`. From feb4aea05b0bd12f1bba0534a7b6653d0f220da9 Mon Sep 17 00:00:00 2001 From: Miro Hrončok Date: Aug 27 2020 16:20:04 +0000 Subject: [PATCH 8/10] Python: Drop all "since Fedora 31" --- diff --git a/guidelines/modules/ROOT/pages/Python.adoc b/guidelines/modules/ROOT/pages/Python.adoc index 66b97a4..f4ed367 100644 --- a/guidelines/modules/ROOT/pages/Python.adoc +++ b/guidelines/modules/ROOT/pages/Python.adoc @@ -17,7 +17,7 @@ For guidelines on maintaining already existing python2 packages, see the xref:Py == Multiple Python Runtimes -Since Fedora 31 `/usr/bin/python` is, if it is installed, a symbolic link to `/usr/bin/python3`. +On Fedora `/usr/bin/python` is, if it is installed, a symbolic link to `/usr/bin/python3`. It was a symbolic link to `/usr/bin/python2` on previous releases. Packages in Fedora MUST NOT use `/usr/bin/python`. Instead packages for Python 3 MUST use `/usr/bin/python3` (even if upstream supports both Python 2 and 3). As a result of that `/usr/bin/python` (as well as `/usr/bin/env python` and similar) MUST NOT be used in shebang lines or as a dependency of a package. All uses of unversioned python executables in shebang lines will fail the build. @@ -96,7 +96,7 @@ python3.7dist(tornado) >= 4 python3.7dist(traitlets) >= 4.2.1 .... -Note that since Fedora 31, `+.0+` suffixes are removed from version numbers to match the behavior of Python tools. +Note that any `+.0+` suffixes are removed from version numbers to match the behavior of Python tools. (https://www.python.org/dev/peps/pep-0440/#final-releases[PEP 440] specifies that `+X.Y+` and `+X.Y.0+` are treated as equal.) This generator is enabled by default in Fedora. If a packager wishes to explicitly opt out of the generator because the upstream metadata are not applicable, a packager SHOULD opt out explicitly by adding: From 47242015e11fc2fdd9ddde1b218e2df7f2bd9eac Mon Sep 17 00:00:00 2001 From: Miro Hrončok Date: Aug 27 2020 16:20:04 +0000 Subject: [PATCH 9/10] Document https://fedoraproject.org/wiki/Changes/PythonExtras Co-authored-by: Petr Viktorin --- diff --git a/guidelines/modules/ROOT/pages/Python.adoc b/guidelines/modules/ROOT/pages/Python.adoc index f4ed367..de0a34f 100644 --- a/guidelines/modules/ROOT/pages/Python.adoc +++ b/guidelines/modules/ROOT/pages/Python.adoc @@ -41,6 +41,13 @@ The source package for a Python library MUST be named with the `python-` prefix. This rule does not apply to applications. +The character `+` in names of built packages (i.e. non-SRPM) +that include `.dist-info` or `.egg-info` directories is reserved for <> and MUST NOT be used for any other purpose. +The `+` character triggers the automatic dependency generator for extras. +Replace any `+` signs in the upstream name with `-`, +or omit them when at the beginning of the name. +As an exception, `+` characters are permitted at the _end_ of the name. + == Dependencies Packages building for Python 3 will need `BuildRequires: python3-devel`. @@ -105,6 +112,111 @@ This generator is enabled by default in Fedora. If a packager wishes to explicit Although this statement can be used anywhere in the spec, we recommend putting it just before the main package's `+%description+` declaration. +=== Python Extras + +https://www.python.org/dev/peps/pep-0508/#extras[Python extras] are a way for Python projects to declare that extra dependencies are required for additional functionality. + +For example, `requests` has several standard dependencies (e.g. `urllib3`). +But it also declares an _extra_ named `+requests[security]+`, +which lists additional dependencies (e.g. `cryptography`). +Unlike RPM subpackages, +extras can only specify additional dependencies, +not additional files. +The main package will work if the optional dependency is not installed, +but it might have limited functionality. + +Python tools treat extras as virtual packages. +For example, +if a user runs `+pip install requests[security]+`, +or installs a project that depends on `+requests[security]+`, +both `requests` and `cryptography` will be installed. + +Starting with Fedora 33, +extras are usually provided by packages with no files. +Instead of square brackets, +Fedora package names conventionally use the `+` character to separate the package name and the _extra_ name, +e.g. the package would be named `python3-requests+security`. +The plus sign is valid in RPM package names, +but not in Python canonical project names nor in extras identifiers. + +Python packages SHOULD have `Provides` for all extras the upstream project specifies, +except those that are not useful for other packages +(for example build/development requirements, +commonly named `dev`, `doc` or `test`). + +A package that provides a Python extra MUST provide `+python3dist(…[…])+` and `+python3.Xdist(…[…])+`, +for example, `+python3.9dist(requests[security])+`. +These requirements SHOULD be generated using the automatic dependency generator. + +A package that provides a Python extra MUST require the extra's main package with exact NEVR. + +A subpackage that primarily provides one Python extra SHOULD be named by appending `+` and the extra name to the main package name. +For example, `++python3-requests+security++`. + +The most straightforward way to provide an extra is with a dedicated subpackage containing no files (a "metapackage"). +This case can be automated with the `+%python_extras_subpkg+` macro. + +Alternative approach: +when some extra is always useful in a distro, +it can be provided by the main package; +when several extras are related, +they may be provided by a single subpackage. +However, +having one dedicated subpackage per extra +allows you to use the automatic dependency generator to ensure that the extras' requirements will stay in sync with upstream. +If you create a dedicated subpackage and want it to be always/usually installed, you MAY Require/Recommend/Suggest it from the main package. + +The dependency generator for extras activates if the following holds: + +- The package must contain the `.egg-info`/`.dist-info` directory, usually as `+%ghost+`. +- The package name must end with `+EXTRA` (where `EXTRA` is the extra name). + +As an example, +the extra subpackage for `+requests[security]+` can be specified using the `+%python_extras_subpkg+` convenience macro as follows. +The macro takes the main package name and name(s) of the extra(s) as well as path to the `.egg-info` or `.dist-info` directory: + +---- +%{?python_extras_subpkg:%python_extras_subpkg -n python3-requests -i %{python3_sitelib}/*.egg-info security} +---- + +For this case, +the extras dependency generator will read upstream metadata from the `.egg-info` directory. +If it finds that the `security` extra has a dependency on `cryptography`, +it will generate `+Requires: python3.Xdist(cryptography)+`, +`+Provides: python3dist(requests[security])+` +(and the corresponding `+python3.Xdist+` variant). + +If you need additional features that the `+%python_extras_subpkg+` macro does not cover, +you will need to write the subpackage sections manually. +Such features can be, for example: + +- Obsoleting/providing other names (e.g. obsoleted extras packages) +- Manual strong or weak dependencies on other (possibly non-Python) packages +- Including files excluded from the main package (if such files only make sense with the extra and the base package does not fail without them) + +As an example of what you need to write in these cases, +the `+%python_extras_subpkg+` macro invocation above expands to the following: + +---- +%package -n python3-requests+security +Summary: Metapackage for python3-requests: security extras +Requires: python3-requests = %{?epoch:%{epoch}:}%{version}-%{release} +%description -n python3-requests+security +This is a metapackage bringing in security extras requires for python3-requests. +It contains no code, just makes sure the dependencies are installed. + +%files -n python3-requests+security +%ghost %{python3_sitelib}/*.egg-info +---- + +Note that the dependency generator does not add a dependency on the main package +(the `+Requires: python3-setuptools_scm = ...+` above). +If you are not using the `+%python_extras_subpkg+` macro, +you need to add it manually. + +NOTE: The `+%python_extras_subpkg+` can take multiple extras names to generate multiple packages. +For more options, see the https://fedoraproject.org/wiki/Changes/PythonExtras[change proposal which introduced this]. + == Provides For any module `foo` intended to be used in Python 3 with `import foo`, From 163c0ca77908977e5deea9bfbf3d6d94185abad0 Mon Sep 17 00:00:00 2001 From: Miro Hrončok Date: Aug 27 2020 16:20:04 +0000 Subject: [PATCH 10/10] Bump the review date of Python guidelines --- diff --git a/guidelines/modules/ROOT/pages/Python.adoc b/guidelines/modules/ROOT/pages/Python.adoc index de0a34f..f201fdb 100644 --- a/guidelines/modules/ROOT/pages/Python.adoc +++ b/guidelines/modules/ROOT/pages/Python.adoc @@ -1,5 +1,5 @@ = Python Packaging Guidelines -:last-reviewed: 2020-01-01 +:last-reviewed: 2020-08-05 :toc: == Python Version Support