I would like to give this a shot and package it. This package depends on mizani, which is not yet in Fedora. So, that needs packaging first.

Do I need to be in the neuro-sig group to assign to me?

Do I need to be in the neuro-sig group to assign to me?

You might (or might not) need to be in the group to do the assigning. I have assigned it to you. Thanks!

Thanks @music!

I took a closer look at plotnine. It provides a set of extras, one of which has another unsatisfied dependency: adjustText.

Looks like I will need to package at least python-mizani first. If I understand the Python Packaging Guidelines correctly, I will also need to provide python-adjusttext.

Hi @gui1ty : thanks for working on this. I've added you to the neuro-sig pagure group so you can also modify tickets here. Are you already a package maintainer, if yes, would you like to join the neuro-sig package maintainers group too? Otherwise we'll also be happy to help you get sponsored to the package maintainers group and then add you to the neuro-sig specific one.

On the topic of dependencies: yes, unfortunately, we need to package up all the deps (at least the necessary ones) before we can package the software itself :(

It’s permissible to omit extras metapackages with unsatisfied dependencies. It sometimes makes sense to do this in cases where the dependencies they would add are excessively difficult to package, or can’t be packaged in a way that is compliant with the packaging guidelines. However, whenever we can, it’s much better to package all the extras and their associated dependencies.

I made some progress.

For the base package python-plotnine requires python-mizani, which requires python-palettable. I packaged those and put them in a Copr repo

A first build without tests for now, but %pyproject_check_import instead, failed with:

File "/builddir/build/BUILDROOT/python-plotnine-0.9.0-1.fc38.x86_64/usr/lib/python3.11/site-packages/plotnine/__init__.py", line 1, in <module>
    from .qplot import qplot            # noqa: F401
    ^^^^^^^^^^^^^^^^^^^^^^^^

I'm not that familiar with all things python. But qplot is part of plotnine. Is that an issue with the syntax used in connection with the build system?

PS: Sorry for the noise. Case of fat fingers :blush:

On second thought the root issue might be a few lines up in the log with _bootstrap._gcd_import:

Check import: plotnine
Traceback (most recent call last):
  File "/usr/lib/rpm/redhat/import_all_modules.py", line 171, in <module>
    main()
  File "/usr/lib/rpm/redhat/import_all_modules.py", line 167, in main
    import_modules(modules)
  File "/usr/lib/rpm/redhat/import_all_modules.py", line 100, in import_modules
    importlib.import_module(module)
  File "/usr/lib64/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/builddir/build/BUILDROOT/python-plotnine-0.9.0-1.fc38.x86_64/usr/lib/python3.11/site-packages/plotnine/__init__.py", line 1, in <module>
    from .qplot import qplot            # noqa: F401
    ^^^^^^^^^^^^^^^^^^^^^^^^

Of course, it's down the rabbit hole we go. The revealing message was at the bottom:

ModuleNotFoundError: No module named 'matplotlib._contour'

Yet, I was unable to solve it. I tried adding python3-matplotlib to BuildRequires explicitly to no avail. I also tried different ways of having the requirements for testing determined by macros. I parked the issue for now and continued without tests. The failed build, mentioned above is still in Copr along with a couple of other failed attempts. Pointers as to what I'm missing are very much appreciated.

Since I went through several iterations, the spec files are a bit messy. Please don't be distracted by the comments in there.

The matplotlib docs say this was removed/renamed in 3.0.1:

https://matplotlib.org/3.5.3/api/prev_api_changes/api_changes_3.0.1.html?highlight=_contour

Seems to be fixed upstream now (5 days ago) but hasn't made it to the release yet:

https://github.com/has2k1/plotnine/commit/c48d5a26ab0d0a54ed47bbefc6a83b5fcb2af616

You could try carrying the patch to see if it applies cleanly:

Patch: https://github.com/has2k1/plotnine/commit/c48d5a26ab0d0a54ed47bbefc6a83b5fcb2af616.patch

...
BuildRequires:  git-core

%autosetup ... -S git

Thank you for the pointer. I will try the patch.

I was looking at my own system, which is still on f35 [1]. In python3-matplotlib-3.5.3-1.fc35 matplotlib.contour._contour is still present. The matplotlib doc says these API changes were made for 3.0.1. So, why does f35 using version 3.5.3 not have them?

Regarding your suggestion, I like the git approach. I haven't seen it done like this before. I was more inclined to use a local patch file in the SRPM. Is this acceptable/save to use in official packages (not Copr)? I haven't found anything in the Packaging Guidelines about directly applying patches from upstream. I would need to enable internet access in the Copr project to use the git approach, I assume.

One last question: The patch should be applied conditionally along the lines of %if 0%{?fedora} >= 37, right?

[1] Maybe I should update to f37 now it is in beta. I will certainly enable some more chroots, seeing that f36 uses the same release of matplotlib still as f35.

Hi @gui1ty : thanks for working on this. I've added you to the neuro-sig pagure group so you can also modify tickets here. Are you already a package maintainer, if yes, would you like to join the neuro-sig package maintainers group too? Otherwise we'll also be happy to help you get sponsored to the package maintainers group and then add you to the neuro-sig specific one.

Thank you, @ankursinha. Yes, I was very recently sponsored into the package maintainers group.

What are the responsibilities of the neuro-sig package maintainers and how much commitment is required time wise? It certainly sounds interesting and would provide some hands on experience with packaging in Fedora. Moreover, I'm currently focusing on packaging Python software. In that respect neuro-sig appears to be a good match.

On the topic of dependencies: yes, unfortunately, we need to package up all the deps (at least the necessary ones) before we can package the software itself :(

That's not a problem and quite understandable. With respect to plotnine, it probably means a couple more dependencies that require packaging. :snake:

Thank you for the pointer. I will try the patch.

I was looking at my own system, which is still on f35 [1]. In python3-matplotlib-3.5.3-1.fc35 matplotlib.contour._contour is still present. The matplotlib doc says these API changes were made for 3.0.1. So, why does f35 using version 3.5.3 not have them?

I think 3.0.1 removed matpotlib._contour as part of some clean up, not matplotlib.contour._contour. It's surprising that plotnine was still using it tbh---their CI should've caught this sort of thing, but maybe they pinned the matplotlib version there etc.

Regarding your suggestion, I like the git approach. I haven't seen it done like this before. I was more inclined to use a local patch file in the SRPM. Is this acceptable/save to use in official packages (not Copr)? I haven't found anything in the Packaging Guidelines about directly applying patches from upstream. I would need to enable internet access in the Copr project to use the git approach, I assume.

Ah, sorry, I wasn't clear. We still download the patch and check it into SCM as a local file---no network access required there. autosetup goes through all the patches and applies them for us:

https://rpm-software-management.github.io/rpm/manual/autosetup.html

One last question: The patch should be applied conditionally along the lines of %if 0%{?fedora} >= 37, right?

If required, yes. In this case, though, all current releases have matplotlib > 3.5.0, so the patch will probably be needed everywhere:

https://packages.fedoraproject.org/pkgs/python-matplotlib/python3-matplotlib/

[1] Maybe I should update to f37 now it is in beta. I will certainly enable some more chroots, seeing that f36 uses the same release of matplotlib still as f35.

Sure---we usually use mock (using fedpkg mockbuild) to run test builds for different releases, and as a package maintainer, you can also run scratch builds on koji already:

https://docs.fedoraproject.org/en-US/package-maintainers/Using_the_Koji_Build_System/#scratch_builds

Hi @gui1ty : thanks for working on this. I've added you to the neuro-sig pagure group so you can also modify tickets here. Are you already a package maintainer, if yes, would you like to join the neuro-sig package maintainers group too? Otherwise we'll also be happy to help you get sponsored to the package maintainers group and then add you to the neuro-sig specific one.

Thank you, @ankursinha. Yes, I was very recently sponsored into the package maintainers group.

What are the responsibilities of the neuro-sig package maintainers and how much commitment is required time wise? It certainly sounds interesting and would provide some hands on experience with packaging in Fedora. Moreover, I'm currently focusing on packaging Python software. In that respect neuro-sig appears to be a good match.

It's really just a way of giving us all access to the packages so we can all maintain them together---similar to the python sig or the rust sig. We meet once every two weeks just to go over the status of all our packages etc. Folks that can't make it to the meeting go over the logs to keep themselves up to date. Other than that, we spend whatever time we can squeeze out after work and so on :)

We do have lots of Python tools packaged, and in the queue, so it'll be really great to have you on the team to help us look after them too :sparkles:

It's really just a way of giving us all access to the packages so we can all maintain them together---similar to the python sig or the rust sig. We meet once every two weeks just to go over the status of all our packages etc. Folks that can't make it to the meeting go over the logs to keep themselves up to date. Other than that, we spend whatever time we can squeeze out after work and so on :)

I think that will work for me :wink:. I will try to join the meeting, next Monday.

We do have lots of Python tools packaged, and in the queue, so it'll be really great to have you on the team to help us look after them too :sparkles:

I'll be happy to help where I can. I just subscribed to neurofedora.

Thank you for your mentorship @ankursinha! I really appreciate it.

Status Update

For python-adjustText I patched in the LICENSE file from the GitHub for the time being. I opened a ticket asking for the file to be included in the PyPI tarball. But the project looks rather dead. No activity in two years.

Going with Murphy's Law, I tackled the most challenging dependency last. The python-scikit-misc packages uses Cython with C and Fortran code. The build fails due to some header file not being found. I filed an issue upstream.

I applied the patch as suggested. Tests are running now. A lot of them fail. Some require scikit-misc. So the total picture on tests is incomplete until the build issue with python-scikit-misc is solved. Due to setuptools >= 59 required, plotnine will only be available in F36 and up.

All the packages in their current state are available in Copr along with the logs and fedora-review results where applicable.

Great, I've added you to the neuro-sig packager group now. Please log out and back into src.fp.o---that updates group membership there IIRC.

Please also subscribe to this ML for updates on all our bugs:

https://lists.fedoraproject.org/admin/lists/neuro-sig.lists.fedoraproject.org/

It's private because all the Bugzilla notifications (including private bugs) go there.

For these packages, I think we can now start submitting them for formal review and we can deal with remaining issues during the review. Could you open reviews for them when you have a minute, and please block the fedora-neuro bug that we use to track all our reviews?

https://bugzilla.redhat.com/show_bug.cgi?id=fedora-neuro

With the help of python-sig I managed to get scikit-misc past the build stage. However, it seems the tests cannot import what the package itself provides. Running just %pyproject_check_import also fails. I think the underlying issue is the same.

There's also a library provided by the package of which I'm not sure the %pyproject macros are handling that correctly (or are even supposed to). A second pair of eyes would be appreciated.

Build with import test: https://copr.fedorainfracloud.org/coprs/gui1ty/neuro-sig/build/4877025/
Build with pytest: https://copr.fedorainfracloud.org/coprs/gui1ty/neuro-sig/build/4877091/

That means python-scikit-misc and python-plotnine are ready for review. I will submit them shortly.

Sadly, none of the earlier submitted packages has yet been picked up for review.

Sounds great @gui1ty . Sorry, busy period at work. If no one takes them up, I'll try and do them this week.

A good way of getting packages reviewed is also to ask for "review swaps" on the -devel list. I.e., you review this for me, and I review a package for you in return.

Eg: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/CFIVDX3D2VFQ2JPLJO7EVBFP3PNDYJPP/#CFIVDX3D2VFQ2JPLJO7EVBFP3PNDYJPP

Given the dependency chain and the work involved in getting scikit-misc (Cython, tests) packaged, I do no longer consider this easy.

A good way of getting packages reviewed is also to ask for "review swaps" on the -devel list. I.e., you review this for me, and I review a package for you in return.

Yep, I already have a message in draft, which will go out at the beginning of next week, if needed.

This is getting close to being finished. All packages have been reviewed and are available from rawhide.

I hit one last issue regarding tests in python-plotnine. Two tests are failing with:

E   ImportError: attempted relative import with no known parent package

This happens only in f36. In f37 and rawhide I don't see this error. @music would you have any idea on where to look or how to fix this? The full log is available in Copr:

https://download.copr.fedorainfracloud.org/results/gui1ty/neuro-sig/fedora-36-x86_64/04988240-python-plotnine/builder-live.log.gz

I hit one last issue regarding tests in python-plotnine. Two tests are failing with:

python E ImportError: attempted relative import with no known parent package

I disabled the offending tests for f36 and opened an issue upstream.