#2443 F33 System-Wide Change: FlexiBLAS as BLAS/LAPACK manager
Closed: Accepted 3 years ago by decathorpe. Opened 3 years ago by bcotton.

BLAS/LAPACK packages will be compiled against the FlexiBLAS wrapper library, which will set OpenBLAS as system-wide default backend, and at the same time will provide a proper switching mechanism that currently Fedora lacks.


With the GPLv3 Licensing / Linking issue resolved (packages linking against BLAS like normal and FlexiBLAS only acting as a shim at runtime), I think this looks good. +1

+1 assuming we wait for:

Consequently, the authors are going to add the "Linking over a controlled interface" exception in a new release, so that the GPLv3 terms do not apply to the BLAS/LAPACK interface.

Update: The author told me that the exception was added, but he is going to ship a bugfix too, so it may take a few days for testing before pushing the new release. I'll post another update here as soon as it lands in rawhide.

@iucar is the commit with the license change visible anywhere?

Not yet, so let's wait.

New release available. Files under the src directory contain the exception, e.g., this one.

I'm afraid that custom exception might need a statement from our legal.

@iucar can you please ask for review on legal@lists.fp.o ?

If we get input from @spot here, this will not be necessary.

I am no longer responsible for Fedora's Legal issues, however, this seems fine to me.

It's not custom, it's exactly this one. (But of course happy to ask in the list if you consider it necessary.)

It's not custom, it's exactly this one.

Oh, sorry about the confusion, you are right.

FYI: https://bugzilla.redhat.com/show_bug.cgi?id=1859553. This makes no sense to me, but I asked him to please ask legal if he considers it necessary.

After well more than a week, I count the vote as (+3,0,-0). By policy, I am processing this proposal as approved.

Metadata Update from @bcotton:
- Issue tagged with: pending announcement

3 years ago

Metadata Update from @decathorpe:
- Issue close_status updated to: Accepted
- Issue status updated to: Closed (was: Open)

3 years ago

I started this process 12 days ago with 14 packages, and there are several issues not directly related to the change itself:

  • Many packages use cmake, so I not only have to adapt the SPEC to build against FlexiBLAS, but also adapt it to conform to the cmake change proposal. That's fine, happy to help with this, but it's more work.
  • The mass rebuild process has happened (is happening?) at the same time, which means that, if a maintainer is not responsive, then I have to rebase PRs and that's a PITA; basically duplicate work.
  • Some maintainers are probably on vacation, which is great, but bad for me (see last point).
  • Some maintainers did make changes these days while the PR was open, so they either didn't notice it or just ignored it. So more rebasing for me.

In summary, from the initial 14 packages, only 5 are already merged 12 days later, and I rebased the remaining ones:

I'm not sure if this is the right place for this, but I would like to kindly ask for some help from a provenpackager, to merge the PRs above and the ones to come (I took some days off, but next week I will be working on this).

@iucar well, there certainly are worse places to ask, since there's a lot of provenpackagers reading these tickets :) You can ping me via IRC or email when things are ready and I'll help with merging / building

Many thanks, @decathorpe. The ones linked above are ready. The only possible issue is that I rebased some of them manually and the PR pages are missing commits in those cases, e.g. what we are discussing here: https://src.fedoraproject.org/rpms/DSDP/pull-request/2. I reported it upstream here: https://pagure.io/pagure/issue/4952. I hope this is just a bug in how Pagure shows the PR info, but the merge should be ok and contain all the commits.

@iucar do you have an estimate / count of how many package still need changes?

We still have to adapt

  • ~15 packages that link to libopenblas*;
  • ~10 that link to libblas and/or liblapack;
  • ~5 packages that link to lib(s|t)atlas*;
  • 1 package that links to libblis.

Also, I've found that

  • Julia can't be adapted to the change. As discussed with its maintainer, Julia packages do not inherit flags and shared libraries from Julia (as, e.g., R packages do), so they can link to whatever they want, which forces Julia to link to a suffixed version of OpenBLAS to avoid symbol collisions. So we need to keep Julia as an exception.
  • OpenCV and ScaLAPACK have issues to link to FlexiBLAS. I'll discuss this with the author of FlexiBLAS to see how this can be addressed. Also, dependencies of these packages that link simultaneously to them and BLAS/LAPACK cannot be adapted to FlexiBLAS until OpenCV and ScaLAPACK are adapted as well. I've identified ga and nwchem in this category so far.

And BTW 43 packages have been adapted so far (some of them have pending FTBFS for other reasons, some of them have pending builds...).

Update: all packages linking to OpenBLAS and ATLAS addressed (19 merges pending). One package (psfex) specifically needs to link to ATLAS, so no changes implemented there. And there were no packages linking to BLIS (it was bliss, which is another stuff altogether).

Remaining: packages linking to reference BLAS/LAPACK, and issues with ScaLAPACK, OpenCV, scamp and sextractor (I need to discuss these with Martin, but he's on vacation).

Great. I'll give the maintainers responsible for those open PRs a day or two before merging them as provenpackager, if that's okay for you.

No problem. But please, merge linbox, which was opened 3 days ago, and I need that one to work on sagemath and Macaulay2.

Update:

  • 50 packages merged.
  • 12 pending merges.
  • 5 packages with issues (ScaLAPACK-related), but Martin is working on a bugfix for this.
  • 2 packages cannot be adapted (Julia, psfex and 3 packages using LAPACKe; LAPACKe will be supported in a future version of FlexiBLAS though).

But now I've discovered more packages that require some BLAS/LAPACK -devel package (with the dnf trick to obtain BuildRequires from sources) but do not end up linking against the library (??), so still 30 packages to go I guess. :-\

What should I do with this change? (Updated status in my last comment in the tracker bug). In summary:

  • Most code is complete.
  • ScaLAPACK and its 4 dependencies cannot be adapted until we have a patch release of FlexiBLAS (see #3).
  • opencv, scamp and sextractor cannot be adapted until we have a new release of FlexiBLAS with LAPACKe support (this will take longer, see #2).
  • There are some merges still pending. Some of them because the packages had a FTBFS bug open, others due to maintainer inaction.
  • There are a few packages (8) linking against reference BLAS/LAPACK that need to be adapted.
  • There are other packages (19) that require some BLAS/LAPACK implementation but does not end up linking against it, so they can be adapted at any time, because I suppose they only run tests and stuff with that.

@iucar I think the current state is working and testable for the package where it was already implemented. So, I'd say it's fine to continue working on the remaining stragglers, especially those that only seem to run BLAS/LAPACK during the build and not at runtime.

Thanks. Should I change then the bug tracker status to ON_QA?

I think so. Most of the change is implemented, with only a few packages remaining.

Metadata Update from @bcotton:
- Issue untagged with: F33
- Issue set to the milestone: Fedora 33

3 years ago

Login to comment on this ticket.

Metadata