Current state: a large number of packages fail [1,2] to build because: 1. gcc has been removed from the build root [https://fedoraproject.org/wiki/Changes/Remove_GCC_from_BuildRoot] 2. g++ has been removed from the build root [same change, but not included in the title, so less obvious] 3. /usr/bin/python has been removed from the build root [https://fedoraproject.org/wiki/Changes/Move_usr_bin_python_into_separate_package] 4. %python_sitelib, %python_sitearch evaluate to empty, so packages using that in %files fail to build 5. %python_sitelib, %python_sitearch evaluate to empty, so packages using that in %files build, but are not installable, and depend packages fail to build
1-2 are much more common, but really simple to fix. 3-5 are more complicated. 5 in particular breaks builds of other packages, so is particularly unpleasant. There are right now 2578 failed builds, but we don't know what percentage 1-5 are responsible for. From the builds I looked at, I'd estimate 1→25 cases, 2→10 cases, 3→8 cases, 4→10 cases, 6→2 cases, compared to 5 cases of failures from other reasons. So "gcc removal" is the biggest culprit, but "python separate package" also causes some noticeable percentage of issues, so I'm discussing both of those changes in the same ticket.
The first question is how much time we have. The mass rebuild started a few days ago, and it's not been very long. But judging from the past, I don't expect maintainers to fix packages quickly. With 2k+ packages this will take forever.
Some things we can do: a. There have been calls on fedora-devel to revert the gcc change. The simple contingency plan of adding gcc and gcc-c++ in the buildroot is easy to do. b. We could ask Change owners and proven packagers and other packagers to fix packages c. Alternatively, we could not do b, and instead wait to see if maintainers step up to do the (simple) fixes required for their packages. I think that at least python-sig has the sentiment that if packagers cannot do the simple updates for their packages, those packages should be orphaned and retired. d. Write and run a script to do the fixes automatically. Can Change owners do this?
For 1-2, my thinking is that we shouldn't pull the plug on the gcc change yet. A script could look at every failing package in turn, try to build it after adding gcc and after adding gcc-c++ to BR, and if that succeeds, push a change to do that to dist-git and build the package in koji. This is a different approach than ignatenko's regexp-based approach, more brute-force.
For 4-5, with %python_sitelib/sitearch, @churchyard proposed (IIRC) adjusting all macros that have those macros with a script. I think we should do that as soon as possible. The number of packages that need such a fix is not that high, but it's still large enough to be worth an automatic fix.
After the fixes in two previous paragraphs are done, the number of ftbfs packages should drop significantly, and then we can consider if further contingency actions are needed. Hopefully not.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1602938 [2] https://kojipkgs.fedoraproject.org/mass-rebuild/f29-failures.html
Comments, ideas, alternative approaches? I think we should discuss this tomorrow, so I'm setting the meeting tag.
/cc @ignatenkobrain, @churchyard, @pviktori
Where are you going to run this? I'm leaving my current job so I won't have any more the server with 128 cores. I'm rather willing to spend few days of my time to manually analyze and fix packages.
Also if we don't do it now, we basically can't do it ever ;)
Where are you going to run this?
I think it's feasible to run this on any bigger machine. Last year I did a mass rebuild of python packages on my 12 core workstation and it was good enough. If that doesn't work, it's possible to do single-arch scratch builds in koji.
. I'm rather willing to spend few days of my time to manually analyze and fix packages.
I don't think "manual" is good enough. I did some packages yesterday, and some today, and it's just too slow.
I don't think this is true. We could (not saying that I think this is the best way, but is certainly is a feasible way) revert the change now, and fix spec files over the next few months, and undo the revert for F30 or F31 or even later. And also, there isn't any absolute need to do this change. We should do it if the benefits outweigh the problems. If we need to fix 2k packages manually, then this isn't worth it.
As for Python, I see three categories:
1) packages that FTBFS (and haven't been rebuilt for 3.7 while having Python 3 bits):
I'm monitoring those and will apply python macro fixes if it makes the package builds. Currently we have 82 packages that still runtime require Python 3.6, yet the number that fails to build (only) due to python related macro changes is (close to) 0.
2) packages that FTBFS (and already have been rebuilt for 3.7 or don't have Python 3 bits):
One of the desired side effects of the change. @adamwill nicely summarized it in this e-mail. Please don't fix those. Them not being rebuilt does not cause any trouble (of course, if you find a case where it does, fix it and rebuild it). The policy for FTBFS packages need to happen here and we need to get rid of 100 years old legacy Python cruft.
3) packages that list %{python_sitexxx}/* in %files, hence owning / and /usr, etc.
I've collected those, opened PRs for all of them and will merge and build in 1 week of no activity (that's tomorrow afternoon UTC). Several have been already merged by responsive maintainers. The rest is ignored by the nonresponsive ones (or possibly vacations, it has only been 6 days). Funny enough it has been 42 PRs in total and 21 remain. Which is quite sad but reflects the feeling I got when @ishcherb was filling PRs for python -> python2 requires. About half of the PRs remained unanswered (hopefully point 2 from above would help with this unfortunate situation).
It seems like the python bits are under control, and ftbfs will be used as a pruning mechanism for old packages. The downside is that this adds a lot of noise to the process. But OK, I can live with that.
I've added dependencies to the bug: https://bugzilla.redhat.com/show_bug.cgi?id=1551327
Going to auto-fix all those packages within few days.
Oh, that's great. Cool, thanks.
We will discuss this in today's meeting, which starts in about an hour in #fedora-meeting-1.
It was agreed in today's meeting that there is no further action needed on this ticket as we can just let the FTBFS process do its job.
Metadata Update from @bowlofeggs: - Issue untagged with: meeting - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.