#9858 Fedora 34 System Toolchain Rebase
Closed: Fixed 3 years ago by kevin. Opened 3 years ago by law.

  • Describe the issue

All Fedora releases must be released using a released and supported version of the system toolchain which contains glibc, gcc, and binutils.

The Fedora toolchain team is responsible for ensuring that Fedora Rawhide stabilizes static linking, code generation, and library ABI, before a Fedora release, or that after the branch that the Fedora release is rebased (a very small rebase) to the final released version. This is a requirement for Fedora to inherit the ABI and API guarantees provided by upstream. The leaders for GCC (law@redhatcom), GLIBC (codonell@redhat.com) and binutils (nickc@redhat.com) are here to coordinate with Fedora release engineering to address any mass rebuild issues, fix any ABI/code generation issues, ec.

We are requesting a mass rebuild for Fedora 34. This has been standard operating procedure for even numbered Fedora releases to pick up the new compiler each spring release. Typically the mass rebuild occurs in late Jan or early Feb and that schedule should work for the toolchain teams.

  • When do you need this? (2021/02/01)

  • When is this no longer needed or useful? (2021/05/01)

Change page for review: https://fedoraproject.org/wiki/Changes/GNUToolchain
Owner: Carlos O'Donell carlos@redhat.com and Jeff Law law@redhat.com

  • If we cannot complete your request, what is the impact?
    Packages would not be built with the new compiler. Existing packages would continue to work but would not benefit from code generation improvements or new/improved diagnostics in the compiler.

There may be problems with the float128 transition if a mass rebuild can not be completed. This would only affect IBM Power. However, we consider the float128 transition as a whole at risk as the upstream bits are not yet complete. The float128 transition may end up being pushed back to Fedora 36.


@law You might want to consider renaming the change proposal page to indicate some versioning to indicate what it has, because you do this regularly (34, 36, 38, and so on).

Metadata Update from @humaton:
- Issue tagged with: change-ack, changes, f34

3 years ago

Metadata Update from @mohanboddu:
- Issue tagged with: mass rebuild

3 years ago

Will the mass rebuild be ordered at least to some extent (by package dependencies if there aren't cycles)?
A big question is if we should enable --with-long-double-format=ieee on ppc64le, which is an important ABI change (glibc and gcc should stay ABI compatible, but otherwise libraries that use long double in their interfaces might not be. For that to work, it would be best if library packages were rebuilt first, starting with those on which most of the stuff depends on, tag those rebuilds (perhaps into a side-tag) and only then rebuild dependencies.

The mass rebuild does packages a-z order, and uses a side tag, so it does NOT update the build root.

Perhaps you could do this in a side tag before mass rebuild and merge it in before the mass rebuild starts?

@law Also whats the status on this change as f34 mass rebuild is scheduled for this Wed, Jan 20th 2021.

We need some rpm debugedit fixes (to be able to cope with DWARF 5 which is now the default in GCC 11) and then if all goes right I can start a gcc build tomorrow afternoon. Unless it gets stuck it should be ready for the mass rebuild. But the ppc64le long double change needs to be postponed.

@jakub I see a gcc-11.0.0-0.15.fc34 build happening, does that include all the fixes needed for gcc 11 for mass rebuild?

For gcc, yes it does. Mark Wielaard found some dwz bug that might be good to get fixed, but that doesn't need rebuilding gcc and dwz build is quick.
For dwarf5, we need the latest gcc, rpm (for debugedit), binutils and dwz.

@jakub Thanks for the update, it seems like all of them landed in f34/rawhide (gcc is still building). So, we are good for mass rebuild on this change. Thanks a lot.

Except for the dwz fixes, those will hopefully be done tomorrow.
At what time starts the mass rebuild?

Except for the dwz fixes, those will hopefully be done tomorrow.
At what time starts the mass rebuild?

Oh right, sorry, I misinterpreted your comment as dwz fix is not really required for mass rebuild.

There is no specific time to start mass rebuild, we can start in the evening of US time tomorrow, does that give you enough time?

I hope yes. Once we have a fix and do some testing perhaps with scratch builds, if we have it earlier, we'll let you know.

@jakub Thanks, we can extend the time by a day as well, but we prefer to start on the scheduled date, keep me posted and we can figure it out.

So I'm doing test builds to see how pervasive the annobin vs FORTIFY_SOURCE and the dwz assertion fail issues are. I've only got about a dozen machines busy right now, but will probably scale that up to 20 or 30 through the day. Meaning we should have a couple thousand test builds done by tomorrow am and we can more reasonably assess the impact of those two problems and hopefully get mjw a nice small testcase for the dwz issue :-)

There is also this possibly related issue https://pagure.io/fedora-infrastructure/issue/9589
"rpmbuild crashes on s390x just after "Checking for unpackaged file": Child return code was: -11"

The GCC bugs (PR98765 and issue 9589) should be hopefully fixed in
https://koji.fedoraproject.org/koji/taskinfo?taskID=60103978
https://koji.fedoraproject.org/koji/taskinfo?taskID=60105277
that I'm now building for f34 (side-tag) and eln.
ETA 12-24 hours.
The two dwz bugs AFAIK aren't fixed yet.

Probably not related to the GNU toolchain changes, but I'm posting this here because I expect the relevant audience is watching, and this is about the larger Fedora toolchain. There is an RPM oddity with the Fedora 34 packages I can't explain:

Both fixed gcc and dwz are in now in f34 buildroots:
koji wait-repo --build=dwz-0.13-6.fc34 --build=gcc-11.0.0-0.16.fc34 f34-build
Successfully waited 0:01 for dwz-0.13-6.fc34 and gcc-11.0.0-0.16.fc34 to appear in the f34-build repo
So from my POV everything for the mass rebuild is done (though would be nice if we can throw a couple of small rpms that previously failed to build, whether due to the strip of LTO problems, dwz crashes or rpm crashes on s390x during writing packages.

Note that boost and binutils would also like to make it into the mass rebuild:
https://pagure.io/releng/issue/9950 (boost 1.75)
https://pagure.io/releng/issue/9898 (binutils 2.36)

boost is currently building, but binutils might not make it till monday:
https://pagure.io/fesco/issue/2555 (Request to postpone the mass rebuild to Jan 25th, 2021)

Mass rebuild is now done.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

3 years ago

Just made a comment/open suggestion regarding new DWARF
support for casual debugging purposes of distro packages at
accompanying Binutils change ticket, not sure where it is more fitting
nor whether any subsequent "user experience" tweaks are warranted,
but I for one would like to see it work more nicely:
https://pagure.io/releng/issue/9539#comment-718052

Login to comment on this ticket.

Metadata