#2451 Delay mass rebuild (was: aarch64 PAC/BTI depends on bug fixes in gcc 10.2)
Closed: Accepted 2 years ago by ignatenkobrain. Opened 2 years ago by jlinton.


The approved PAC/BTI feature https://fedoraproject.org/wiki/Changes/Aarch64_PointerAuthentication should be dependent on the latest gcc 10.x branch because PAC testing uncovered a couple bugs which have been fixed https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94891.
The gcc 10.2 branch is frozen and is targeted towards 7/23 for a release. https://gcc.gnu.org/pipermail/gcc/2020-July/233136.html

Can we hold up the (aarch64?) mass rebuild for the day it takes to get the gcc release? Although I haven't yet heard from the gcc team whether they will manage to land it in time.


Metadata Update from @ignatenkobrain:
- Issue tagged with: fast track

2 years ago

I see that also LTO stuff did not land yet, so I would support idea for delaying mass rebuild for a day or two.

+1 from me FTR


But, uh, do we even need to vote for this? I thought we gave the brave people in releng the power to delay for up to 5 days without asking FESCo:


PS: Also, is the LTO stuff you mention the thing that was already approved for F32 but postponed?

Pinging @mohanboddu for his awareness.

But, uh, do we even need to vote for this? I thought we gave the brave people in releng the power to delay for up to 5 days without asking FESCo:

They have the power to do this without asking FESCo, but this request didn't come from releng; it came from the AARCH64 folks. FESCo still has the power to say "please delay the start of the rebuild".

+1 from me.

I think we're also waiting for the new binutils which I think is due end of this week.

So I spoke via IRC with Jakub. upstream gcc-10.2 should go out Thursday and the builds for Fedora would likely land Friday after COB.

Not sure if that affects plans or not. Just relaying the info.

The LTO stuff is ready and just needs a final pull-request to be issued.

APPROVED (+7, 0, -0)

@kevin @mohanboddu all yours to delay rebuild until new gcc is in place (guess this Friday, according to comments here).

Metadata Update from @ignatenkobrain:
- Issue tagged with: pending announcement

2 years ago

FYI, I am planning on delaying it until next Monday due to binutils 2.35 release.


Announced by Mohan from the releng side.

Metadata Update from @ignatenkobrain:
- Issue close_status updated to: Accepted
- Issue status updated to: Closed (was: Open)

2 years ago

I realize this was closed, but figured this was the best place to comment on the binutils issues and my thoughts on the mass rebuild.

In simplest terms LTO has exposed a latent binutils bug. While I haven't really hacked binutils in decades, I was able to dig into it enough to understand what's happening, why and give a good description of the issue to Nick.

It's my opinion that the bug is unlikely to occur often, but I suspect we will find other packages that are affected, even if they aren't obviously failing at this time. We have a way to look at executables and determine if they're affected by this issue.

In the immediate term I've disabled LTO in binutils and issued a fresh build. I would recommend going forward with the mass rebuild. I'll set up some scripting locally to pull the resultant .rpms from the mass rebuild, scan them for the tell-tale signs of the binutils bug so that we can address any such issues on a package-by-package basis.

I'll also be watching for FTBFS issues to jump on any LTO related problems. I'm aware of the gnulib ppc64 float module testsuite failure. This is highly likely an instance of a configure test that is compromised by LTO. I need to verify that tomorrow AM. I wouldn't consider that to be a blocker issue. I'll be able to spot them quickly if there's more instances.

@law if there is anything we can help you with from releng side please use this issue https://pagure.io/releng/issue/9616

Login to comment on this ticket.