I think this is a good idea.

We probably need some clear definition of a "major toolchain upgrade", but I agree in principle.

For example: is it a major upgrade if we rev GCC from a beta to an RC? What about if we have had a GA version for a while, but want to change the distro compilation flags?

I meant "major" in the "version" way, like GCC 14 -> 15, or Go 1.23 -> 1.24, or LLVM 19 -> 20.

CC: @siddhesh @jakub

cc @codonell and @nickc as well for glibc and binutils respectively. FWIW, they're less affected by this because glibc and binutils are rebased every 1-2 weeks in rawhide, so they get more or less continuous verification throughout the lifecycle of rawhide. So it's essentially gcc we're talking about here and possibly, rebuild testing of a subset of packages before the mass rebuild. I'll let Carlos and Nick speak for glibc and binutils.

For gcc, stage 1 closes in mid-November and it typically needs another couple of weeks to incorporate changes that came in late and then another week or so to get the compiler to a point that it's not breaking everything. Around the first couple of weeks of December we start our mass prebuild. Since it's an early compiler, it takes a couple of iterations to get results, at which point it's Christmas holidays and much of the team goes on vacation, returning the next year.

Finally in the new year we have most hands on deck sifting through the prebuild failures and getting a side tag going to have, e.g. libtool, annobin and ada ready for the new gcc. This goes right up to the day of the mass rebuild to get as many compiler fixes in as we can, so that they're not the source of mass rebuild failures.

So simply adding the new gcc to the buildroot isn't very helpful IMO. What may help though is for us to file FTBFS bugs from mass prebuild failures early (i.e. not wait to conclude triage that we typically do) so that maintainers are involved. This would mean more noise for package maintainers in December when the compiler isn't nearly as stable, but the benefit would be that the mass rebuild goes much smoother.

So I'd say instead of:

Major toolchain upgrades must land at least 7 days before the scheduled start of the mass rebuild.

I would suggest adding the following checkpoint to the release:

• Major GCC upgrade proposal must have a mass prebuild with FTBFS reports at least 3 weeks before the scheduled start of mass rebuild.*

For the gcc team this would mean:

1. In December, we create a side tag with a gcc build that fixed packages can subsequently build into.
2. We run a mass prebuild and immediately file FTBFS reports for packages that built with the previous gcc but not with the new one. this would probably be an RFE for mass-prebuilder @fberat
3. Package maintainers help with the triage, moving obvious gcc bugs (e.g. ICEs) to gcc and fixing actual code issues
4. Merge the side tag into rawhide before the mass rebuild commences.

@jakub do you see an issue with that?

1-1.5 months lived side-tag is very dangerous.
Only perhaps for scratch builds or if it keeps just gcc/annobin/libtool in it and nothing else for most of the duration, because otherwise we risk conflicts between what is built into that side-tag and what is built into rawhide.

BTW, I think this is closely related to when the mass rebuild is actually done. Looking at past years, I see it was happening in Feb 3, Feb 7, Jan 31, 28, 26, 20, 19, 16 (from older to newer, was looking for Rebuilt for https://fedoraproject.org/wiki/Fedora_NN_Mass_Rebuild commit messages), so with one exception each year it is earlier and earlier. Which makes this request to add further -7 days for that really hard to achieve.

Note, when the mass rebuild was at bearable dates, gcc side tag used to be merged into rawhide around Jan 15th or so, so there would be a week with it before the mass rebuild.

Only perhaps for scratch builds or if it keeps just gcc/annobin/libtool in it and nothing else for most of the duration, because otherwise we risk conflicts between what is built into that side-tag and what is built into rawhide.

I suppose that's fair; package fixes for gcc updates would essentially be package bugs and those fixes ought to work with the older gcc as well. So those fixes can go right into rawhide and don't have to go into the side tag.

How about things like the ada package updates though? Or in general, ABI updates that need package rebuilds against that specific gcc? I reckon those need to go into the side tag, but maybe we could allow them in later not for the entire 1+ month duration. Are there any other package groups that need similar treatment?

Maybe discussion like that should happen on the mailing list first?

(Note that I didn't want to single out GCC here at all, Go and other toolchains are in similar boats.)

Ahh yes, cc @jistone @alexsaezm and @tstellar too for rust, golang and llvm. it's probably not very relevant for llvm and rust, but just in case.

Rust is updated every six weeks across all branches, so it wouldn't be affected here. LLVM is updated during beta freeze and always gets an excpetion from FESCo for landing the update late.

Still, I'd prefer if this discussion happened on the mailing list, and not on this ticket - hence I posted:

https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/AOZZQJDIMS5G2Q3TIP6LIKTDWCBYSBLT/

I think it might also make sense for GHC. cc @petersen

The discussion on fedora-devel happened. I'll try to summarize it briefly:
- people generally agree that it'd be beneficial to have the new compilers available earlier.
- gcc folks do test rebuilds with the new compiler version already in December, but so far have held back on opening bugzillas before the results are triaged.
- for the compiler update to really help, builds of dependent packages need to be done and failures reported.
- a suggestion was made to introduce the new compiler after branching. That'd mean that delaying the compiler by 6 months, or doing the introduction a second time in branched. So this doesn't seem like a good idea.
- there was also a huge side discussion whether mass rebuilds are necessary, and whether more targeted mass rebuilds would be better. Approx. 50% of packages are compiled, but determining the full set of dependent packages is not obvious. Personally, I think there are good reasons to keep the mass rebuilds, but either way, this should be a separate discussion.

The original proposal was:

Major toolchain upgrades must land at least 7 days before the scheduled start of the mass rebuild.

+1 in the light of the discussion

Should this milestone also include a mass prebuild and test requirement in December or should we treat it as an implicit consensus and file FTBFS bugs before a full triage of build failures with the new gcc, as I had suggested in the thread?

Without that, putting the toolchain upgrade into rawhide sooner won't be very effective, especially considering the constantly creeping up dates for the mass rebuild in future Fedora.

should we treat it as an implicit consensus and file FTBFS bugs before a full triage of build failures with the new gcc

I think that's reasonable. I wouldn't want to make an official policy though, because that'd mean putting a strong responsibility on people over the X-mas break.

putting the toolchain upgrade into rawhide sooner won't be very effective

Once the compiler package is updated, we'll have about a week for interested parties to test and possibly deal with any FTBFS. We'll see how it works out in practice. But I think that that it's better to start with this small step for which we have general agreement that it is a step in the right direction.

Major toolchain upgrades must land at least 7 days before the scheduled start of the mass rebuild.

+1

AGREED: APPROVED(5,0,0)