#12527 Fedora 42 Mass Rebuild Tracker
Opened a month ago by jnsamyak. Modified 2 days ago

The Fedora 42 schedule[1] has a mass rebuild scheduled for Wed 2025-01-15, We need to plan and coordinate all tasks in preparation for it. For the driving changes please refer[2].

Please note that if you need to exclude any packages from the rebuild,
you can use the PKG_SKIP_LIST or add a noautobuild file to the root of
your distgit repository. This will instruct the mass rebuild script to
skip these packages. Previously, some OCaml packages were rebuilt, but
they can now be opted out using this method if required.

[1] https://fedorapeople.org/groups/schedule/f-42/f-42-key-tasks.html
[2] https://fedoraproject.org/wiki/Fedora_42_Mass_Rebuild#Driving_Features


Metadata Update from @phsmoura:
- Issue tagged with: medium-gain, medium-trouble

a month ago

gcc/annobin/libtool for this mass rebuild (and over tomorrow some Ada packages rebuilt against that) is still in the f42-build-side-103300 tag. I plan to build another gcc there right now and maybe another one during Tuesday. This should be tagged into f42 proper before the mass rebuild starts.

Thanks, @jakub & @decathorpe for your comment, we will be waiting on the rebuild for these changes to land, would you be kind once these have landed properly, so we have a confirmation that these changes are in rawhide

I will. Note, earlier gcc build finished in 17 hours except for i686 where it is ongoing, earlier finished all arches in just under 24 hours.
The one I'd really like to see in the mass rebuild is now 6h20m into the build, so it might be done at 8am UTC tomorrow, but it could be at 1pm UTC as well or later.

Sure, thanks for the update

gcc & annobin & libtool and some Ada packages are now in rawhide proper.

Thanks, @decathorpe can you confirm if the golang changes are landed?

Okay, I see the build failed :( I'll hold the mass rebuild off until I get a confirmation for this!

Yes, it seems to be a gcc15 issue (same build worked with 14).

ok, golang is all ready now.

However, I see that https://pagure.io/releng/pull-request/12529 isn't merged and has comments to look at/address.

Not sure if we should just fire the mass rebuild now, wait and fix and merge the above or just merge it now and fire off mass rebuild.

I'm just going to wait and if it's not running by tomorrow when I get up, I'll fire it off with the existing script.

Mass rebuild is fired off

It compled yesterday. We had a number of packages where it failed to commit to them, we have identified those and re-run things to build them.

Now we have 117 packages that are on needs-rebuild and not on failed.

These are in various states:

  • Never imported, so no spec file.
  • Fails to make a srpm, so koji doesn't know when it failed to build.
  • There is a newer build, but it's in a side-tag (which the mass rebuild will see and not rebuild).

We should fix those, but I see no blockage to merge the sidetag back now and get bugs filed.

need-rebuild-but-not-failure

There are more possible states:

  • package build was submitted successfully, but it's still running (likely stuck in test suites):

    hdf5-1.14.5-3.fc42
    rust-temp_testdir-0.2.3-7.fc42
    rakudo-2024.12-2.fc42
    perl-Future-IO-0.16-2.fc42
    nvidia-texture-tools-2.1.2-12.fc42

  • package has noautobuild file in dist-git

    freeipa (weird)
    pesign

  • subset of "never imported" packages that appear to be EPEL-only, but which were not retired in rawhide:

    forge-srpm-macros-epel
    python3.11-babel
    python3.11-flit-core-epel
    python3.11-psutil
    python3.11-pygit2
    python3.11-pytz
    python3.11-rpmautospec
    python3.11-rpmautospec-core
    python3.11-setuptools_scm-epel
    python3.11-testpath-epel
    python3.11-tomli-w-epel

There are packages that have been created 7 years ago and which were never imported, like this one: https://src.fedoraproject.org/rpms/python3-cryptography-vectors/commits/rawhide

Would it make sense to do a FESCo-approved mass retirement of cases like this?

I would support that. We would have to sort thru the list and put things in the various cases and then propose retirement for ones that make sense for that. (epel only, never imported, etc)

For the merging part, all the f42-rebuild packages were tagged to the f42 tag, the count was 22000 builds.

For future reference, I would suggest an additional criterion be considered: whether any build of the package has ever been tagged into the target (e.g. f42 for the f42-rebuild). In at least two cases discovered so far, there were packages committed to rawhide which were in the process of replacing others (one a rename, another a compat package) but the migration was still underway. The only existing builds of these packages were in a side-tag which predated the mass rebuild, and they were not ready yet for mainline. Having them built and then merged subsequently broke the buildroots for rawhide and ELN, and probably would have broke composes too if not detected earlier.

Really though, even in a case where a package had been previously built and tagged in rawhide, any incompatible changes which are in pending side-tags when the mass rebuild begins would be susceptible to the same situation; that's probably a lot harder to detect though. If nothing else, perhaps there should be a review of open side-tags prior to the mass rebuild?

See #12540 #12541

Really though, even in a case where a package had been previously built and tagged in rawhide, any incompatible changes which are in pending side-tags when the mass rebuild begins would be susceptible to the same situation; that's probably a lot harder to detect though. If nothing else, perhaps there should be a review of open side-tags prior to the mass rebuild?

Turns out this happened as well, namely with the qt5 stack, which was in the middle of an upgrade in a side-tag when the mass rebuild started. (Qt components have an exact version dependency on each other, so while mostly compatible to dependents, between themselves they are incompatible and must be updated in tandem.)

Metadata Update from @jnsamyak:
- Issue assigned to jnsamyak

18 days ago

Looking at the list from https://pagure.io/releng/issue/12527#comment-951850, I have doubts it is correct. E.g. looking at rubygem-gio2, it is healthy package. OTOH, it is missing rubygem-hashie which is unhealthy starting by its unretirement

Yeah, the list should of course be checked.

rubygem-gio2 failed mass rebuild, but not sure why it didn't appear in the failed and needs rebuild lists.

rubygem-hashie is one of the ones that fails to build srpm, so koji can't tell that the failed build was that. ;( There's an old ticket outstanding for that issue.

Quick script through the list from https://pagure.io/releng/issue/12527#comment-951850 and this one is already retired:

gstreamer1-plugin-openh264 f42 kalev [BLOCKED]

Metadata Update from @jnsamyak:
- Issue untagged with: medium-trouble
- Issue tagged with: high-trouble

2 days ago

Log in to comment on this ticket.

Metadata
Attachments 1