#11952 Untag jpegxl-0.9.2-2.fc41 and friends from FEDORA-2024-72fe625282
Closed: Fixed 3 months ago by kevin. Opened 3 months ago by adamwill.

  • Describe the issue
    Please untag all the builds from https://bodhi.fedoraproject.org/updates/FEDORA-2024-72fe625282 (and equivalent builds from ELN, if they have been tagged). The update is missing a rebuild of webkitgtk. We cannot have webkitgtk FTI, it will break composes. This unfortunately wasn't gated because of the holes around branching (we don't immediately apply a gating policy to Rawhide after branching, and I think Bodhi maybe doesn't have a critpath definition for Rawhide until the next time the critpath generation job runs, because the update wasn't marked critpath when it should be).

  • When do you need this? (YYYY/MM/DD)
    ASAP.

  • When is this no longer needed or useful? (YYYY/MM/DD)
    When a webkitgtk rebuild is done.

  • If we cannot complete your request, what is the impact?
    Rawhide composes will probably fail.


Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

3 months ago

Thanks. Not sure how we go about re-landing this when a webkitgtk build is possible? Do we need to create a new side tag, tag all the builds onto that, add a webkitgtk build, and create a new update?

Not sure that will work because bodhi already has those builds in an update. ;(
But possibly: make new sidetag, tag in builds from here that are needed in the buildroot, rebuild webkitgtk, untag builds except webkitgtk from the sidetag, make a update for webkitgtk, retag all these in f41 and then webkitgtk should pass and go in?

ok , sorry for the throuble

No worries at all... things happen. :)

It looks like https://koji.fedoraproject.org/koji/taskinfo?taskID=113499057 somehow wound up in the default build channel instead of the heavybuilder channel.

The builder itself looks like it has an appropriate amount of RAM for the job, and this is further enforced by the %limit_build macro. But since it's not using the heavybuilder channel, perhaps one or more unrelated builds got scheduled at the same time, after the compilation started? %limit_build works by computing an appropriate amount of parallelism at the time the job starts and so basically only works if nothing else is happening on the builder at the same time; it will fail if another build consumes a significant amount of memory.

It is using the heavybuilder channel. ;)

✗ koji list-hosts --enabled --channel heavybuilder --arch i386
Hostname Enb Rdy Load/Cap Arches Last Update


buildhw-x86-13.iad2.fedoraproject.org Y N 12.0/2.0 x86_64,i386 Wed, 14 Feb 2024 15:15:27 PST
buildhw-x86-14.iad2.fedoraproject.org Y N 6.0/2.0 x86_64,i386 Wed, 14 Feb 2024 15:15:18 PST
buildhw-x86-15.iad2.fedoraproject.org Y Y 0.0/2.0 x86_64,i386 Wed, 14 Feb 2024 15:15:18 PST
buildhw-x86-16.iad2.fedoraproject.org Y N 6.0/2.0 x86_64,i386 Wed, 14 Feb 2024 15:15:21 PST

all 4 of those are bare hardware builders with 48 cpus and 128gb memory.

Unfortunately, i686 is 32bit, so it will not be able to address more than 8gb. I'm not sure what changed since the last working 32bit build that would have caused more memory use...

It is using the heavybuilder channel. ;)

Hm, no, this particular build was using the "default" channel which you can see here. It just got assigned to a machine that also handles the heavybuilder channel, right? Anyway, seems that's irrelevant.

Unfortunately, i686 is 32bit, so it will not be able to address more than 8gb.

Oh yeah. Well, drat. Well, that's probably the end of the road for this package on i686 then. That's one single cc1plus program compiling just one file. We knew this day would come eventually; actually, I'm a little surprised it didn't happen sooner....

If we really need to, we could maybe make it work by turning off unified build, which would lower the memory requirements somewhat. But the cost will be many additional build failures, because this configuration is not well-tested. I'd rather ExcludeArch. But it's not a leaf package, so all dependencies will also need ExcludeArch. And there are a lot of them.

Ah, right, it was default because that was a scratch build. Only official builds get put on heavybuilder. ;(

Could try it on a buildvm (smaller, but possibly qemu/kvm does better handling it?)

JFTR this build [1] was in heavybuilder channel and also failed

[1]
https://koji.fedoraproject.org/koji/taskinfo?taskID=113473592

cc1plus: out of memory allocating 6034444 bytes after a total of 89718784 bytes - isn't it just ~90 MB, unless I'm misreading it somehow? That should be far from the 4 GB per-process memory limit that 32 bit architectures have.

I put a potential fix in https://src.fedoraproject.org/rpms/webkitgtk/pull-request/8 - it's completely untested, but hopefully helps. Please don't merge unless the CI scratch build succeeds :)

cc1plus: out of memory allocating 6034444 bytes after a total of 89718784 bytes - isn't it just ~90 MB, unless I'm misreading it somehow? That should be far from the 4 GB per-process memory limit that 32 bit architectures have.

You're right. Now I have no clue what the problem is here. :) It is indeed failing to allocate very small amounts of memory.

I will try bumping %limit_build up to 3 GB per vCPU. Maybe this will work? Probably not, but worth a try.

Hm, we already require 3 GB per vCPU. I think it surely should not require 4 GB per vCPU. That's going to reduce parallelism too much. But I guess we'd better try it anyway....

Still fails with the 4 GB %limit_build:

cc1plus: out of memory allocating 6034444 bytes after a total of 87654400 bytes

Tellingly, the amounts are very similar each time.

I put a potential fix in https://src.fedoraproject.org/rpms/webkitgtk/pull-request/8 - it's completely untested, but hopefully helps. Please don't merge unless the CI scratch build succeeds :)

OK, this did not help at all, but I have a new theory now, which is that it's a kernel regression. I've opened an infra ticket at https://pagure.io/fedora-infrastructure/issue/11775 to test the theory.

Login to comment on this ticket.

Metadata