There seems to be a problem with Koji/x86_64 builders, they may not have enough ram to build Firefox. Local build works as well as Fedora 42 and Fedora 40 (we build the same packages everywhere).
RH BZ: https://bugzilla.redhat.com/show_bug.cgi?id=2327276
build log: https://kojipkgs.fedoraproject.org//work/tasks/1469/125991469/build.log
build error:
31:04.53 process didn't exit successfully: CARGO=/usr/bin/cargo CARGO_CRATE_NAME=gkrust CARGO_MANIFEST_DIR=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/toolkit/library/rust CARGO_PKG_AUTHORS='The Mozilla Project Developers' CARGO_PKG_DESCRIPTION='Rust code for libxul' CARGO_PKG_HOMEPAGE='' CARGO_PKG_LICENSE=MPL-2.0 CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME=gkrust CARGO_PKG_README='' CARGO_PKG_REPOSITORY='' CARGO_PKG_RUST_VERSION='' CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE='' CARGO_PRIMARY_PACKAGE=1 LD_LIBRARY_PATH=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/release/deps /usr/bin/rustc --crate-name gkrust --edition=2015 toolkit/library/rust/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type staticlib --emit=dep-info,link -C opt-level=2 -C panic=abort -C embed-bitcode=no -Clto --cfg 'feature="mozilla-central-workspace-hack"' --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values("mozilla-central-workspace-hack"))' -C metadata=4c0393d7b47f1bf6 -C extra-filename=-4c0393d7b47f1bf6 --out-dir /builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -C linker=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/build/cargo-linker -C strip=debuginfo -L dependency=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps -L dependency=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/release/deps --extern gkrust_shared=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps/libgkrust_shared-71562ab5c69d5bc3.rlib --extern lmdb_sys=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps/liblmdb_sys-c88bc00694315534.rlib --extern mozglue_static=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps/libmozglue_static-73cba90dbd312cee.rlib --extern mozilla_central_workspace_hack=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps/libmozilla_central_workspace_hack-dcb2f7431174c37c.rlib --extern swgl=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps/libswgl-4bac420510ca07eb.rlib -C debuginfo=2 --cap-lints warn -Copt-level=3 -Cdebuginfo=2 -Ccodegen-units=1 -Cstrip=none -Cforce-frame-pointers=yes --cap-lints=warn -Cembed-bitcode=yes -C codegen-units=1 -L native=/usr/lib64 -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/x86_64-unknown-linux-gnu/release/build/audioipc2-b96252f74a6da322/out -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/dist/bin -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/security/nss/lib/nss/nss_nss3 -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/security/nss/lib/ssl/ssl_ssl3 -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/config/external/nspr/pr -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/x86_64-unknown-linux-gnu/release/build/lmdb-rkv-sys-43a35bc3600a50f0/out -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/dist/bin -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/security/nss/lib/nss/nss_nss3 -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/security/nss/lib/ssl/ssl_ssl3 -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/config/external/nspr/pr -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/x86_64-unknown-linux-gnu/release/build/mozglue-static-a9cf1c457f1ef2b3/out -L native=/usr/lib64 -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/dist/bin -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/security/nss/lib/nss/nss_nss3 -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/security/nss/lib/ssl/ssl_ssl3 -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/config/external/nspr/pr -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/x86_64-unknown-linux-gnu/release/build/swgl-556d68d8f8052e6f/out (signal: 9, SIGKILL: kill)
CARGO=/usr/bin/cargo CARGO_CRATE_NAME=gkrust CARGO_MANIFEST_DIR=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/toolkit/library/rust CARGO_PKG_AUTHORS='The Mozilla Project Developers' CARGO_PKG_DESCRIPTION='Rust code for libxul' CARGO_PKG_HOMEPAGE='' CARGO_PKG_LICENSE=MPL-2.0 CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME=gkrust CARGO_PKG_README='' CARGO_PKG_REPOSITORY='' CARGO_PKG_RUST_VERSION='' CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE='' CARGO_PRIMARY_PACKAGE=1 LD_LIBRARY_PATH=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/release/deps /usr/bin/rustc --crate-name gkrust --edition=2015 toolkit/library/rust/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type staticlib --emit=dep-info,link -C opt-level=2 -C panic=abort -C embed-bitcode=no -Clto --cfg 'feature="mozilla-central-workspace-hack"' --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values("mozilla-central-workspace-hack"))' -C metadata=4c0393d7b47f1bf6 -C extra-filename=-4c0393d7b47f1bf6 --out-dir /builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -C linker=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/build/cargo-linker -C strip=debuginfo -L dependency=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps -L dependency=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/release/deps --extern gkrust_shared=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps/libgkrust_shared-71562ab5c69d5bc3.rlib --extern lmdb_sys=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps/liblmdb_sys-c88bc00694315534.rlib --extern mozglue_static=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps/libmozglue_static-73cba90dbd312cee.rlib --extern mozilla_central_workspace_hack=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps/libmozilla_central_workspace_hack-dcb2f7431174c37c.rlib --extern swgl=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps/libswgl-4bac420510ca07eb.rlib -C debuginfo=2 --cap-lints warn -Copt-level=3 -Cdebuginfo=2 -Ccodegen-units=1 -Cstrip=none -Cforce-frame-pointers=yes --cap-lints=warn -Cembed-bitcode=yes -C codegen-units=1 -L native=/usr/lib64 -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/x86_64-unknown-linux-gnu/release/build/audioipc2-b96252f74a6da322/out -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/dist/bin -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/security/nss/lib/nss/nss_nss3 -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/security/nss/lib/ssl/ssl_ssl3 -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/config/external/nspr/pr -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/x86_64-unknown-linux-gnu/release/build/lmdb-rkv-sys-43a35bc3600a50f0/out -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/dist/bin -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/security/nss/lib/nss/nss_nss3 -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/security/nss/lib/ssl/ssl_ssl3 -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/config/external/nspr/pr -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/x86_64-unknown-linux-gnu/release/build/mozglue-static-a9cf1c457f1ef2b3/out -L native=/usr/lib64 -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/dist/bin -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/security/nss/lib/nss/nss_nss3 -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/security/nss/lib/ssl/ssl_ssl3 -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/config/external/nspr/pr -L native=/builddir/build/BUILD/firefox-132.0.2-build/firefox-132.0.2/objdir/instrumented/x86_64-unknown-linux-gnu/release/build/swgl-556d68d8f8052e6f/out
Metadata Update from @zlopez: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: Needs investigation
Top level build: https://koji.fedoraproject.org/koji/buildinfo?buildID=2586254
The x86_64 one was on buildvm-x86-11.
This has 15gb memory and 6 cpus.
It was indeed oom:
Nov 18 20:34:07 buildvm-x86-11.iad2.fedoraproject.org kernel: Out of memory: Killed process 3257450 (rustc) total-vm:15261424kB, anon-rss:7328344kB, file-rss:664kB, shmem-rss:0kB, UID:1000 pgtables:28268kB oom_score_adj:0
Whats different about f41 though? Looks like the same cargo as other branches...
We request 4GB per core by build directive in spec:
%constrain_build -m 4096
is that ignored? Or is it possible to get number of memory and adjust cores accordingly?
We request 4GB per core by build directive in spec: %constrain_build -m 4096 is that ignored? Or is it possible to get number of memory and adjust cores accordingly?
The latter is actually what the macro does, from /usr/lib/rpm/macros.d/macros.build-constraints:
/usr/lib/rpm/macros.d/macros.build-constraints
# Macros to constrain resource use during the build process # Changes _smp_build_ncpus depending on various factors # # -c cpus constrains the CPU count to "cpus" # -m mem constrains the CPU count to the total amount of memory in the system # (in megabytes) divided by "mem", rounded down # # If no options are passed, sets _smp_build_ncpus to 1. # _smp_build_ncpus will never be raised, only lowered.
I.e. it will restrict %_smp_build_ncpus accordingly, so each parallel build process gets about that number of memory (or more).
%_smp_build_ncpus
The process killed seems to have used almost 16GB in total and 8GB in RAM: … kernel: Out of memory: Killed process 3257450 (rustc) total-vm:15261424kB, anon-rss:7328344kB …
… kernel: Out of memory: Killed process 3257450 (rustc) total-vm:15261424kB, anon-rss:7328344kB …
I.e. it will restrict %_smp_build_ncpus accordingly, so each parallel build process gets about that number of memory (or more). The process killed seems to have used almost 16GB in total and 8GB in RAM: … kernel: Out of memory: Killed process 3257450 (rustc) total-vm:15261424kB, anon-rss:7328344kB …
Firefox uses this construction:
%constrain_build -m 4096 echo "mk_add_options MOZ_MAKE_FLAGS=\"-j%{_smp_build_ncpus}\"" >> .mozconfig
i.e. set number of CPU for build by _smp_build_ncpus. Is that correct? Also I wonder why it's Fedora 41 only, all other works fine.
The 41 only thing makes me think it's something in the f41 toolchain... because builders are used for all builds, there's not any different pools of them or anything.
Is it consistent? I only see the one failed build, but you might have done many more with the same commit (and koji just replaces the old one with new one here).
I tired it three times before reporting this one, all failed.
Before filing this? Have you tried after wed's outage? (builders/hubs were updated to f41).
Can you try some more now and see if the problem persists?
If we can't figure things out and you need to get updates out we could also (relunctantly) add it to the heavybuild channel. That means you will have contention with webkitgtk and chromium tho...
I did try a scratch build and it failed: https://koji.fedoraproject.org/koji/taskinfo?taskID=126219396
Looks like a OOM indeed:
[Mon Nov 25 00:05:50 2024] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=payload ,mems_allowed=0,global_oom,task_memcg=/machine.slice/machine-0bdda9363fbe46ccb4957672e6905e50.scope/payload,task=rustc,pid=2392673,uid=1000 [Mon Nov 25 00:05:50 2024] Out of memory: Killed process 2392673 (rustc) total-vm:16862620kB, anon-rss:8214288kB, file-rss:2208kB, shmem-rss:0kB, UID:1000 pgtables:30256kB oom_score_adj: 0
I fired new 133.0 builds with the same result: https://koji.fedoraproject.org/koji/buildinfo?buildID=2592382
I'm on PTO this week, so lets just try and set it to use heavybuild and see if that gets us past this week and we can look more next.
I've set it now, can you try another build(s)?
Yes, I'm going to build 133.0 builds this week.
133.0 builds seem to be working, so this can be closed from my POV. Thanks for fixing that!
Metadata Update from @zlopez: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Issue status updated to: Open (was: Closed)
Metadata Update from @zlopez: - Issue untagged with: Needs investigation - Issue tagged with: medium-gain, medium-trouble, ops
Was there a reason you reopened this, @zlopez ?
Well, I am not sure I would say this is solved.
I added firefox to heavybuilder and it's working, but... that means a lot more contention for webkitgtk/chromium too...
I am not sure why memory needs on x86_64 increased so much here. ;(
@adamwill That happened when I added labels :( Not intentional.
@kevin Was there any change in defaults for rpm builds? Like some new C++ flags.
I don't know if there were such changes. @stransky ?
we can try a scratch build and see how that does on the 'normal' builders? I will do so.
Couldn't this be the same issue as https://issues.redhat.com/browse/CS-2695 @arrfab said there was some change needed for koji configs
@zlopez it's clearly some languages thing that are in the way, as for the issue you're pointing it, the same infra was able to build firefox/thunderbird with default settings for core/memory but for c10s/el10 it suddenly increased somehow ...
My scratch build failed I think for mass rebuild/rools issues. ;(
Yeah, these big projects with lots of c++ threads use tons of memory. Ideally we could find a way to make them build on the existing builders. Failing that I guess we make a few heavybuilders, but then that decreases capacity from other builds and causes us to have to curate a list.
webkitgtk is in this same boat. In fact: https://koji.fedoraproject.org/koji/taskinfo?taskID=128394527 / https://pagure.io/fedora-infrastructure/issue/12377
Log in to comment on this ticket.