Learn more about these different git repos.
Other Git URLs
https://bugzilla.redhat.com/show_bug.cgi?id=2220650 reports problems with the i686 builders (buildvm-x86-04.iad2.fedoraproject.org in particular), and it looks like it's because the build uses 64-bit inode numbers. If the file system is XFS, it can be mounted with the inode32 option to address this issue, no mkfs is needed.
inode32
mkfs
The filesystem is not XFS, it's btrfs currently.
Can we change it to ext4 or XFS with the mount option? I don't see a way to use 32-bit inode numbers with btrfs. :disappointed:
Metadata Update from @phsmoura: - Issue assigned to kevin - Issue tagged with: high-gain, high-trouble, ops
Sigh. We can... but then the x86_64 builders are not like all the rest and are special snowflakes again and are using a special option that will fix things for our builds, but break things for people who build elsewhere. ;(
Any ideas why this has become an issue? I suppose because builders got glibc-2.37-4.fc38 and it changed something?
Any other options you can think of? Can we just fix libcap? or I guess thats not enough because there's other unknown ones out there? (but all of them I have seen in the last few weeks have been libcap and I don't think there's any more in the base buildroot)
I'm afraid than not only i686 builders are affected: ppc64le: https://koji.fedoraproject.org/koji/taskinfo?taskID=103200367 s390x: https://koji.fedoraproject.org/koji/taskinfo?taskID=103200374
Uh-oh, then it has to be something else. Any chance to get an strace to confirm that it's the lstat that is failing?
strace
lstat
it was the buildSRPMFromSCMsubtask that failed -- https://koji.fedoraproject.org/koji/taskinfo?taskID=103200517 buildvm-x86-15.iad2.fedoraproject.org Architecture: i686
same, the buildSRPMFromSCM subtask failed -- https://koji.fedoraproject.org/koji/taskinfo?taskID=103200518 buildvm-x86-04.iad2.fedoraproject.org Architecture: i686
In these cases, the SRPM build was also running on i686 despite the main task being run on ppc64le and s390x: https://kojipkgs.fedoraproject.org//work/tasks/517/103200517/hw_info.log https://kojipkgs.fedoraproject.org//work/tasks/518/103200518/hw_info.log
Is this now getting worse somehow?
I'm trying to get four chain-builds over the finish line, but almost all of them now hit this issue at some point and fail. Of my 11 attempts, 9 failed on this i686 issue, two are fingers crossed still running ... and I guess I'll have to resubmit two other chain-builds until they succeed? But they've now already failed twice in a row. :weary:
I'm afraid than not only i686 builders are affected: ppc64le: https://koji.fedoraproject.org/koji/taskinfo?taskID=103200367 s390x: https://koji.fedoraproject.org/koji/taskinfo?taskID=103200374 In these cases, the SRPM build was also running on i686 despite the main task being run on ppc64le and s390x: https://kojipkgs.fedoraproject.org//work/tasks/517/103200517/hw_info.log https://kojipkgs.fedoraproject.org//work/tasks/518/103200518/hw_info.log
Eh, thanks. So it's likely the known inode number range issue.
Unfortunately, it's not really something we can fix in userspace at reasonable cost.
Yeah, but what I don't understand is why this suddenly started happening. Did something around this change in f38 updates glibc?
I can reinstall all the x86_64 builders (and in any case we need to fix it before the mass rebuild next week for sure), but if we do we are going to have to make some noise about this because anyone using mock on their fedora system to build i686 rpms will be hitting it and reporting bugs on it right?
There could be several places that this changed: 1. the filesystem itself could now be using large numbers when it wasn't before. 2. the kernel could have changed in what it is giving back on certain calls. 3. glibc or similar libraries have changed.
With this not happening on every i686 build or tooling.. I would look at 1 or 2. How 'ugly' would it be to set up a dedicated system in staging and label it as the i686 build host versus any x86_64. Then try different tooling to make it not-break consistently: older kernel older filesystem tools older glibc
It is probably a 'why do you even come up with these things.. Stephen' idea as koji and other parts of the build system probably all need fixes to make that happen.
So, glibc didn't actually change here.
I'm looking at other updates now....
So, a few things in the time I had to look at this today:
I can seem to duplicate it locally sometimes in mock, but... it doesn't fail, it just doesn't set the caps on those files. I am not sure why...
<mock-chroot> sh-5.2# rpm -V shadow-utils ........P /usr/bin/newgidmap ........P /usr/bin/newuidmap
Here's the package list of changes on the builders:
appstream-data-38-4.fc38.noarch Sun 25 Jun 2023 04:41:10 PM UTC atk-2.48.3-1.fc38.x86_64 Sun 25 Jun 2023 04:39:37 PM UTC at-spi2-atk-2.48.3-1.fc38.x86_64 Sun 25 Jun 2023 04:41:03 PM UTC at-spi2-core-2.48.3-1.fc38.x86_64 Sun 25 Jun 2023 04:39:37 PM UTC bind-libs-9.18.16-1.fc38.x86_64 Sun 25 Jun 2023 04:40:41 PM UTC bind-license-9.18.16-1.fc38.noarch Sun 25 Jun 2023 04:40:41 PM UTC bind-utils-9.18.16-1.fc38.x86_64 Sun 25 Jun 2023 04:40:41 PM UTC btrfs-progs-6.3.2-1.fc38.x86_64 Sun 25 Jun 2023 04:40:34 PM UTC corosynclib-3.1.7-3.fc38.x86_64 Sun 25 Jun 2023 04:41:05 PM UTC distribution-gpg-keys-1.89-1.fc38.noarch Sun 25 Jun 2023 04:40:40 PM UTC dnf-4.16.1-1.fc38.noarch Sun 25 Jun 2023 04:40:40 PM UTC dnf-automatic-4.16.1-1.fc38.noarch Sun 25 Jun 2023 04:40:45 PM UTC dnf-data-4.16.1-1.fc38.noarch Sun 25 Jun 2023 04:40:39 PM UTC dnsmasq-2.89-5.fc38.x86_64 Sun 25 Jun 2023 04:41:00 PM UTC freeipa-client-4.10.1-4.fc38.x86_64 Sun 25 Jun 2023 04:40:43 PM UTC freeipa-client-common-4.10.1-4.fc38.noarch Sun 25 Jun 2023 04:39:42 PM UTC freeipa-common-4.10.1-4.fc38.noarch Sun 25 Jun 2023 04:40:02 PM UTC freeipa-selinux-4.10.1-4.fc38.noarch Sun 25 Jun 2023 04:39:50 PM UTC fwupd-1.9.2-1.fc38.x86_64 Sun 25 Jun 2023 04:39:41 PM UTC fwupd-plugin-flashrom-1.9.2-1.fc38.x86_64 Sun 25 Jun 2023 04:39:40 PM UTC fwupd-plugin-modem-manager-1.9.2-1.fc38.x86_64 Sun 25 Jun 2023 04:39:40 PM UTC fwupd-plugin-uefi-capsule-data-1.9.2-1.fc38.x86_64 Sun 25 Jun 2023 04:39:40 PM UTC gdb-minimal-13.2-1.fc38.x86_64 Sun 25 Jun 2023 04:41:04 PM UTC git-core-2.41.0-1.fc38.x86_64 Sun 25 Jun 2023 04:40:39 PM UTC gstreamer1-1.22.4-1.fc38.x86_64 Sun 25 Jun 2023 04:41:05 PM UTC hwdata-0.371-1.fc38.noarch Sun 25 Jun 2023 04:41:05 PM UTC ipcalc-1.0.3-1.fc38.x86_64 Sun 25 Jun 2023 04:41:05 PM UTC kernel-core-6.3.8-200.fc38.x86_64 Sun 25 Jun 2023 04:39:49 PM UTC kernel-modules-6.3.8-200.fc38.x86_64 Sun 25 Jun 2023 04:40:51 PM UTC kernel-modules-core-6.3.8-200.fc38.x86_64 Sun 25 Jun 2023 04:39:48 PM UTC koji-1.33.0-1.fc38.2.noarch Sun 25 Jun 2023 04:39:49 PM UTC koji-builder-1.33.0-1.fc38.2.noarch Sun 25 Jun 2023 04:40:41 PM UTC koji-builder-plugins-1.33.0-1.fc38.2.noarch Sun 25 Jun 2023 04:40:45 PM UTC libatomic-13.1.1-4.fc38.x86_64 Sun 25 Jun 2023 04:41:05 PM UTC libestr-0.1.11-6.fc38.x86_64 Sun 25 Jun 2023 04:40:38 PM UTC libfastjson-1.2304.0-1.fc38.x86_64 Sun 25 Jun 2023 04:40:38 PM UTC libgcc-13.1.1-4.fc38.i686 Sun 25 Jun 2023 04:41:10 PM UTC libgcc-13.1.1-4.fc38.x86_64 Sun 25 Jun 2023 04:39:04 PM UTC libgexiv2-0.14.1-1.fc38.x86_64 Sun 25 Jun 2023 04:41:04 PM UTC libgomp-13.1.1-4.fc38.x86_64 Sun 25 Jun 2023 04:41:05 PM UTC libmd-1.1.0-1.fc38.x86_64 Sun 25 Jun 2023 04:41:05 PM UTC librsvg2-2.56.1-1.fc38.x86_64 Sun 25 Jun 2023 04:41:04 PM UTC libstdc++-13.1.1-4.fc38.x86_64 Sun 25 Jun 2023 04:39:04 PM UTC libtracker-sparql-3.5.3-1.fc38.x86_64 Sun 25 Jun 2023 04:40:34 PM UTC liburing-2.4-2.fc38.x86_64 Sun 25 Jun 2023 04:41:05 PM UTC libwbclient-4.18.3-3.fc38.x86_64 Sun 25 Jun 2023 04:39:41 PM UTC libxcrypt-4.4.35-1.fc38.x86_64 Sun 25 Jun 2023 04:39:38 PM UTC libxcrypt-compat-4.4.35-1.fc38.x86_64 Sun 25 Jun 2023 04:41:03 PM UTC llvm-libs-16.0.5-1.fc38.x86_64 Sun 25 Jun 2023 04:39:06 PM UTC lm_sensors-libs-3.6.0-13.fc38.x86_64 Sun 25 Jun 2023 04:40:41 PM UTC mesa-dri-drivers-23.1.2-1.fc38.x86_64 Sun 25 Jun 2023 04:40:42 PM UTC mesa-filesystem-23.1.2-1.fc38.x86_64 Sun 25 Jun 2023 04:39:38 PM UTC mesa-libEGL-23.1.2-1.fc38.x86_64 Sun 25 Jun 2023 04:40:41 PM UTC mesa-libgbm-23.1.2-1.fc38.x86_64 Sun 25 Jun 2023 04:40:41 PM UTC mesa-libGL-23.1.2-1.fc38.x86_64 Sun 25 Jun 2023 04:40:42 PM UTC mesa-libglapi-23.1.2-1.fc38.x86_64 Sun 25 Jun 2023 04:40:41 PM UTC mesa-va-drivers-23.1.2-1.fc38.x86_64 Sun 25 Jun 2023 04:39:49 PM UTC mesa-vulkan-drivers-23.1.2-1.fc38.x86_64 Sun 25 Jun 2023 04:41:04 PM UTC mock-4.1-1.fc38.noarch Sun 25 Jun 2023 04:40:41 PM UTC mock-core-configs-38.6-1.fc38.noarch Sun 25 Jun 2023 04:40:40 PM UTC mock-filesystem-4.1-1.fc38.noarch Sun 25 Jun 2023 04:39:37 PM UTC mozjs102-102.12.0-1.fc38.x86_64 Sun 25 Jun 2023 04:41:04 PM UTC nspr-4.35.0-7.fc38.x86_64 Sun 25 Jun 2023 04:39:04 PM UTC nss-3.90.0-1.fc38.x86_64 Sun 25 Jun 2023 04:40:34 PM UTC nss-softokn-3.90.0-1.fc38.x86_64 Sun 25 Jun 2023 04:40:34 PM UTC nss-softokn-freebl-3.90.0-1.fc38.x86_64 Sun 25 Jun 2023 04:40:34 PM UTC nss-sysinit-3.90.0-1.fc38.x86_64 Sun 25 Jun 2023 04:40:34 PM UTC nss-tools-3.90.0-1.fc38.x86_64 Sun 25 Jun 2023 04:40:34 PM UTC nss-util-3.90.0-1.fc38.x86_64 Sun 25 Jun 2023 04:39:04 PM UTC ostree-libs-2023.4-1.fc38.x86_64 Sun 25 Jun 2023 04:41:04 PM UTC passt-0^20230603.g429e1a7-1.fc38.x86_64 Sun 25 Jun 2023 04:40:05 PM UTC passt-selinux-0^20230603.g429e1a7-1.fc38.noarch Sun 25 Jun 2023 04:40:05 PM UTC perl-HTTP-Tiny-0.084-1.fc38.noarch Sun 25 Jun 2023 04:41:05 PM UTC postfix-3.7.4-2.fc38.x86_64 Sun 25 Jun 2023 04:41:01 PM UTC publicsuffix-list-dafsa-20230614-1.fc38.noarch Sun 25 Jun 2023 04:41:05 PM UTC pyproject-srpm-macros-1.9.0-1.fc38.noarch Sun 25 Jun 2023 04:41:05 PM UTC python3-deprecated-1.2.14-1.fc38.noarch Sun 25 Jun 2023 04:41:05 PM UTC python3-dnf-4.16.1-1.fc38.noarch Sun 25 Jun 2023 04:40:40 PM UTC python3-ipaclient-4.10.1-4.fc38.noarch Sun 25 Jun 2023 04:40:04 PM UTC python3-ipalib-4.10.1-4.fc38.noarch Sun 25 Jun 2023 04:40:03 PM UTC python3-koji-1.33.0-1.fc38.2.noarch Sun 25 Jun 2023 04:39:37 PM UTC python3-pycurl-7.45.2-3.fc38.x86_64 Sun 25 Jun 2023 04:41:05 PM UTC python3-urllib3-1.26.15-1.fc38.noarch Sun 25 Jun 2023 04:40:35 PM UTC python3-urllib3+socks-1.26.15-1.fc38.noarch Sun 25 Jun 2023 04:40:35 PM UTC qt5-srpm-macros-5.15.10-1.fc38.noarch Sun 25 Jun 2023 04:41:04 PM UTC rsyslog-8.2306.0-1.fc38.x86_64 Sun 25 Jun 2023 04:40:39 PM UTC rsyslog-logrotate-8.2306.0-1.fc38.x86_64 Sun 25 Jun 2023 04:40:38 PM UTC samba-client-libs-4.18.3-3.fc38.x86_64 Sun 25 Jun 2023 04:39:42 PM UTC samba-common-4.18.3-3.fc38.noarch Sun 25 Jun 2023 04:39:41 PM UTC samba-common-libs-4.18.3-3.fc38.x86_64 Sun 25 Jun 2023 04:39:42 PM UTC selinux-policy-38.17-1.fc38.noarch Sun 25 Jun 2023 04:39:06 PM UTC selinux-policy-targeted-38.17-1.fc38.noarch Sun 25 Jun 2023 04:39:21 PM UTC sudo-1.9.13-2.p2.fc38.x86_64 Sun 25 Jun 2023 04:39:37 PM UTC sudo-python-plugin-1.9.13-2.p2.fc38.x86_64 Sun 25 Jun 2023 04:39:37 PM UTC tracker-3.5.3-1.fc38.x86_64 Sun 25 Jun 2023 04:40:35 PM UTC vim-common-9.0.1649-1.fc38.x86_64 Sun 25 Jun 2023 04:40:38 PM UTC vim-data-9.0.1649-1.fc38.noarch Sun 25 Jun 2023 04:39:37 PM UTC vim-enhanced-9.0.1649-1.fc38.x86_64 Sun 25 Jun 2023 04:40:45 PM UTC vim-filesystem-9.0.1649-1.fc38.noarch Sun 25 Jun 2023 04:40:35 PM UTC vim-minimal-9.0.1649-1.fc38.x86_64 Sun 25 Jun 2023 04:41:03 PM UTC vte291-0.72.2-1.fc38.x86_64 Sun 25 Jun 2023 04:40:45 PM UTC vte-profile-0.72.2-1.fc38.x86_64 Sun 25 Jun 2023 04:40:35 PM UTC xxd-9.0.1649-1.fc38.x86_64 Sun 25 Jun 2023 04:40:35 PM UTC yum-4.16.1-1.fc38.noarch Sun 25 Jun 2023 04:40:40 PM UTC zfs-fuse-0.7.2.2-27.fc38.x86_64 Sun 25 Jun 2023 04:41:04 PM UTC
I wonder... how crazy would it be to disable caps ? ie, --tsflags=nocaps ? Probibly pretty crazy. ;(
We could also rebuild libcap with different flags.
But the only change will be that something else fails later, typically that's going to be update-alternatives from chkconfig. Then we fixed that, and it goes on and on and on. Not a productive use of our time.
update-alternatives
chkconfig
I looked at the kernel sources and it seems that this is a time bomb in the btrfs file system. I think it assigns inode numbers sequentially using a 64-bit counter:
int btrfs_get_free_objectid(struct btrfs_root *root, u64 *objectid) { int ret; mutex_lock(&root->objectid_mutex); if (unlikely(root->free_objectid >= BTRFS_LAST_FREE_OBJECTID)) { btrfs_warn(root->fs_info, "the objectid of root %llu reaches its highest value", root->root_key.objectid); ret = -ENOSPC; goto out; } *objectid = root->free_objectid++; ret = 0; out: mutex_unlock(&root->objectid_mutex); return ret; }
Currently, the only remedy seems to be to recreate the file system after two billion files have been created, to reset the counter. I'll bring it to the devel list. The on-disk btrfs data structures probably allow reusing inode numbers in gaps, but there could be a performance impact.
It definitely seems like it got worse today. IMO this should be considered a blocker to the F39 mass rebuild.
Yep. I consider it such.
I've filed a ticket about extending mock to create/destroy subvolumes instead of directories for chroots: https://github.com/rpm-software-management/mock/issues/1146
That may also require changes in koji for failed build chroots... currently koji keeps them around for a configurable time and then deletes them, so it would have to know to delete the subvolume instead.
If you need to fight the fire, you can try to use LVM https://rpm-software-management.github.io/mock/Plugin-LvmRoot That will use different filesystems for each chroot too. When correctly configured, it can even improve the performance.
All the buildvm-x86* builders are reinstalled now.
Please let me know if you see the build failure again and on what exact builder it happened.
(We still need a longer term solution, but I just wanted to get past the anoying failures now and the mass rebuild before making any radical changes)
Login to comment on this ticket.