I tried to build a new Hyperscale kernel yesterday based on 6.10.4. While a scratch build worked for c9s, it fails for c10s.
I checked one of the kernels that built for CentOS Stream 10 in Stream Koji on CBS, and that build fails in CBS.
It fails with a bizarre error about hmac signing:
kernel.spec:2253: hmac sign the kernel for FIPS kernel.spec:: Creating hmac file: /builddir/build/BUILDROOT/kernel-6.11.0-0.rc3.18.hsx.el10.aarch64/boot/.vmlinuz-6.11.0-0.rc3.18.hsx.el10.aarch64+rt-debug.hmac + ls -l /builddir/build/BUILDROOT/kernel-6.11.0-0.rc3.18.hsx.el10.aarch64/boot/vmlinuz-6.11.0-0.rc3.18.hsx.el10.aarch64+rt-debug -rwxr-xr-x. 1 mockbuild mock 21670136 Aug 15 02:59 /builddir/build/BUILDROOT/kernel-6.11.0-0.rc3.18.hsx.el10.aarch64/boot/vmlinuz-6.11.0-0.rc3.18.hsx.el10.aarch64+rt-debug + cd /builddir/build/BUILDROOT/kernel-6.11.0-0.rc3.18.hsx.el10.aarch64/boot + sha512hmac vmlinuz-6.11.0-0.rc3.18.hsx.el10.aarch64+rt-debug Allocation of hmac(sha512) cipher failed (ret=-111)
This also happens with the Hyperscale kernel SRPM. It's a fairly consistent error that at this point I think is something about CBS now.
Here's the buildsystem installed package diff:
--- stream-installpkgs.log 2024-08-15 09:18:44.266724792 -0400 +++ cbs-installpkgs.log 2024-08-15 09:17:54.653156523 -0400 @@ -5,7 +5,7 @@ alsa-lib-1.2.12-1.el10.aarch64 alternatives-1.30-1.el10.aarch64 annobin-docs-12.55-2.el10.noarch annobin-plugin-gcc-12.55-2.el10.aarch64 -asciidoc-10.2.0-10.el10.noarch +asciidoc-10.2.0-11.el10.noarch audit-libs-4.0-9.el10.aarch64 audit-libs-devel-4.0-9.el10.aarch64 authselect-1.5.0-7.el10.aarch64 @@ -19,16 +19,17 @@ binutils-devel-2.41-45.el10.aarch64 binutils-gold-2.41-45.el10.aarch64 bison-3.8.2-8.el10.aarch64 boost-regex-1.83.0-4.el10.aarch64 +buildsys-macros-el10s-1.0-2.el10.noarch bzip2-1.0.8-19.el10.aarch64 bzip2-libs-1.0.8-19.el10.aarch64 ca-certificates-2023.2.62_v7.0.401-7.el10.noarch cairo-1.18.0-4.el10.aarch64 cairo-gobject-1.18.0-4.el10.aarch64 -centos-gpg-keys-10.0-0.16.el10.noarch -centos-sb-certs-10.0-0.16.el10.noarch -centos-stream-release-10.0-0.16.el10.noarch -centos-stream-repos-10.0-0.16.el10.noarch -centpkg-minimal-2.1.0-4.el10.noarch +centos-gpg-keys-10.0-0.17.el10.noarch +centos-sb-certs-10.0-0.17.el10.noarch +centos-stream-release-10.0-0.17.el10.noarch +centos-stream-repos-10.0-0.17.el10.noarch +centpkg-minimal-2.1.0-5.el10.noarch clang-18.1.8-1.el10.aarch64 clang-libs-18.1.8-1.el10.aarch64 clang-resource-filesystem-18.1.8-1.el10.noarch @@ -76,7 +77,7 @@ expat-2.6.2-1.el10.aarch64 file-5.45-6.el10.aarch64 file-libs-5.45-6.el10.aarch64 filesystem-3.18-9.el10.aarch64 -findutils-1:4.10.0-1.el10.aarch64 +findutils-1:4.10.0-4.el10.aarch64 flex-2.6.4-18.el10.aarch64 fontconfig-2.15.0-6.el10.aarch64 fonts-filesystem-1:2.0.5-16.el10.noarch @@ -134,20 +135,18 @@ javapackages-filesystem-6.2.0-11.el10.no jbig2dec-libs-0.20-5.el10.aarch64 jbigkit-libs-2.1-30.el10.aarch64 json-c-0.17-4.el10.aarch64 -kbd-2.6.4-4.el10.aarch64 -kbd-legacy-2.6.4-4.el10.noarch -kbd-misc-2.6.4-4.el10.noarch +kbd-2.6.4-5.el10.aarch64 +kbd-legacy-2.6.4-5.el10.noarch +kbd-misc-2.6.4-5.el10.noarch kernel-headers-6.11.0-0.rc2.16.el10.aarch64 kernel-rpm-macros-205-24.el10.noarch kernel-srpm-macros-1.0-24.el10.noarch keyutils-libs-1.6.3-4.el10.aarch64 kmod-31-6.el10.aarch64 kmod-libs-31-6.el10.aarch64 -krb5-libs-1.21.3-1.el10.aarch64 -krb5-pkinit-1.21.3-1.el10.aarch64 -krb5-workstation-1.21.3-1.el10.aarch64 +krb5-libs-1.21.3-2.el10.aarch64 lcms2-2.16-5.el10.aarch64 -less-661-1.el10.aarch64 +less-661-2.el10.aarch64 libacl-2.3.2-3.el10.aarch64 libaio-0.3.111-20.el10.aarch64 libarchive-3.7.2-7.el10.aarch64 @@ -189,7 +188,6 @@ libicu-74.2-2.el10.aarch64 libidn2-2.3.7-2.el10.aarch64 libijs-0.35-23.el10.aarch64 libjpeg-turbo-3.0.2-3.el10.aarch64 -libkadm5-1.21.3-1.el10.aarch64 libkcapi-1.5.0-2.el10.aarch64 libkcapi-hasher-1.5.0-2.el10.aarch64 libkcapi-hmaccalc-1.5.0-2.el10.aarch64 @@ -203,18 +201,17 @@ libmpc-1.3.1-6.el10.aarch64 libnghttp2-1.61.0-2.el10.aarch64 libpaper-1:2.1.1-6.el10.aarch64 libpkgconf-2.1.0-2.el10.aarch64 -libpng-2:1.6.40-4.el10.aarch64 +libpng-2:1.6.40-7.el10.aarch64 libpsl-0.21.5-4.el10.aarch64 libpwquality-1.4.5-11.el10.aarch64 librsvg2-2.57.1-7.el10.aarch64 libseccomp-2.5.3-9.el10.aarch64 -libselinux-3.7-2.el10.aarch64 -libselinux-devel-3.7-2.el10.aarch64 -libsemanage-3.7-1.el10.aarch64 -libsepol-3.7-1.el10.aarch64 -libsepol-devel-3.7-1.el10.aarch64 +libselinux-3.7-3.el10.aarch64 +libselinux-devel-3.7-3.el10.aarch64 +libsemanage-3.7-2.el10.aarch64 +libsepol-3.7-2.el10.aarch64 +libsepol-devel-3.7-2.el10.aarch64 libsmartcols-2.40.2-4.el10.aarch64 -libss-1.47.1-2.el10.aarch64 libssh-0.10.6-7.el10.aarch64 libssh-config-0.10.6-7.el10.noarch libstdc++-14.1.1-5.el10.aarch64 @@ -225,8 +222,8 @@ libtiff-4.6.0-3.el10.aarch64 libtool-ltdl-2.4.7-12.el10.aarch64 libtraceevent-1.8.2-5.el10.aarch64 libtraceevent-devel-1.8.2-5.el10.aarch64 -libtracefs-1.8.0-4.el10.aarch64 -libtracefs-devel-1.8.0-4.el10.aarch64 +libtracefs-1.8.0-5.el10.aarch64 +libtracefs-devel-1.8.0-5.el10.aarch64 libubsan-14.1.1-5.el10.aarch64 libunistring-1.1-9.el10.aarch64 libutempter-1.2.1-14.el10.aarch64 @@ -278,7 +275,7 @@ ncurses-c++-libs-6.4-13.20240127.el10.aa ncurses-devel-6.4-13.20240127.el10.aarch64 ncurses-libs-6.4-13.20240127.el10.aarch64 nettle-3.10-1.el10.aarch64 -net-tools-2.0-0.71.20160912git.el10.aarch64 +net-tools-2.0-0.72.20160912git.el10.aarch64 newt-0.52.24-4.el10.aarch64 newt-devel-0.52.24-4.el10.aarch64 npth-1.6-19.el10.aarch64 @@ -408,12 +405,12 @@ popt-devel-1.19-7.el10.aarch64 procps-ng-4.0.4-4.el10.aarch64 publicsuffix-list-dafsa-20240107-4.el10.noarch pyproject-srpm-macros-1.12.0-2.el10.noarch -python3-3.12.4-4.el10.aarch64 +python3-3.12.5-1.el10.aarch64 python3-cffi-1.16.0-5.el10.aarch64 python3-cryptography-41.0.7-2.el10.aarch64 -python3-devel-3.12.4-4.el10.aarch64 +python3-devel-3.12.5-1.el10.aarch64 python3-docutils-0.20.1-4.el10.noarch -python3-libs-3.12.4-4.el10.aarch64 +python3-libs-3.12.5-1.el10.aarch64 python3-packaging-23.2-5.el10.noarch python3-pefile-2023.2.7-8.el10.noarch python3-pip-wheel-23.3.2-3.el10.noarch @@ -431,10 +428,10 @@ qt6-srpm-macros-6.7.1-4.el10.noarch readline-8.2-9.el10.aarch64 redhat-rpm-config-285-1.el10.noarch redhat-text-vf-fonts-4.0.3-12.el10.noarch -rpm-4.19.1.1-2.el10.aarch64 -rpm-build-4.19.1.1-2.el10.aarch64 -rpm-build-libs-4.19.1.1-2.el10.aarch64 -rpm-libs-4.19.1.1-2.el10.aarch64 +rpm-4.19.1.1-3.el10.aarch64 +rpm-build-4.19.1.1-3.el10.aarch64 +rpm-build-libs-4.19.1.1-3.el10.aarch64 +rpm-libs-4.19.1.1-3.el10.aarch64 rpm-sequoia-1.6.0-3.el10.aarch64 rsvg-pixbuf-loader-2.57.1-7.el10.aarch64 rsync-3.3.0-4.el10.aarch64
Metadata Update from @arrfab: - Issue tagged with: cbs, high-gain, high-trouble, investigation
I had a quick look but don't see any obvious difference. Does it work for previous minor kernel ? I see latest one turned some hmac function for fips but don't know if that's related
It fails with 6.10 as well, I tested 6.11 to see if it would fix it, but it doesn't.
sorry, had no time to have a deeper look as have other tasks that need my focus :/ just curious : is that working on different building env ? (copr, local mock, $else) ?
interesting : I just took the src.rpm from Stream koji and submitted a scratch build in that same env and now it fails too in Stream koji : https://kojihub.stream.centos.org/koji/taskinfo?taskID=4595896
So that means something in the buildroot actually doesn't work
Hmm, maybe @scweaver might be able to help us out here? I don't know what's broken.
This builds locally with a local mock build with a config generated from the CBS target using the following command:
cbs mock-config --target hyperscale10s-packages-experimental-el10s --arch x86_64 --name hyperscale10s-exp-koji > hyperscale10s-exp-koji-mock.cfg
Just wanted to mention that there is a known issue with sha512hmac on CBS for rhel8 build roots due to certain combination of kernel version running on build host and libkcapi version of buildroot, see 1. While the error message is different, the cause might be similar.
Do we have any CBS builders on rhel9/c9s yet? I've looked at the logs and mostly seen RHEL 8.9 so far...
All koji builders, for CBS and Stream are still on RHEL 8 ... I just wanted to verify something so I just reconfigured hyperscale10s-packages-experimental-el10s-build buildtag with mock.new_chroot=0
mock.new_chroot=0
I gave it a quick try for a --scratch build and only aarch64 : https://cbs.centos.org/koji/taskinfo?taskID=4186395 and it seems it passed the hmac operation already : https://cbs.centos.org/kojifiles/work/tasks/6397/4186397/build.log :
kernel.spec:: Creating hmac file: /builddir/build/BUILDROOT/kernel-6.11.0-0.rc5.22.hsx.el10.aarch64/boot/.vmlinuz-6.11.0-0.rc5.22.hsx.el10.aarch64+rt-debug.hmac + ls -l /builddir/build/BUILDROOT/kernel-6.11.0-0.rc5.22.hsx.el10.aarch64/boot/vmlinuz-6.11.0-0.rc5.22.hsx.el10.aarch64+rt-debug -rwxr-xr-x. 1 mockbuild mock 21719288 Aug 28 14:08 /builddir/build/BUILDROOT/kernel-6.11.0-0.rc5.22.hsx.el10.aarch64/boot/vmlinuz-6.11.0-0.rc5.22.hsx.el10.aarch64+rt-debug + cd /builddir/build/BUILDROOT/kernel-6.11.0-0.rc5.22.hsx.el10.aarch64/boot + sha512hmac vmlinuz-6.11.0-0.rc5.22.hsx.el10.aarch64+rt-debug + cp /builddir/build/BUILDROOT/kernel-6.11.0-0.rc5.22.hsx.el10.aarch64/boot/.vmlinuz-6.11.0-0.rc5.22.hsx.el10.aarch64+rt-debug.hmac /builddir/build/BUILDROOT/kernel-6.11.0-0.rc5.22.hsx.el10.aarch64/lib/modules/6.11.0-0.rc5.22.hsx.el10.aarch64+rt-debug/.vmlinuz.hmac
@ngompa : just to confirm that thing, can you try resubmitting some kernel build jobs for c10s ? But even if that works, I'm really surprised as c10s Stream koji kernels are built with systemd-nspwan too so wondering why/how it would still work there ...
Metadata Update from @arrfab: - Issue assigned to arrfab
New scratch build task submitted: https://cbs.centos.org/koji/taskinfo?taskID=4187543
... and it worked. While I'm happy that it works for you, it doesn't make any sense about why the "classical" vs "systemd-nspawn" chroot would make such an impact. I read that there was a potential issue with libkcapi through a container and not able to reach host kernel for crypto operations, but then wondering why and how kernels for CentOS Stream 10 are built, as they are built through nspawn and on top of rhel8 hosts.
https://kojihub.stream.centos.org/koji/taginfo?tagID=1563 => mock.new_chroot 1 Also visible in any build.log from latest kernels built there : Using nspawn with args ['
mock.new_chroot 1
Using nspawn with args ['
@ngompa : Do you consider it's now working for you ? The only reason why we enforced nspawn isolation for c10s build tags in CBS was that it was enforced at the Stream builders side, and so being as close as original builders was a good idea.
The fact that it works makes me kind of unhappy, because it makes no sense. :anger:
I would rather not have to switch our tags to simple chroot. :frowning:
ack .. so do you want me to revert to mock.new_chroot=1 like for Stream builders ? BTW, interesting thing : as there will be stream maintenance this Friday (announced also on devel list) , the stream builders will be rebooting on exact same kernel running for cbs builders so we'll see if that then breaks or continue to work. Difficult to spot the reason why on similar setup we have different results
mock.new_chroot=1
ack .. so do you want me to revert to mock.new_chroot=1 like for Stream builders ?
Yes please.
BTW, interesting thing : as there will be stream maintenance this Friday (announced also on devel list) , the stream builders will be rebooting on exact same kernel running for cbs builders so we'll see if that then breaks or continue to work. Difficult to spot the reason why on similar setup we have different results
@scweaver said in the CentOS Kernel Matrix room that there were was some kind of specially patched things on the builders that are in the pesign channel. Maybe worth finding out what was done there.
Well, as I manage the CentOS Stream infra, I can tell you for sure that there is no patched kernel running there :) :
https://kojihub.stream.centos.org/kojifiles/work/tasks/4620/4594620/root.log => DEBUG buildroot.py:802: kernel version == 4.18.0-513.5.1.el8_9.aarch64
DEBUG buildroot.py:802: kernel version == 4.18.0-513.5.1.el8_9.aarch64
https://kojihub.stream.centos.org/kojifiles/work/tasks/4623/4594623/root.log => DEBUG buildroot.py:802: kernel version == 4.18.0-513.18.1.el8_9.x86_64
DEBUG buildroot.py:802: kernel version == 4.18.0-513.18.1.el8_9.x86_64
and so on ... The only thing that will happen this Friday is that they'll all be rebooted on latest RHEL8 kernel but we have zero patched kernel running on centos stream infra
Are any of them on RHEL 8.10 instead of RHEL 8.9?
updated on daily basis but not yet rebooted , which will happen during the maintenance window tomorrow, reason why I'm curious about the impact appearing or not, and so then triggering internal attention/focus to solve this
Not to muddy the waters any further because why it's working in some build env and not others isn't at all clear but it was proposed that rhel8 might need 91b05a7e7d803 ('crypto: user - make NETLINK_CRYPTO work inside netns') for the hmac signing to work inside an nspawn container. So it was proposed to update the builders to rhel9 or patch rhel8. In the mean time it was suggested that for stream koji builds one could use the c10s-candidate-nspwanless build target but of course that might not be available in cbs.
91b05a7e7d803 ('crypto: user - make NETLINK_CRYPTO work inside netns')
c10s-candidate-nspwanless
it is, and that was what was tested : switching back to classical chroot. BTW, this is how it was configured for Stream 9 and SIGs for cbs. It was just updated for c10s recently, and also I learned that RHEL is also using classical chroot (no systemd-nspawn) ...
@ngompa : as reverting to mock.new_chroot=0 works for you, do you still want to enforce nspawn ? Tempted to just close this ticket as there is a solution that can work for now, it's just a matter of accepting it or not
.. other thing we can temporary try : still use nspawn but on specific channel (the image one) where the x86_64/aarch64 builders are running RHEL8 but with higher kernel provided by the kmod SIG (kernel 6.6.x). Tempted to give it a try ?
image
That could work.
quickly wrote a hub policy to ensure it would landing in the image channel and submitted a x86_64 only scratch build : https://cbs.centos.org/koji/taskinfo?taskID=4196595 Let's see
it worked and just fine-tuned the policy and tested it again for now both arches : https://cbs.centos.org/koji/taskinfo?taskID=4196755
and it worked for both x86_64 and aarch64. @ngompa : would that work for you then ? still using nspawn isolation and just 'abusing' the hosts in the image channel for the hyperscale10s* build tag (only for kernel that is)
Metadata Update from @arrfab: - Issue priority set to: Waiting on Reporter (was: Needs Review)
@pjgeorg ^ don't know if you plan on providing kernels for c10s (or kmods) but just to let you know that it would be the workaround if you want to
and it worked for both x86_64 and aarch64. would that work for you then ? still using nspawn isolation and just 'abusing' the hosts in the image channel for the hyperscale10s* build tag (only for kernel that is)
Works for me.
adapted/tuned kojihub policy now definitely in place (git/ansible) so let's close this request
Metadata Update from @arrfab: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.