pungi.global.log
[LIVE_MEDIA ] [ERROR ] [FAIL] Live media (variant Labs, arch *, subvariant Design_suite) failed, but going on anyway. [LIVE_MEDIA ] [ERROR ] Live media task failed: 68314759. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20210520.n.0/logs/x86_64/livemedia-Labs-Design_suite.x86_64.log for more details.
Variant: Labs, subvariant: Design_suite task failed. Pinging maintainers: @luya
[LIVE_MEDIA ] [ERROR ] [FAIL] Live media (variant Labs, arch *, subvariant Comp_Neuro) failed, but going on anyway. [LIVE_MEDIA ] [ERROR ] Live media task failed: 68314756. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20210520.n.0/logs/x86_64/livemedia-Labs-Comp_Neuro.x86_64.log for more details.
Variant: Labs, subvariant: Comp_Neuro task failed. Pinging maintainers: @dan1mal @ankursinha @neuro-sig @bt0dotninja
[LIVE_MEDIA ] [ERROR ] [FAIL] Live media (variant Labs, arch *, subvariant Scientific_KDE) failed, but going on anyway. [LIVE_MEDIA ] [ERROR ] Live media task failed: 68314765. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20210520.n.0/logs/x86_64/livemedia-Labs-Scientific_KDE.x86_64.log for more details.
Variant: Labs, subvariant: Scientific_KDE task failed. Pinging maintainers: @scitech
[LIVE_MEDIA ] [ERROR ] [FAIL] Live media (variant Labs, arch *, subvariant Astronomy_KDE) failed, but going on anyway. [LIVE_MEDIA ] [ERROR ] Live media task failed: 68314755. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20210520.n.0/logs/x86_64/livemedia-Labs-Astronomy_KDE.x86_64.log for more details.
Variant: Labs, subvariant: Astronomy_KDE task failed. Pinging maintainers: @lupinix
[LIVE_MEDIA ] [ERROR ] [FAIL] Live media (variant Labs, arch *, subvariant Robotics) failed, but going on anyway. [LIVE_MEDIA ] [ERROR ] Live media task failed: 68314771. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20210520.n.0/logs/x86_64/livemedia-Labs-Robotics.x86_64.log for more details.
Variant: Labs, subvariant: Robotics task failed. Pinging maintainers: @robotics-sig @rmattes
[LIVE_MEDIA ] [ERROR ] [FAIL] Live media (variant Spins, arch *, subvariant SoaS) failed, but going on anyway. [LIVE_MEDIA ] [ERROR ] Live media task failed: 68314788. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20210520.n.0/logs/x86_64/livemedia-Spins-SoaS.x86_64.log for more details.
Variant: Spins, subvariant: SoaS task failed. Pinging maintainers: @chimosky @aperezbios
[LIVE_MEDIA ] [ERROR ] [FAIL] Live media (variant Spins, arch *, subvariant Mate) failed, but going on anyway. [LIVE_MEDIA ] [ERROR ] Live media task failed: 68314798. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20210520.n.0/logs/x86_64/livemedia-Spins-Mate.x86_64.log for more details.
[IMAGE_BUILD ] [ERROR ] [FAIL] Image build (variant Cloud, arch *, subvariant Cloud_Base) failed, but going on anyway. [IMAGE_BUILD ] [ERROR ] ImageBuild task failed: 68314768. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20210520.n.0/logs/x86_64/imagebuild-Cloud-Cloud_Base-tar-gz.x86_64.log for more details.
[IMAGE_BUILD ] [ERROR ] [FAIL] Image build (variant Cloud, arch *, subvariant Cloud_Base) failed, but going on anyway. [IMAGE_BUILD ] [ERROR ] ImageBuild task failed: 68314780. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20210520.n.0/logs/x86_64/imagebuild-Cloud-Cloud_Base-vagrant-libvirt-vagrant-virtualbox.x86_64.log for more details.
[IMAGE_BUILD ] [ERROR ] [FAIL] Image build (variant Labs, arch *, subvariant Python_Classroom) failed, but going on anyway. [IMAGE_BUILD ] [ERROR ] ImageBuild task failed: 68314814. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20210520.n.0/logs/x86_64/imagebuild-Labs-Python_Classroom-vagrant-libvirt-vagrant-virtualbox.x86_64.log for more details.
Variant: Labs, subvariant: Python_Classroom task failed. Pinging maintainers: @python-sig @churchyard
[IMAGE_BUILD ] [ERROR ] [FAIL] Image build (variant Labs, arch *, subvariant Scientific) failed, but going on anyway. [IMAGE_BUILD ] [ERROR ] ImageBuild task failed: 68314824. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20210520.n.0/logs/x86_64/imagebuild-Labs-Scientific-vagrant-libvirt-vagrant-virtualbox.x86_64.log for more details.
Variant: Labs, subvariant: Scientific task failed. Pinging maintainers: @scitech
[IMAGE_BUILD ] [ERROR ] [FAIL] Image build (variant Server, arch *, subvariant Server) failed, but going on anyway. [IMAGE_BUILD ] [ERROR ] ImageBuild task failed: 68314915. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20210520.n.0/logs/aarch64-armhfp/imagebuild-Server-Server-raw-xz.aarch64-armhfp.log for more details.
[IMAGE_BUILD ] [ERROR ] [FAIL] Image build (variant Container, arch *, subvariant Container_Minimal_Base) failed, but going on anyway. [IMAGE_BUILD ] [ERROR ] ImageBuild task failed: 68314802. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20210520.n.0/logs/aarch64-armhfp-ppc64le-s390x-x86_64/imagebuild-Container-Container_Minimal_Base-docker.aarch64-armhfp-ppc64le-s390x-x86_64.log for more details.
[IMAGE_BUILD ] [ERROR ] [FAIL] Image build (variant Container, arch *, subvariant Container_Base) failed, but going on anyway. [IMAGE_BUILD ] [ERROR ] ImageBuild task failed: 68314792. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20210520.n.0/logs/aarch64-armhfp-ppc64le-s390x-x86_64/imagebuild-Container-Container_Base-docker.aarch64-armhfp-ppc64le-s390x-x86_64.log for more details.
[IMAGE_BUILD ] [ERROR ] [FAIL] Image build (variant Labs, arch *, subvariant Python_Classroom) failed, but going on anyway. [IMAGE_BUILD ] [ERROR ] ImageBuild task failed: 68314887. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20210520.n.0/logs/aarch64-armhfp/imagebuild-Labs-Python_Classroom-raw-xz.aarch64-armhfp.log for more details.
[IMAGE_BUILD ] [ERROR ] [FAIL] Image build (variant Spins, arch *, subvariant KDE) failed, but going on anyway. [IMAGE_BUILD ] [ERROR ] ImageBuild task failed: 68314933. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20210520.n.0/logs/aarch64-armhfp/imagebuild-Spins-KDE-raw-xz.aarch64-armhfp.log for more details.
Variant: Spins, subvariant: KDE task failed. Pinging maintainers: @rdieter @svahl
[IMAGE_BUILD ] [ERROR ] [FAIL] Image build (variant Spins, arch *, subvariant SoaS) failed, but going on anyway. [IMAGE_BUILD ] [ERROR ] ImageBuild task failed: 68314945. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20210520.n.0/logs/aarch64-armhfp/imagebuild-Spins-SoaS-raw-xz.aarch64-armhfp.log for more details.
[IMAGE_BUILD ] [ERROR ] [FAIL] Image build (variant Spins, arch *, subvariant Xfce) failed, but going on anyway. [IMAGE_BUILD ] [ERROR ] ImageBuild task failed: 68314950. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20210520.n.0/logs/aarch64-armhfp/imagebuild-Spins-Xfce-raw-xz.aarch64-armhfp.log for more details.
Variant: Spins, subvariant: Xfce task failed. Pinging maintainers: @nonamedotc @kevin @maxamillion
[ERROR ] Compose run failed: ImageBuild task failed: 68314758. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20210520.n.0/logs/aarch64-ppc64le-s390x-x86_64/imagebuild-Cloud-Cloud_Base-qcow2-raw-xz.aarch64-ppc64le-s390x-x86_64.log for more details.
Compose Total time: 4:52:41 Compose phase INIT time: 0:00:25. Compose phase PKGSET time: 0:24:37. Compose phase WEAVER: FAILED. Compose phase BUILDINSTALL time: 1:52:24. Compose phase GATHER time: 2:09:21. Compose phase OSTREE time: 0:40:02. Compose phase OSTREE_INSTALLER time: 1:09:37. Compose phase CREATEREPO time: 0:14:35. Compose phase REPOCLOSURE time: 0:02:07. Compose phase CREATEISO time: 0:10:22. Compose phase IMAGE_BUILD: FAILED. Compose phase LIVE_MEDIA time: 0:37:10.
So we're not sure what exactly the issue is here, but we are failing in early boot during image builds right after a major dracut bump landed, so that's the obvious suspect (the first failed compose was 20210518.n.0, the first compose after that build was tagged). @harald @johannbg I think the new dracut build has broken this.
I dont see how... "2021-05-20 08:12:15,043 INFO pylorax: The installation was stopped due to an error which occurred while running in non-interactive cmdline mode. Since there cannot be any questions in cmdline mode, edit your kickstart file and retry installation. 2021-05-20 08:12:15,043 INFO pylorax: The exact error message is: 2021-05-20 08:12:15,043 INFO pylorax: 2021-05-20 08:12:15,043 INFO pylorax: The following mandatory spokes are not completed: 2021-05-20 08:12:15,043 INFO pylorax: Software selection. 2021-05-20 08:12:15,043 INFO pylorax: 2021-05-20 08:12:15,044 INFO pylorax: The installer will now terminate. 2021-05-20 08:15:16,734 ERROR pylorax: Running anaconda failed: process '['unshare', '--pid', '--kill-child', '--mount', '--propagation', 'unchanged', 'anaconda', '--kickstart', '/chroot_tmpdir/koji-image-f35-build-68314785.ks', '--cmdline', '--dirinstall', '--remotelog', '127.0.0.1:33501']' exited with status 1 2021-05-20 08:15:16,734 INFO livemedia-creator: Shutting down log processing 2021-05-20 08:15:16,767 ERROR pylorax: Install failed: novirt_install failed 2021-05-20 08:15:16,767 INFO pylorax: Removing bad disk image 2021-05-20 08:15:16,769 ERROR pylorax: ERROR: Image creation failed: novirt_install failed 2021-05-20 08:15:16,769 ERROR livemedia-creator: Image creation failed: novirt_install failed"
has anything to do with dracut...
you're not looking at the right failure. That was non-fatal. The fatal one is the one where it says "Compose run failed because:" - this one.
@kevin figured out that it seems to be because /etc/pki is completely missing from the initramfs environment. So curl fails trying to download stage2 via https because it can't load the CA certificates.
You can check this for yourself, btw - grab the Everything boot ISO from a failed compose, e.g. here, and boot with rd.break, and you can see that /etc/pki doesn't exist. Do the same thing with an earlier ISO, e.g. the F34 final release one, and you will find that /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem and /etc/pki/tls/certs/ca-bundle.crt are there.
rd.break
/etc/pki
/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
/etc/pki/tls/certs/ca-bundle.crt
Hmm that's probably url-lib module then and the shell check fixes in Mars might have broken it then.
https://github.com/dracutdevs/dracut/commit/8e84fa7262d5d3273684473538d94a008b5f583e#diff-539f0eeecd578ef02978c8bc2a3368171c855907c1e821749b65aa6a819e32ad
You could isolated it by simply replacing the url-lib module with the same module from an older dracut release ( v53 ) and try triggering the builds again.
we did a slightly bigger version of that - we just untagged the newer dracut builds and fired a new compose. if that works at least we're sure the issue is dracut. it does seem likely, though.
that commit does indeed look like the most likely suspect, I can't offhand see what the issue is, though.
ah. from the Everything build log:
2021-05-20 05:55:42,867 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 35: /dev/fd/63: No such file or directory 2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 37: /dev/fd/63: No such file or directory 2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 35: /dev/fd/63: No such file or directory 2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 37: /dev/fd/63: No such file or directory 2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 35: /dev/fd/63: No such file or directory 2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 37: /dev/fd/63: No such file or directory 2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 35: /dev/fd/63: No such file or directory 2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 37: /dev/fd/63: No such file or directory 2021-05-20 05:55:42,868 INFO: dracut: Couldn't find SSL CA cert bundle or libnssckbi.so; HTTPS won't work.
note lines 35 and 37 are the ones that call read using that < <(somecommand) construction. There are a couple other similar lines in the log:
read
< <(somecommand)
2021-05-20 05:55:42,867 INFO: /usr/lib/dracut/modules.d/00systemd/module-setup.sh: line 168: /dev/fd/63: No such file or directory 2021-05-20 05:55:42,868 INFO: /usr/lib/dracut/modules.d/90kernel-modules-extra/module-setup.sh: line 77: /dev/fd/63: No such file or directory
those lines use the same < <(somecommand) construction. That seems to be the problem, somehow.
Note there's also a metric buttload of 2021-05-20 05:55:42,868 INFO: logger: socket /dev/log: No such file or directory. Not sure what's causing those.
2021-05-20 05:55:42,868 INFO: logger: socket /dev/log: No such file or directory
Filed https://bugzilla.redhat.com/show_bug.cgi?id=1962975 .
Metadata Update from @humaton: - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.