pungi.global.log
[INFO ] [FAIL] Buildinstall (variant Everything, arch s390x) failed, but going on anyway. [INFO ] Runroot task failed: 26940138. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/s390x/buildinstall-Everything.s390x.log for more details.
[INFO ] [FAIL] Buildinstall (variant Server, arch s390x) failed, but going on anyway. [INFO ] Runroot task failed: 26940139. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/s390x/buildinstall-Server.s390x.log for more details.
[CREATEISO ] [INFO ] [FAIL] Iso (variant Server, arch s390x) failed, but going on anyway. [CREATEISO ] [INFO ] Runroot task failed: 26942697. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/s390x/createiso-Fedora-Server-dvd-s390x-Rawhide-20180513.n.0.iso.s390x.log for more details.
[LIVE_MEDIA ] [INFO ] [FAIL] Live media (variant Labs, arch *, subvariant Scientific_KDE) failed, but going on anyway. [LIVE_MEDIA ] [INFO ] Live media task failed: 26942701. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/i386-x86_64/livemedia-Labs-Scientific_KDE.i386-x86_64.log for more details.
[LIVE_MEDIA ] [INFO ] [FAIL] Live media (variant Labs, arch *, subvariant Astronomy_KDE) failed, but going on anyway. [LIVE_MEDIA ] [INFO ] Live media task failed: 26942692. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/i386-x86_64/livemedia-Labs-Astronomy_KDE.i386-x86_64.log for more details.
[IMAGE_BUILD ] [INFO ] [FAIL] Image build (variant Server, arch *, subvariant Server) failed, but going on anyway. [IMAGE_BUILD ] [INFO ] ImageBuild task failed: 26942739. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/aarch64/imagebuild-Server-Server-raw-xz.aarch64.log for more details.
[IMAGE_BUILD ] [INFO ] [FAIL] Image build (variant Workstation, arch *, subvariant Workstation) failed, but going on anyway. [IMAGE_BUILD ] [INFO ] ImageBuild task failed: 26942745. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/aarch64/imagebuild-Workstation-Workstation-raw-xz.aarch64.log for more details.
[IMAGE_BUILD ] [INFO ] [FAIL] Image build (variant Spins, arch *, subvariant Minimal) failed, but going on anyway. [IMAGE_BUILD ] [INFO ] ImageBuild task failed: 26942742. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/aarch64/imagebuild-Spins-Minimal-raw-xz.aarch64.log for more details.
[IMAGE_BUILD ] [INFO ] [FAIL] Image build (variant Labs, arch *, subvariant Python_Classroom) failed, but going on anyway. [IMAGE_BUILD ] [INFO ] ImageBuild task failed: 26942727. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/i386-x86_64/imagebuild-Labs-Python_Classroom-vagrant-libvirt-vagrant-virtualbox.i386-x86_64.log for more details.
[IMAGE_BUILD ] [INFO ] [FAIL] Image build (variant Container, arch *, subvariant Container_Minimal_Base) failed, but going on anyway. [IMAGE_BUILD ] [INFO ] ImageBuild task failed: 26942723. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/aarch64-armhfp-ppc64le-s390x-x86_64/imagebuild-Container-Container_Minimal_Base-docker.aarch64-armhfp-ppc64le-s390x-x86_64.log for more details.
[IMAGE_BUILD ] [INFO ] [FAIL] Image build (variant Container, arch *, subvariant Container_Base) failed, but going on anyway. [IMAGE_BUILD ] [INFO ] ImageBuild task failed: 26942720. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/aarch64-armhfp-ppc64le-s390x-x86_64/imagebuild-Container-Container_Base-docker.aarch64-armhfp-ppc64le-s390x-x86_64.log for more details.
[INFO ] [FAIL] Image build (variant AtomicHost, arch aarch64, subvariant AtomicHost) failed, but going on anyway. [IMAGE_BUILD ] [INFO ] Hardlinking /mnt/koji/packages/Fedora-AtomicHost/Rawhide/20180513.n.0/images/Fedora-AtomicHost-Rawhide-20180513.n.0.ppc64le.qcow2 to /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/compose/AtomicHost/ppc64le/images/Fedora-AtomicHost-Rawhide-20180513.n.0.ppc64le.qcow2
[ERROR ] Compose run failed: ImageBuild task failed: 26942699. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/aarch64-ppc64-ppc64le-s390x-x86_64/imagebuild-Cloud-Cloud_Base-qcow2-raw-xz.aarch64-ppc64-ppc64le-s390x-x86_64.log for more details.
Container builds: https://bugzilla.redhat.com/show_bug.cgi?id=1577405
So, I have done a bit of digging on the aarch64 cloud base image (the one thats failing the composes).
It's in oz/Imagefactory and since aarch64 has no graphics device defined we cannot get a screenshot. The logs are useless, simply telling us the install timed out.
I did some scratch builds and grabbed a disk and examined it. It seems like it's installed everything (at least there is 1 dnf transaction with 343 packages in it:
1 | | 2018-05-12 18:21 | Install | 345 EE
The errors from this are:
Scriptlet output: 1 /var/tmp/rpm-tmp.GJOQ1z: line 1: rm: command not found 2 /usr/share/crypto-policies/reload-cmds.sh: line 1: systemctl: command not found 3 /usr/share/crypto-policies/reload-cmds.sh: line 2: systemctl: command not found 4 /usr/share/crypto-policies/reload-cmds.sh: line 3: systemctl: command not found 5 dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 6 Running in chroot, ignoring request: daemon-reload
But there's no changes in dracut, kernel, anaconda, crypto-policies, systemd between the last success and the first failure of this kind, and the x86_64 image completes.
The composers have been upgraded to f28... but there's no errors or anything on them.
I'm open to ideas how to further debug this...
@pwhalen @pbrobinson @adamw
Got some console output...
... [ 296.064946] random: fast init done [ 471.920176] random: crng init done [ 472.114006] systemd[1]: systemd 238 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +U TMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hy brid) [ 472.118029] systemd[1]: Detected virtualization kvm. [ 472.118874] systemd[1]: Detected architecture arm64. [ 472.119719] systemd[1]: Running in initial RAM disk. ...
Looks like we are hitting... The slow random again, but only on aarch64.
Maybe the aarch64 has a special path for oz and we need to add the RNG to that too.
I also know the fix the screenshot functionality in oz/imagefactory on arm, I'm hoping to get time to test that soon and do a PR.
You can also just connect to the console while imagefactory is running via virsh/virt-manager and watch the install happen there
@pbrobinson Me and Kevin only realized later that the problem is that the host is starved of entropy, thus /dev/random blocks (and virtio-rng is less effectivce). The fact that virtio-rng is working causes it all to eventually boot, since without that it would wait for entropy indefinitely.
We just need to get entropy on the build hosts up.
There's a xgene_rng driver and it has HW rng so maybe install rng-tools and enable the rngd service to get HW entropy mixed into /dev/random?
Just installed and enabled rngd on builders, and they have sufficient entropy on the hosts now that they should be able to support VMs prng init. (it automatically loaded xgene_rng indeed.) Thanks.
Login to comment on this ticket.