#308 Fedora-Rawhide-20180513.n.0 DOOMED
Opened 5 years ago by dustymabe. Modified 5 years ago

pungi.global.log

[INFO    ] [FAIL] Buildinstall (variant Everything, arch s390x) failed, but going on anyway.
[INFO    ] Runroot task failed: 26940138. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/s390x/buildinstall-Everything.s390x.log for more details.
[INFO    ] [FAIL] Buildinstall (variant Server, arch s390x) failed, but going on anyway.
[INFO    ] Runroot task failed: 26940139. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/s390x/buildinstall-Server.s390x.log for more details.
[CREATEISO       ] [INFO    ] [FAIL] Iso (variant Server, arch s390x) failed, but going on anyway.
[CREATEISO       ] [INFO    ] Runroot task failed: 26942697. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/s390x/createiso-Fedora-Server-dvd-s390x-Rawhide-20180513.n.0.iso.s390x.log for more details.
[LIVE_MEDIA      ] [INFO    ] [FAIL] Live media (variant Labs, arch *, subvariant Scientific_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [INFO    ] Live media task failed: 26942701. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/i386-x86_64/livemedia-Labs-Scientific_KDE.i386-x86_64.log for more details.
[LIVE_MEDIA      ] [INFO    ] [FAIL] Live media (variant Labs, arch *, subvariant Astronomy_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [INFO    ] Live media task failed: 26942692. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/i386-x86_64/livemedia-Labs-Astronomy_KDE.i386-x86_64.log for more details.
[IMAGE_BUILD     ] [INFO    ] [FAIL] Image build (variant Server, arch *, subvariant Server) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] ImageBuild task failed: 26942739. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/aarch64/imagebuild-Server-Server-raw-xz.aarch64.log for more details.
[IMAGE_BUILD     ] [INFO    ] [FAIL] Image build (variant Workstation, arch *, subvariant Workstation) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] ImageBuild task failed: 26942745. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/aarch64/imagebuild-Workstation-Workstation-raw-xz.aarch64.log for more details.
[IMAGE_BUILD     ] [INFO    ] [FAIL] Image build (variant Spins, arch *, subvariant Minimal) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] ImageBuild task failed: 26942742. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/aarch64/imagebuild-Spins-Minimal-raw-xz.aarch64.log for more details.
[IMAGE_BUILD     ] [INFO    ] [FAIL] Image build (variant Labs, arch *, subvariant Python_Classroom) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] ImageBuild task failed: 26942727. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/i386-x86_64/imagebuild-Labs-Python_Classroom-vagrant-libvirt-vagrant-virtualbox.i386-x86_64.log for more details.
[IMAGE_BUILD     ] [INFO    ] [FAIL] Image build (variant Container, arch *, subvariant Container_Minimal_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] ImageBuild task failed: 26942723. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/aarch64-armhfp-ppc64le-s390x-x86_64/imagebuild-Container-Container_Minimal_Base-docker.aarch64-armhfp-ppc64le-s390x-x86_64.log for more details.
[IMAGE_BUILD     ] [INFO    ] [FAIL] Image build (variant Container, arch *, subvariant Container_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] ImageBuild task failed: 26942720. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/aarch64-armhfp-ppc64le-s390x-x86_64/imagebuild-Container-Container_Base-docker.aarch64-armhfp-ppc64le-s390x-x86_64.log for more details.
  • No Task ID, look at log statement
[INFO    ] [FAIL] Image build (variant AtomicHost, arch aarch64, subvariant AtomicHost) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-AtomicHost/Rawhide/20180513.n.0/images/Fedora-AtomicHost-Rawhide-20180513.n.0.ppc64le.qcow2 to /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/compose/AtomicHost/ppc64le/images/Fedora-AtomicHost-Rawhide-20180513.n.0.ppc64le.qcow2
  • Compose run failed because: - 26942699
[ERROR   ] Compose run failed: ImageBuild task failed: 26942699. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180513.n.0/logs/aarch64-ppc64-ppc64le-s390x-x86_64/imagebuild-Cloud-Cloud_Base-qcow2-raw-xz.aarch64-ppc64-ppc64le-s390x-x86_64.log for more details.

So, I have done a bit of digging on the aarch64 cloud base image (the one thats failing the composes).

It's in oz/Imagefactory and since aarch64 has no graphics device defined we cannot get a screenshot.
The logs are useless, simply telling us the install timed out.

I did some scratch builds and grabbed a disk and examined it. It seems like it's installed everything (at least there is 1 dnf transaction with 343 packages in it:

ID | Command line | Date and time | Action(s) | Altered

 1 |                          | 2018-05-12 18:21 | Install        |  345 EE

The errors from this are:

Scriptlet output:
1 /var/tmp/rpm-tmp.GJOQ1z: line 1: rm: command not found
2 /usr/share/crypto-policies/reload-cmds.sh: line 1: systemctl: command not found
3 /usr/share/crypto-policies/reload-cmds.sh: line 2: systemctl: command not found
4 /usr/share/crypto-policies/reload-cmds.sh: line 3: systemctl: command not found
5 dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y
6 Running in chroot, ignoring request: daemon-reload

But there's no changes in dracut, kernel, anaconda, crypto-policies, systemd between the last success and the first failure of this kind, and the x86_64 image completes.

The composers have been upgraded to f28... but there's no errors or anything on them.

I'm open to ideas how to further debug this...

@pwhalen @pbrobinson @adamw

Got some console output...

...
[ 296.064946] random: fast init done
[ 471.920176] random: crng init done
[ 472.114006] systemd[1]: systemd 238 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +U
TMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hy
brid)
[ 472.118029] systemd[1]: Detected virtualization kvm.
[ 472.118874] systemd[1]: Detected architecture arm64.
[ 472.119719] systemd[1]: Running in initial RAM disk.
...

Looks like we are hitting... The slow random again, but only on aarch64.

Looks like we are hitting... The slow random again, but only on aarch64.

Maybe the aarch64 has a special path for oz and we need to add the RNG to that too.

I also know the fix the screenshot functionality in oz/imagefactory on arm, I'm hoping to get time to test that soon and do a PR.

You can also just connect to the console while imagefactory is running via virsh/virt-manager and watch the install happen there

@pbrobinson Me and Kevin only realized later that the problem is that the host is starved of entropy, thus /dev/random blocks (and virtio-rng is less effectivce).
The fact that virtio-rng is working causes it all to eventually boot, since without that it would wait for entropy indefinitely.

We just need to get entropy on the build hosts up.

There's a xgene_rng driver and it has HW rng so maybe install rng-tools and enable the rngd service to get HW entropy mixed into /dev/random?

Just installed and enabled rngd on builders, and they have sufficient entropy on the hosts now that they should be able to support VMs prng init. (it automatically loaded xgene_rng indeed.) Thanks.

Login to comment on this ticket.

Metadata