pungi.global.log
[INFO ] [FAIL] Buildinstall (variant Modular, arch i386) failed, but going on anyway. [INFO ] Runroot task failed: 25530936. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180306.n.1/logs/i386/buildinstall-Modular.i386.log for more details.
[INFO ] [FAIL] Buildinstall (variant Server, arch i386) failed, but going on anyway. [INFO ] Runroot task failed: 25530937. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180306.n.1/logs/i386/buildinstall-Server.i386.log for more details.
[INFO ] [FAIL] Buildinstall (variant Everything, arch i386) failed, but going on anyway. [INFO ] Runroot task failed: 25530935. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180306.n.1/logs/i386/buildinstall-Everything.i386.log for more details.
[INFO ] [FAIL] Buildinstall (variant Workstation, arch i386) failed, but going on anyway. [INFO ] Runroot task failed: 25530938. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180306.n.1/logs/i386/buildinstall-Workstation.i386.log for more details.
[INFO ] [FAIL] Buildinstall (variant Server, arch x86_64) failed, but going on anyway. [INFO ] Runroot task failed: 25530956. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180306.n.1/logs/x86_64/buildinstall-Server.x86_64.log for more details.
[INFO ] [FAIL] Buildinstall (variant Server, arch aarch64) failed, but going on anyway. [INFO ] Runroot task failed: 25530922. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180306.n.1/logs/aarch64/buildinstall-Server.aarch64.log for more details.
[INFO ] [FAIL] Buildinstall (variant Server, arch armhfp) failed, but going on anyway. [INFO ] Runroot task failed: 25530927. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180306.n.1/logs/armhfp/buildinstall-Server.armhfp.log for more details.
[INFO ] [FAIL] Buildinstall (variant Everything, arch ppc64le) failed, but going on anyway. [INFO ] Runroot task failed: 25530946. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180306.n.1/logs/ppc64le/buildinstall-Everything.ppc64le.log for more details.
[INFO ] [FAIL] Buildinstall (variant Modular, arch ppc64le) failed, but going on anyway. [INFO ] Runroot task failed: 25530947. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180306.n.1/logs/ppc64le/buildinstall-Modular.ppc64le.log for more details.
[INFO ] [FAIL] Buildinstall (variant Everything, arch ppc64) failed, but going on anyway. [INFO ] Runroot task failed: 25530940. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180306.n.1/logs/ppc64/buildinstall-Everything.ppc64.log for more details.
[INFO ] [FAIL] Buildinstall (variant Modular, arch ppc64) failed, but going on anyway. [INFO ] Runroot task failed: 25530941. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180306.n.1/logs/ppc64/buildinstall-Modular.ppc64.log for more details.
[INFO ] [FAIL] Buildinstall (variant Cloud, arch ppc64le) failed, but going on anyway. [INFO ] Runroot task failed: 25530944. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180306.n.1/logs/ppc64le/buildinstall-Cloud.ppc64le.log for more details.
[INFO ] [FAIL] Buildinstall (variant Server, arch ppc64le) failed, but going on anyway. [INFO ] Runroot task failed: 25530948. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180306.n.1/logs/ppc64le/buildinstall-Server.ppc64le.log for more details.
[INFO ] [FAIL] Buildinstall (variant Cloud, arch ppc64) failed, but going on anyway. [INFO ] Runroot task failed: 25530939. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180306.n.1/logs/ppc64/buildinstall-Cloud.ppc64.log for more details.
[INFO ] [FAIL] Buildinstall (variant Server, arch ppc64) failed, but going on anyway. [INFO ] Runroot task failed: 25530943. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180306.n.1/logs/ppc64/buildinstall-Server.ppc64.log for more details.
[INFO ] [FAIL] Buildinstall (variant Cloud, arch s390x) failed, but going on anyway. [INFO ] Runroot task failed: 25530949. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180306.n.1/logs/s390x/buildinstall-Cloud.s390x.log for more details.
[INFO ] [FAIL] Buildinstall (variant Everything, arch s390x) failed, but going on anyway. [INFO ] Runroot task failed: 25530950. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180306.n.1/logs/s390x/buildinstall-Everything.s390x.log for more details.
[INFO ] [FAIL] Buildinstall (variant Modular, arch s390x) failed, but going on anyway. [INFO ] Runroot task failed: 25530951. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180306.n.1/logs/s390x/buildinstall-Modular.s390x.log for more details.
[INFO ] [FAIL] Buildinstall (variant Server, arch s390x) failed, but going on anyway. [INFO ] Runroot task failed: 25530952. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180306.n.1/logs/s390x/buildinstall-Server.s390x.log for more details.
[ERROR ] Compose run failed: Runroot task failed: 25530954. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180306.n.1/logs/x86_64/buildinstall-Everything.x86_64.log for more details.
Seems like all of them failed with a cp command during lorax run to create buildinstall media:
subprocess.CalledProcessError: Command '['/bin/cp', '-alx', '/var/tmp/lorax.lviy6jf7/installtree', '/var/tmp/lorax.lviy6jf7/installroot']' returned non-zero exit status 1.
Are we running out of disk space somehow? Wonder why this would have changed so much in the last 5 days or so. Maybe issue with new pungi?
Nope, the actual failure is the scriptlets failing abobve so there's no install tree to copy there.
It looks like we got a nice new shiny systemd version right before freeze and some or all of it's scriptlets fail in the chroot.
@zbyszek Can you please test systemd in a mock install chroot and fix the scriptlets?
Ah, looks like most of it was https://bugzilla.redhat.com/show_bug.cgi?id=1551793 but there's still a kernel install issue (along with an lvm2 one?)
Yes, #1551793 should fix this. I screwed this one up. I spent a few hours testing this version before pushing it out, but I didn't test installation of other packages ;). Apologies.
Branched has the same problem , unfortunately - may need an FE there.
Yeah, looks like systemd-238-2.fc29 fixes most of those scriptlet issues, however, we still have:
DEBUG util.py:439: 2018-03-06 23:20:36,795: Non-fatal POSTIN scriptlet failure in rpm package lvm2
Not sure what this one is... lvm2 hasn't changed since 2018-02-10.
DEBUG util.py:439: 2018-03-06 23:21:22,950: Non-fatal <unknown> scriptlet failure in rpm package nfs-utils DEBUG util.py:439: 2018-03-06 23:21:23,018: Non-fatal <unknown> scriptlet failure in rpm package nfs-utils
This also hasn't changed since 2018-02-15.
DEBUG util.py:439: 2018-03-06 23:22:41,901: Non-fatal POSTTRANS scriptlet failure in rpm package kernel-core
The kernel-core one smells like systemd changing /bin/kernel-install. @zbyszek has kernel-installed changed in the new systemd in a way that wouldn't work in a chroot?
So, perhaps the kernel-core one is the one causing the issues and the others are ok now? Or this is some completely different bug in lorax... ?
Most of lvm2's scriptlets relate to systemd services:
https://src.fedoraproject.org/rpms/lvm2/blob/master/f/lvm2.spec#_194-244
so it certainly makes sense to me that those could be broken by a systemd update.
@zbyszek "I spent a few hours testing this version before pushing it out, but I didn't test installation of other packages ;). Apologies."
Why is this not part of upstream CI?
Well, upstream CI is for upstream stuff, i.e. unit tests and generic tests. Creating an rpm package, installing it, and then installing additional rpm packages on top is something that is more suitable for downstream distributions that use rpm.
A related reason is that most of our upstream CI runs on Ubuntu because that's most easily available in a way that a) is fairly reliable, b) can be configured and managed by our contributors. We do have one or two CI services that use RPM: one was "CentOS CI", and another is "Fedora Rawhide CI". But the first one broke and had to be disabled, and the second one sometimes works, sometimes doesn't, and nobody knows where to go to fix it. With enough effort I'd probably be able to gain the credentials as a Fedora person or a Red Hat employee, but it's unlikely that arbitrary contributors could gain access to manage it. Without a pool of people to restart builds, diagnose failures, tweak configuration, etc, CI is not going to stay up.
If there's a way to run a compose from upstream CI, I'll love to have that. But I don't think that's feasible, considering how slow and resource consumptive this is.
we have the tests https://github.com/systemd/systemd-fedora-ci which is in progress. We need a infra where we need to run these tests with each PR.
CentOS CI is certainly intended to enable use cases like that; @bstinson should be able to help with it.
You could previously do it in Fedora infra Jenkins, but AIUI, that is currently being retired in favour of CentOS CI.
Login to comment on this ticket.