releng / failed-composes

#2126 Fedora-Rawhide-20201214.n.0 FINISHED_INCOMPLETE
Closed by humaton. Opened by releng.

Closed
Reopen Issue

pungi.global.log

[ERROR   ] [FAIL] Ostree (variant Silverblue, arch x86_64) failed, but going on anyway.
[ERROR   ] Runroot task failed: 57412984. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20201214.n.0/logs/x86_64/Silverblue/ostree-1/runroot.log for more details.

[ERROR   ] [FAIL] Ostree (variant Silverblue, arch ppc64le) failed, but going on anyway.
[ERROR   ] Runroot task failed: 57412985. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20201214.n.0/logs/ppc64le/Silverblue/ostree-2/runroot.log for more details.

[ERROR   ] [FAIL] Ostree (variant Silverblue, arch aarch64) failed, but going on anyway.
[ERROR   ] Runroot task failed: 57412986. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20201214.n.0/logs/aarch64/Silverblue/ostree-3/runroot.log for more details.

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant SoaS) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 57422429. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20201214.n.0/logs/armhfp/liveimage-Fedora-SoaS-armhfp-Rawhide-20201214.n.0.armhfp.log for more details.

Variant: Spins, subvariant: SoaS task failed. Pinging maintainers: @chimosky @pbrobinson 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Security) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 57422419. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20201214.n.0/logs/x86_64/livemedia-Labs-Security.x86_64.log for more details.

Variant: Labs, subvariant: Security task failed. Pinging maintainers: @jsimon @fab 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Scientific_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 57422404. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20201214.n.0/logs/x86_64/livemedia-Labs-Scientific_KDE.x86_64.log for more details.

Variant: Labs, subvariant: Scientific_KDE task failed. Pinging maintainers: @scitech 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant SoaS) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 57422456. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20201214.n.0/logs/x86_64/livemedia-Spins-SoaS.x86_64.log for more details.

Variant: Spins, subvariant: SoaS task failed. Pinging maintainers: @chimosky @pbrobinson 

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant KDE) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 57422402. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20201214.n.0/logs/armhfp/liveimage-Fedora-KDE-armhfp-Rawhide-20201214.n.0.armhfp.log for more details.

[ERROR   ] [FAIL] Image build (variant Cloud, arch ppc64le, subvariant Cloud_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: qcow2-raw-xz, arches: aarch64-ppc64le-s390x-x86_64, variant: Cloud, subvariant: Cloud_Base) (task id: 57422399)

[ERROR   ] [FAIL] Image build (variant Container, arch ppc64le, subvariant Container_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: docker, arches: aarch64-armhfp-ppc64le-s390x-x86_64, variant: Container, subvariant: Container_Base) (task id: 57422465)

[ERROR   ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: docker, arches: aarch64-armhfp-ppc64le-s390x-x86_64, variant: Container, subvariant: Container_Base) (task id: 57422465)

[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Labs, arch *, subvariant Scientific) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 57422536. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20201214.n.0/logs/x86_64/imagebuild-Labs-Scientific-vagrant-libvirt-vagrant-virtualbox.x86_64.log for more details.

Variant: Labs, subvariant: Scientific task failed. Pinging maintainers: @scitech 

[ERROR   ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Minimal_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: docker, arches: aarch64-armhfp-ppc64le-s390x-x86_64, variant: Container, subvariant: Container_Minimal_Base) (task id: 57422489)

[ERROR   ] [FAIL] Image build (variant Container, arch ppc64le, subvariant Container_Minimal_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: docker, arches: aarch64-armhfp-ppc64le-s390x-x86_64, variant: Container, subvariant: Container_Minimal_Base) (task id: 57422489)

Compose Total time: 6:16:28
Compose phase INIT time: 0:00:24.
Compose phase PKGSET time: 0:30:23.
Compose phase WEAVER: FAILED.
Compose phase BUILDINSTALL time: 2:07:44.
Compose phase GATHER time: 3:16:36.
Compose phase OSTREE time: 0:02:32.
Compose phase OSTREE_INSTALLER time: 0:41:24.
Compose phase CREATEREPO time: 0:24:28.
Compose phase CREATEISO time: 0:12:29.
Compose phase LIVE_IMAGES time: 0:46:31.
Compose phase REPOCLOSURE time: 0:02:41.
Compose phase LIVE_MEDIA time: 0:40:00.
Compose phase IMAGE_BUILD time: 1:51:20.
Compose phase IMAGE_CHECKSUM time: 0:10:56.
Compose phase TEST time: 0:00:02.

I think this is most likely to be fallout from an updated mock. It's also affecting IoT which is also using pungi/mock. Copying discussion from IRC there:

<walters> i thought we'd worked around this in rpm-ostree; for coreos-assembler right now in OpenShift we fall back to virtualization since effectively nested containers require privileges
<walters> ok right found https://github.com/coreos/rpm-ostree/commit/0649567ffc5632529cea1f1295a9b5f67fd9bede so I thought we'd settled on using mock --new-chroot in pungi

OK this is also affecting 33, see e.g.
https://kojipkgs.fedoraproject.org/compose/updates/Fedora-33-updates-testing-20201214.0/logs/x86_64/Everything/ostree-1/create-ostree-repo.log

But this one was successful:
https://kojipkgs.fedoraproject.org/compose/updates/Fedora-33-updates-testing-20201208.1/logs/x86_64/Everything/ostree-1/create-ostree-repo.log

So I think we have a "last week" time bound here.

(Moving to this ticket from https://github.com/fedora-silverblue/issue-tracker/issues/112#issuecomment-744599568 )

The task logs show a bump from 2.6 to 2.7 indeed:

  • INFO: mock.py version 2.6 starting (python version = 3.9.0)...
  • INFO: mock.py version 2.7 starting (python version = 3.9.0, NVR = mock-2.7-1.fc33)...

Yeah, these are run in rootroot tasks on koji, so they use the mock version on the builders.

They do set new_chroot = True tho.

see https://koji.fedoraproject.org/koji/taskinfo?taskID=57399846 for example.

So, perhaps we should downgrade builders to mock 2.6 for now until we can sort this out?

So, perhaps we should downgrade builders to mock 2.6 for now until we can sort this out?

If that's not a problem, yes let's do that.

I looked over the diff in mock and there's a fair bit of code churn (files moved) but I don't see an obvious problem so far.

It sure looks to me like both --isolation=nspawn and --new-chroot simply stopped using nspawn in mock 2.7.

Downgrading now.

https://github.com/rpm-software-management/mock/issues/678

Many thanks for the debugging!

So my first suspect here is https://github.com/rpm-software-management/mock/commit/877b332610173f849bde8fbccd98abb2dda4345b , which moves around some of the relevant bits here. In particular there's some weirdness related to a var called USE_NSPAWN. Before that commit, it was only ever declared in mock/py/mockbuild/util.py; a few other things imported it from there. That commit moves some of the logic related to it into mock/py/mockbuild/config.py, but the result is kinda odd. It's still declared in util.py , from where config.py imports it and sets it as a global and does stuff to it. Other things still import it from util, not from config.

I haven't entirely traced out the consequences, but I'd bet a medium sized amount of money that's the problem.

https://github.com/rpm-software-management/mock/pull/679 should fix mock, I think.

Thanks @walters for debugging it and thank you @adamwill for a fix.

@kevin Let me know if you need any help.

Metadata Update from @humaton:
- Issue status updated to: Closed (was: Open)
Metadata
None

FINISHED_INCOMPLETE Fedora Rawhide

None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.