releng / failed-composes

#1761 Fedora-Rawhide-20200914.n.0 DOOMED
Closed by humaton. Opened by releng.

Closed
Reopen Issue

pungi.global.log

[ERROR   ] [FAIL] Ostree (variant Silverblue, arch x86_64) failed, but going on anyway.
[ERROR   ] Runroot task failed: 51417270. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/x86_64/Silverblue/ostree-1/runroot.log for more details.

[ERROR   ] [FAIL] Ostree (variant Silverblue, arch ppc64le) failed, but going on anyway.
[ERROR   ] Runroot task failed: 51417271. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/ppc64le/Silverblue/ostree-2/runroot.log for more details.

[ERROR   ] [FAIL] Buildinstall (variant Everything, arch s390x) failed, but going on anyway.
[ERROR   ] Runroot task failed: 51417301. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/s390x/buildinstall-Everything.s390x.log for more details.

[ERROR   ] [FAIL] Buildinstall (variant Server, arch s390x) failed, but going on anyway.
[ERROR   ] Runroot task failed: 51417303. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/s390x/buildinstall-Server.s390x.log for more details.

[ERROR   ] [FAIL] Ostree (variant Silverblue, arch aarch64) failed, but going on anyway.
[ERROR   ] Runroot task failed: 51417272. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/aarch64/Silverblue/ostree-3/runroot.log for more details.

[CREATEISO       ] [ERROR   ] [FAIL] Iso (variant Server, arch x86_64) failed, but going on anyway.
[CREATEISO       ] [ERROR   ] Runroot task failed: 51431189. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/x86_64/createiso-Fedora-Server-dvd-x86_64-Rawhide-20200914.n.0.iso.x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Astronomy_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 51431171. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/x86_64/livemedia-Labs-Astronomy_KDE.x86_64.log for more details.

Variant: Labs, subvariant: Astronomy_KDE task failed. Pinging maintainers: @lupinix 

[CREATEISO       ] [ERROR   ] [FAIL] Iso (variant Server, arch ppc64le) failed, but going on anyway.
[CREATEISO       ] [ERROR   ] Runroot task failed: 51431186. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/ppc64le/createiso-Fedora-Server-dvd-ppc64le-Rawhide-20200914.n.0.iso.ppc64le.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Comp_Neuro) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 51431174. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/x86_64/livemedia-Labs-Comp_Neuro.x86_64.log for more details.

Variant: Labs, subvariant: Comp_Neuro task failed. Pinging maintainers: @ankursinha @neuro-sig @bt0dotninja @dan1mal 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Scientific_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 51431178. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/x86_64/livemedia-Labs-Scientific_KDE.x86_64.log for more details.

Variant: Labs, subvariant: Scientific_KDE task failed. Pinging maintainers: @scitech 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Design_suite) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 51431176. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/x86_64/livemedia-Labs-Design_suite.x86_64.log for more details.

Variant: Labs, subvariant: Design_suite task failed. Pinging maintainers: @luya 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Games) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 51431184. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/x86_64/livemedia-Labs-Games.x86_64.log for more details.

Variant: Labs, subvariant: Games task failed. Pinging maintainers: @dagostinelli 

[CREATEISO       ] [ERROR   ] [FAIL] Iso (variant Server, arch aarch64) failed, but going on anyway.
[CREATEISO       ] [ERROR   ] Runroot task failed: 51431183. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/aarch64/createiso-Fedora-Server-dvd-aarch64-Rawhide-20200914.n.0.iso.aarch64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Security) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 51431193. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/x86_64/livemedia-Labs-Security.x86_64.log for more details.

Variant: Labs, subvariant: Security task failed. Pinging maintainers: @jsimon @fab 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Robotics) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 51431206. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/x86_64/livemedia-Labs-Robotics.x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant SoaS) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 51431226. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/x86_64/livemedia-Spins-SoaS.x86_64.log for more details.

Variant: Spins, subvariant: SoaS task failed. Pinging maintainers: @pbrobinson @chimosky 

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Labs, arch armhfp, subvariant Python_Classroom) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 51431165. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/armhfp/liveimage-Fedora-Python-Classroom-armhfp-Rawhide-20200914.n.0.armhfp.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Python_Classroom) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 51431211. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/x86_64/livemedia-Labs-Python_Classroom.x86_64.log for more details.

Variant: Labs, subvariant: Python_Classroom task failed. Pinging maintainers: @python-sig @churchyard 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Jam_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 51431202. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/x86_64/livemedia-Labs-Jam_KDE.x86_64.log for more details.

Variant: Labs, subvariant: Jam_KDE task failed. Pinging maintainers: @eeickmeyer 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant Xfce) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 51431222. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/x86_64/livemedia-Spins-Xfce.x86_64.log for more details.

Variant: Spins, subvariant: Xfce task failed. Pinging maintainers: @maxamillion @kevin @nonamedotc 

[CREATEISO       ] [ERROR   ] [FAIL] Iso (variant Server, arch armhfp) failed, but going on anyway.
[CREATEISO       ] [ERROR   ] Runroot task failed: 51431187. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/armhfp/createiso-Fedora-Server-dvd-armhfp-Rawhide-20200914.n.0.iso.armhfp.log for more details.

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Server, arch armhfp, subvariant Server) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 51431168. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/armhfp/liveimage-Fedora-Server-armhfp-Rawhide-20200914.n.0.armhfp.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant LXDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 51431235. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/x86_64/livemedia-Spins-LXDE.x86_64.log for more details.

Variant: Spins, subvariant: LXDE task failed. Pinging maintainers: @cwickert 

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant KDE) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 51431170. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/armhfp/liveimage-Fedora-KDE-armhfp-Rawhide-20200914.n.0.armhfp.log for more details.

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant Mate) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 51431175. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/armhfp/liveimage-Fedora-Mate-armhfp-Rawhide-20200914.n.0.armhfp.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant Cinnamon) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 51431234. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/x86_64/livemedia-Spins-Cinnamon.x86_64.log for more details.

Variant: Spins, subvariant: Cinnamon task failed. Pinging maintainers: @grinnz 

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant Mate) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 51431236. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/x86_64/livemedia-Spins-Mate.x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant LXQt) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 51431238. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/x86_64/livemedia-Spins-LXQt.x86_64.log for more details.

Variant: Spins, subvariant: LXQt task failed. Pinging maintainers: @zsun @lupinix 

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant LXDE) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 51431173. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/armhfp/liveimage-Fedora-LXDE-armhfp-Rawhide-20200914.n.0.armhfp.log for more details.

Variant: Spins, subvariant: LXDE task failed. Pinging maintainers: @cwickert 

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant LXQt) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 51431200. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/armhfp/liveimage-Fedora-LXQt-armhfp-Rawhide-20200914.n.0.armhfp.log for more details.

Variant: Spins, subvariant: LXQt task failed. Pinging maintainers: @zsun @lupinix 

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant SoaS) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 51431181. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/armhfp/liveimage-Fedora-SoaS-armhfp-Rawhide-20200914.n.0.armhfp.log for more details.

Variant: Spins, subvariant: SoaS task failed. Pinging maintainers: @pbrobinson @chimosky 

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant Xfce) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 51431191. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/armhfp/liveimage-Fedora-Xfce-armhfp-Rawhide-20200914.n.0.armhfp.log for more details.

Variant: Spins, subvariant: Xfce task failed. Pinging maintainers: @maxamillion @kevin @nonamedotc 

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Workstation, arch armhfp, subvariant Workstation) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 51431204. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/armhfp/liveimage-Fedora-Workstation-armhfp-Rawhide-20200914.n.0.armhfp.log for more details.

[ERROR   ] [FAIL] Image build (variant Spins, arch armhfp, subvariant Minimal) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: raw-xz, arches: aarch64-armhfp, variant: Spins, subvariant: Minimal) (task id: 51431267)

[ERROR   ] [FAIL] Image build (variant Server, arch armhfp, subvariant Server) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: raw-xz, arches: aarch64-armhfp, variant: Server, subvariant: Server) (task id: 51431263)

[ERROR   ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Minimal_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: docker, arches: aarch64-armhfp-ppc64le-s390x-x86_64, variant: Container, subvariant: Container_Minimal_Base) (task id: 51431224)

[ERROR   ] [FAIL] Image build (variant Container, arch armhfp, subvariant Container_Minimal_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: docker, arches: aarch64-armhfp-ppc64le-s390x-x86_64, variant: Container, subvariant: Container_Minimal_Base) (task id: 51431224)

[ERROR   ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: docker, arches: aarch64-armhfp-ppc64le-s390x-x86_64, variant: Container, subvariant: Container_Base) (task id: 51431208)

[ERROR   ] [FAIL] Image build (variant Container, arch armhfp, subvariant Container_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: docker, arches: aarch64-armhfp-ppc64le-s390x-x86_64, variant: Container, subvariant: Container_Base) (task id: 51431208)

[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Labs, arch *, subvariant Scientific) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 51431248. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/x86_64/imagebuild-Labs-Scientific-vagrant-libvirt-vagrant-virtualbox.x86_64.log for more details.

Variant: Labs, subvariant: Scientific task failed. Pinging maintainers: @scitech 

[ERROR   ] [FAIL] Image build (variant Workstation, arch armhfp, subvariant Workstation) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: raw-xz, arches: aarch64-armhfp, variant: Workstation, subvariant: Workstation) (task id: 51431288)

[ERROR   ] [FAIL] Image build (variant Cloud, arch s390x, subvariant Cloud_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: qcow2-raw-xz-vmdk, arches: aarch64-ppc64le-s390x-x86_64, variant: Cloud, subvariant: Cloud_Base) (task id: 51431172)
  • Compose run failed because: - 51431219
[ERROR   ] Compose run failed: Live media task failed: 51431219. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20200914.n.0/logs/x86_64/livemedia-Spins-KDE.x86_64.log for more details.

Compose Total time: 5:05:22
Compose phase INIT time: 0:00:23.
Compose phase PKGSET time: 0:25:33.
Compose phase WEAVER: FAILED.
Compose phase BUILDINSTALL time: 1:49:36.
Compose phase GATHER time: 2:52:11.
Compose phase OSTREE time: 0:03:45.
Compose phase OSTREE_INSTALLER time: 0:39:30.
Compose phase CREATEREPO time: 0:26:16.
Compose phase CREATEISO time: 0:02:09.
Compose phase REPOCLOSURE time: 0:02:34.
Compose phase LIVE_IMAGES: FAILED.
Compose phase IMAGE_BUILD time: 1:19:24.
Compose phase LIVE_MEDIA: FAILED.

Log says:

INFO: Mock Version: 2.4
Traceback (most recent call last):
  File "/usr/libexec/mock/mock", line 1026, in <module>
    exitStatus = main()
  File "/usr/lib/python3.8/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 805, in main
    result = run_command(options, args, config_opts, commands, buildroot, state)
  File "/usr/lib/python3.8/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 859, in run_command
    commands.init(do_log=True)
  File "/usr/lib/python3.8/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.8/site-packages/mockbuild/backend.py", line 160, in init
    self.buildroot.initialize(**kwargs)
  File "/usr/lib/python3.8/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.8/site-packages/mockbuild/buildroot.py", line 131, in initialize
    self._init(prebuild=prebuild, do_log=do_log)
  File "/usr/lib/python3.8/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.8/site-packages/mockbuild/buildroot.py", line 196, in _init
    self._setup_resolver_config()
  File "/usr/lib/python3.8/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.8/site-packages/mockbuild/buildroot.py", line 293, in _setup_resolver_config
    self._copy_config('resolv.conf')
  File "/usr/lib/python3.8/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.8/site-packages/mockbuild/buildroot.py", line 286, in _copy_config
    shutil.copy2(orig_conf_file, etcdir)
  File "/usr/lib64/python3.8/shutil.py", line 432, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.8/shutil.py", line 261, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/mock/f34-build-22781080-2079948/root/etc/resolv.conf'

I thought I'd read something about this on the -devel list. I'll dig up the archives.

mock_output.log is reporting this:
ERROR: Option --old-chroot has been deprecated. Use --isolation=simple instead.

that has been the case for a very long time, but we do still actually use --old-chroot on purpose in various cases and it is expected to work.

I believe this was caused by this change in systemd %post, because openQA hit exactly the same problem testing the F33 update with the same change in it. @zbyszek
Edited by adamwill

INFO: Mock Version: 2.4
File "/usr/lib64/python3.8/shutil.py", line 261

Hmm, is this executing mock from F33? Then https://bodhi.fedoraproject.org/updates/FEDORA-2020-8b8b3ade7b should fix this.

Nope, but mock-2.6-1.fc34 should fix this, I think.

mock-2.6-1.fc34 uses python3.9, so I don't think it can.

RFE: https://bugzilla.redhat.com/show_bug.cgi?id=1879231

@zbyszek why does python3.9 matter here?

Sorry, I don't know too much about the compose process, so I might be saying something very stupid. It's just that F34 has python3.9 and there's python3.8 paths in the backtrace, so I don't think mock from rawhide which has python3.9 paths can be the version responsible for that backtrace.

Ah, yes... it's mock on the builders. (f32)

Oh, OK. But we have mock-2.5-2.fc32 in stable, so the builder must be not up to date.

We absolutely do not apply all fedora updates to builders all the time. We do apply security updates daily, but non security updates wait until we do a mass update/reboot window. We never do such a window during a freeze. We want the minimum amount of things to be changing under the composes for the milestone we are frozen for.

for the record, in the end we backported the fix for this to mock 2.4 (the version the builders are running), which is why composes are working again.

Metadata Update from @humaton:
- Issue status updated to: Closed (was: Open)
Metadata
None

DOOMED Fedora Rawhide

None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.