releng / failed-composes

Clone
Source Code
GIT

#717 Fedora-Rawhide-20191221.n.0 FINISHED_INCOMPLETE
Closed 2 years ago by humaton. Opened 4 years ago by releng.

Closed
Reopen Issue

pungi.global.log

[ERROR   ] [FAIL] Ostree (variant Silverblue, arch ppc64le) failed, but going on anyway.
[ERROR   ] Runroot task failed: 39829997. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20191221.n.0/logs/ppc64le/Silverblue/ostree-2/runroot.log for more details.

[ERROR   ] [FAIL] Ostree (variant Silverblue, arch aarch64) failed, but going on anyway.
[ERROR   ] Runroot task failed: 39829996. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20191221.n.0/logs/aarch64/Silverblue/ostree-3/runroot.log for more details.

[ERROR   ] [FAIL] Ostree (variant Silverblue, arch x86_64) failed, but going on anyway.
[ERROR   ] Runroot task failed: 39829998. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20191221.n.0/logs/x86_64/Silverblue/ostree-1/runroot.log for more details.

[OSTREE_INSTALLER] [ERROR   ] [FAIL] Ostree installer (variant Silverblue, arch x86_64) failed, but going on anyway.
[OSTREE_INSTALLER] [ERROR   ] Runroot task failed: 39830306. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20191221.n.0/logs/x86_64/Silverblue/ostree_installer-3/runroot.log for more details.

[OSTREE_INSTALLER] [ERROR   ] [FAIL] Ostree installer (variant Silverblue, arch ppc64le) failed, but going on anyway.
[OSTREE_INSTALLER] [ERROR   ] Runroot task failed: 39830305. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20191221.n.0/logs/ppc64le/Silverblue/ostree_installer-2/runroot.log for more details.

[OSTREE_INSTALLER] [ERROR   ] [FAIL] Ostree installer (variant Silverblue, arch aarch64) failed, but going on anyway.
[OSTREE_INSTALLER] [ERROR   ] Runroot task failed: 39830304. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20191221.n.0/logs/aarch64/Silverblue/ostree_installer-1/runroot.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Comp_Neuro) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 39833058. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20191221.n.0/logs/x86_64/livemedia-Labs-Comp_Neuro.x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant LXQt) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 39833163. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20191221.n.0/logs/x86_64/livemedia-Spins-LXQt.x86_64.log for more details.

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant Mate) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 39833066. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20191221.n.0/logs/armhfp/liveimage-Fedora-Mate-armhfp-Rawhide-20191221.n.0.armhfp.log for more details.

[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant SoaS) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 39833085. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20191221.n.0/logs/armhfp/liveimage-Fedora-SoaS-armhfp-Rawhide-20191221.n.0.armhfp.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Astronomy_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 39833050. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20191221.n.0/logs/x86_64/livemedia-Labs-Astronomy_KDE.x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Scientific_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 39833069. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20191221.n.0/logs/x86_64/livemedia-Labs-Scientific_KDE.x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Design_suite) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 39833063. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20191221.n.0/logs/x86_64/livemedia-Labs-Design_suite.x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Security) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 39833102. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20191221.n.0/logs/x86_64/livemedia-Labs-Security.x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Python_Classroom) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 39833126. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20191221.n.0/logs/x86_64/livemedia-Labs-Python_Classroom.x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Robotics) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 39833119. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20191221.n.0/logs/x86_64/livemedia-Labs-Robotics.x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Jam_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 39833112. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20191221.n.0/logs/x86_64/livemedia-Labs-Jam_KDE.x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant Mate) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 39833160. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20191221.n.0/logs/x86_64/livemedia-Spins-Mate.x86_64.log for more details.

[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant SoaS) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 39833146. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20191221.n.0/logs/x86_64/livemedia-Spins-SoaS.x86_64.log for more details.

[ERROR   ] [FAIL] Live media (variant Workstation, arch ppc64le, subvariant Workstation) failed, but going on anyway.
[LIVE_MEDIA      ] [INFO    ] [DONE ] Live media: Fedora-Workstation-Live (arches: ppc64le x86_64, variant: Workstation, subvariant: Workstation) (task id: 39833165)

[ERROR   ] [FAIL] Image build (variant Container, arch armhfp, subvariant Container_Minimal_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: docker, arches: aarch64-armhfp-ppc64le-s390x-x86_64, variant: Container, subvariant: Container_Minimal_Base) (task id: 39833065)

[ERROR   ] [FAIL] Image build (variant Container, arch ppc64le, subvariant Container_Minimal_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: docker, arches: aarch64-armhfp-ppc64le-s390x-x86_64, variant: Container, subvariant: Container_Minimal_Base) (task id: 39833065)

[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Labs, arch *, subvariant Scientific) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 39833084. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20191221.n.0/logs/x86_64/imagebuild-Labs-Scientific-vagrant-libvirt-vagrant-virtualbox.x86_64.log for more details.

[ERROR   ] [FAIL] Image build (variant Container, arch armhfp, subvariant Container_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: docker, arches: aarch64-armhfp-ppc64le-s390x-x86_64, variant: Container, subvariant: Container_Base) (task id: 39833059)

[ERROR   ] [FAIL] Image build (variant Container, arch ppc64le, subvariant Container_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: docker, arches: aarch64-armhfp-ppc64le-s390x-x86_64, variant: Container, subvariant: Container_Base) (task id: 39833059)

[ERROR   ] [FAIL] Image build (variant Cloud, arch ppc64le, subvariant Cloud_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: qcow2-raw-xz-vmdk, arches: aarch64-ppc64le-s390x-x86_64, variant: Cloud, subvariant: Cloud_Base) (task id: 39833044)

[ERROR   ] [FAIL] Image build (variant Server, arch armhfp, subvariant Server) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: raw-xz, arches: aarch64-armhfp, variant: Server, subvariant: Server) (task id: 39833103)

[ERROR   ] [FAIL] Image build (variant Spins, arch armhfp, subvariant Minimal) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: raw-xz, arches: aarch64-armhfp, variant: Spins, subvariant: Minimal) (task id: 39833114)

[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Spins, arch *, subvariant Xfce) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 39833122. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20191221.n.0/logs/aarch64/imagebuild-Spins-Xfce-raw-xz.aarch64.log for more details.

[ERROR   ] [FAIL] Image build (variant Workstation, arch armhfp, subvariant Workstation) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: raw-xz, arches: aarch64-armhfp-ppc64le, variant: Workstation, subvariant: Workstation) (task id: 39833130)

[ERROR   ] [FAIL] Image build (variant Workstation, arch ppc64le, subvariant Workstation) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] [DONE ] Creating image (formats: raw-xz, arches: aarch64-armhfp-ppc64le, variant: Workstation, subvariant: Workstation) (task id: 39833130)

Silverblue failed with some kind of module issue:

error: No available modular metadata for modular package 'c-ares-1.15.0-4.module_f32+5759+e5077333.x86_64'; cannot be installed on the system
Traceback (most recent call last):
  File "/usr/bin/pungi-make-ostree", line 15, in <module>
    ostree.main()
  File "/usr/lib/python3.8/site-packages/pungi/ostree/__init__.py", line 93, in main
    func()
  File "/usr/lib/python3.8/site-packages/pungi/ostree/tree.py", line 129, in run
    self._make_tree()
  File "/usr/lib/python3.8/site-packages/pungi/ostree/tree.py", line 51, in _make_tree
    shortcuts.run(
  File "/usr/lib/python3.8/site-packages/kobo/shortcuts.py", line 364, in run
    raise exc

@dustymabe @walters @jlebon @sgallagh

Is this still occurring? Given that the metadata was missing, I'm guessing it was an incomplete rsync to the mirrors.

This is strange. There is no mirror involved, the task is using a repo via compose machine:

Looking at today compose:

Task: https://koji.fedoraproject.org/koji/taskinfo?taskID=40214178
ostree-config: https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20200107.n.0/work/ostree-3/config_repo/
The problematic package is c-ares-1.15.0-4.module_f32+5759+e5077333.aarch64, which is repo for nodejs:12 module.
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20200107.n.0/work/aarch64/repo/module-nodejs-12-3220191202171146-43bbeeef/repodata/
The package is listed in the metadata though.

Yeah I see what's going on here. The root cause is that rpm-ostree isn't module-aware so it's just trying to install modular packages as regular ones. And a recent-ish change in libdnf no longer likes that.

Anyway, submitted a patch upstream:
https://github.com/coreos/rpm-ostree/pull/1961

(But again, the long-term fix is actually supporting modules correctly: https://github.com/coreos/rpm-ostree/issues/1435).

Thanks @jlebon for debugging it.

I just built https://koji.fedoraproject.org/koji/buildinfo?buildID=1427863 with that patch backported. If it fixes the rawhide issue, I'll backport it to f31 as well.

I'm actually not sure offhand why we're not seeing the same failure there. I suspect it's because bodhi composes aren't exposed to modular repos? A quick rpm -qa | grep \\.module_ on my Silverblue shows we're not currently composing with any modular packages. That's probably for the best for now TBH.

Latest compose had rpm-ostree-2019.7-2.fc32 , but still fails the same way.

Latest compose had rpm-ostree-2019.7-2.fc32 , but still fails the same way.

Thanks for the follow-up. Hmm weird, I've hit this exact issue in the rpm-ostree testsuite, and that patch fixed it. Anyway, I think I have to try to reproduce this locally before debugging further.

Thank you jlebon, I've confirmed that my own Silverblue composer on f31 works correctly with this patched rpm.

OK, it definitely works locally to target the master branch of https://pagure.io/workstation-ostree-config + rpm-ostree-2019.7-2. So I suspect it's specific to how pungi sets up the input repos.

I tried to test with exactly the same repo inputs as a latest pungi run (using artifacts from e.g. https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20200127.n.0/work), but then I hit the fact that the repodata primary XML uses xml:base="file:///mnt/koji/", and so:

Downloading from 'repo-56'... done
error: Curl error (37): Couldn't read a file:// file for file:///mnt/koji/packages/subversion/1.13.0/1.module_f32+6952+03f034b8/data/signed/12c944d0/x86_64/subversion-libs-1.13.0-1.module_f32+6952+03f034b8.x86_64.rpm [Couldn't open file /mnt/koji/packages/subversion/1.13.0/1.module_f32+6952+03f034b8/data/signed/12c944d0/x86_64/subversion-libs-1.13.0-1.module_f32+6952+03f034b8.x86_64.rpm]; Last error: Curl error (37): Couldn't read a file:// file for file:///mnt/koji/packages/subversion/1.13.0/1.module_f32+6952+03f034b8/data/signed/12c944d0/x86_64/subversion-libs-1.13.0-1.module_f32+6952+03f034b8.x86_64.rpm [Couldn't open file /mnt/koji/packages/subversion/1.13.0/1.module_f32+6952+03f034b8/data/signed/12c944d0/x86_64/subversion-libs-1.13.0-1.module_f32+6952+03f034b8.x86_64.rpm]

Will try to hack around this.
Edited 4 years ago by jlebon

I've built rpm-ostree-2019.7.31.g70c38563-1.fc32, which should hopefully really fix this now.

Metadata Update from @humaton:
- Issue status updated to: Closed (was: Open)
2 years ago

Login to comment on this ticket.

Metadata
None
None
None
None
Powered by Pagure 5.13.3
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.