#580 Fedora-Rawhide-20180726.n.0 DOOMED
Opened 10 months ago by dustymabe. Modified 9 months ago

pungi.global.log

[INFO    ] [FAIL] Buildinstall (variant Everything, arch s390x) failed, but going on anyway.
[INFO    ] Runroot task failed: 28613406. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180726.n.0/logs/s390x/buildinstall-Everything.s390x.log for more details.
[INFO    ] [FAIL] Buildinstall (variant Server, arch s390x) failed, but going on anyway.
[INFO    ] Runroot task failed: 28613407. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180726.n.0/logs/s390x/buildinstall-Server.s390x.log for more details.
[INFO    ] [FAIL] Ostree (variant AtomicWorkstation, arch x86_64) failed, but going on anyway.
[INFO    ] Runroot task failed: 28616400. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180726.n.0/logs/x86_64/AtomicWorkstation/ostree-4/runroot.log for more details.
[LIVE_IMAGES     ] [INFO    ] [FAIL] Live (variant Spins, arch armhfp, subvariant KDE) failed, but going on anyway.
[LIVE_IMAGES     ] [INFO    ] LiveImage task failed: 28617672. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180726.n.0/logs/armhfp/liveimage-Fedora-KDE-armhfp-Rawhide-20180726.n.0.armhfp.log for more details.
[LIVE_IMAGES     ] [INFO    ] [FAIL] Live (variant Server, arch armhfp, subvariant Server) failed, but going on anyway.
[LIVE_IMAGES     ] [INFO    ] LiveImage task failed: 28617663. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180726.n.0/logs/armhfp/liveimage-Fedora-Server-armhfp-Rawhide-20180726.n.0.armhfp.log for more details.
[LIVE_IMAGES     ] [INFO    ] [FAIL] Live (variant Labs, arch armhfp, subvariant Python_Classroom) failed, but going on anyway.
[LIVE_IMAGES     ] [INFO    ] LiveImage task failed: 28617660. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180726.n.0/logs/armhfp/liveimage-Fedora-Python-Classroom-armhfp-Rawhide-20180726.n.0.armhfp.log for more details.
[LIVE_IMAGES     ] [INFO    ] [FAIL] Live (variant Spins, arch armhfp, subvariant Mate) failed, but going on anyway.
[LIVE_IMAGES     ] [INFO    ] LiveImage task failed: 28617685. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180726.n.0/logs/armhfp/liveimage-Fedora-Mate-armhfp-Rawhide-20180726.n.0.armhfp.log for more details.
[LIVE_IMAGES     ] [INFO    ] [FAIL] Live (variant Spins, arch armhfp, subvariant LXQt) failed, but going on anyway.
[LIVE_IMAGES     ] [INFO    ] LiveImage task failed: 28617729. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180726.n.0/logs/armhfp/liveimage-Fedora-LXQt-armhfp-Rawhide-20180726.n.0.armhfp.log for more details.
[LIVE_IMAGES     ] [INFO    ] [FAIL] Live (variant Spins, arch armhfp, subvariant LXDE) failed, but going on anyway.
[LIVE_IMAGES     ] [INFO    ] LiveImage task failed: 28617677. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180726.n.0/logs/armhfp/liveimage-Fedora-LXDE-armhfp-Rawhide-20180726.n.0.armhfp.log for more details.
[LIVE_IMAGES     ] [INFO    ] [FAIL] Live (variant Spins, arch armhfp, subvariant SoaS) failed, but going on anyway.
[LIVE_IMAGES     ] [INFO    ] LiveImage task failed: 28617706. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180726.n.0/logs/armhfp/liveimage-Fedora-SoaS-armhfp-Rawhide-20180726.n.0.armhfp.log for more details.
[LIVE_IMAGES     ] [INFO    ] [FAIL] Live (variant Workstation, arch armhfp, subvariant Workstation) failed, but going on anyway.
[LIVE_IMAGES     ] [INFO    ] LiveImage task failed: 28617738. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180726.n.0/logs/armhfp/liveimage-Fedora-Workstation-armhfp-Rawhide-20180726.n.0.armhfp.log for more details.
[LIVE_MEDIA      ] [INFO    ] [FAIL] Live media (variant Labs, arch *, subvariant Jam_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [INFO    ] Live media task failed: 28617707. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180726.n.0/logs/i386-x86_64/livemedia-Labs-Jam_KDE.i386-x86_64.log for more details.
  • No Task ID, look at log statement
[INFO    ] [FAIL] Image build (variant Cloud, arch s390x, subvariant Cloud_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-Cloud-Base/Rawhide/20180726.n.0/images/Fedora-Cloud-Base-Rawhide-20180726.n.0.ppc64.qcow2 to /mnt/koji/compose/rawhide/Fedora-Rawhide-20180726.n.0/compose/Cloud/ppc64/images/Fedora-Cloud-Base-Rawhide-20180726.n.0.ppc64.qcow2
  • No Task ID, look at log statement
[INFO    ] [FAIL] Image build (variant Labs, arch i386, subvariant Python_Classroom) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-Python-Classroom-Vagrant/Rawhide/20180726.n.0/images/Fedora-Python-Classroom-Vagrant-Rawhide-20180726.n.0.x86_64.vagrant-virtualbox.box to /mnt/koji/compose/rawhide/Fedora-Rawhide-20180726.n.0/compose/Labs/x86_64/images/Fedora-Python-Classroom-Vagrant-Rawhide-20180726.n.0.x86_64.vagrant-virtualbox.box
[IMAGE_BUILD     ] [INFO    ] [FAIL] Image build (variant Workstation, arch *, subvariant Workstation) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] ImageBuild task failed: 28617745. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180726.n.0/logs/aarch64/imagebuild-Workstation-Workstation-raw-xz.aarch64.log for more details.
[IMAGE_BUILD     ] [INFO    ] [FAIL] Image build (variant Server, arch *, subvariant Server) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] ImageBuild task failed: 28617737. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180726.n.0/logs/aarch64/imagebuild-Server-Server-raw-xz.aarch64.log for more details.
  • No Task ID, look at log statement
[INFO    ] [FAIL] Image build (variant Container, arch armhfp, subvariant Container_Base) failed, but going on anyway.
[INFO    ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Base) failed, but going on anyway.
  • No Task ID, look at log statement
[INFO    ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-Container-Base/Rawhide/20180726.n.0/images/Fedora-Container-Base-Rawhide-20180726.n.0.x86_64.tar.xz to /mnt/koji/compose/rawhide/Fedora-Rawhide-20180726.n.0/compose/Container/x86_64/images/Fedora-Container-Base-Rawhide-20180726.n.0.x86_64.tar.xz
  • No Task ID, look at log statement
[INFO    ] [FAIL] Image build (variant Container, arch armhfp, subvariant Container_Minimal_Base) failed, but going on anyway.
[INFO    ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Minimal_Base) failed, but going on anyway.
  • No Task ID, look at log statement
[INFO    ] [FAIL] Image build (variant Container, arch s390x, subvariant Container_Minimal_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [INFO    ] Hardlinking /mnt/koji/packages/Fedora-Container-Minimal-Base/Rawhide/20180726.n.0/images/Fedora-Container-Minimal-Base-Rawhide-20180726.n.0.x86_64.tar.xz to /mnt/koji/compose/rawhide/Fedora-Rawhide-20180726.n.0/compose/Container/x86_64/images/Fedora-Container-Minimal-Base-Rawhide-20180726.n.0.x86_64.tar.xz
  • Compose run failed because: - 28617692
[ERROR   ] Compose run failed: LiveImage task failed: 28617692. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180726.n.0/logs/armhfp/liveimage-Fedora-Minimal-armhfp-Rawhide-20180726.n.0.armhfp.log for more details.

[INFO    ] [FAIL] Ostree (variant AtomicWorkstation, arch x86_64) failed, but going on anyway.
[INFO    ] Runroot task failed: 28616400. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180726.n.0/logs/x86_64/AtomicWorkstation/ostree-4/runroot.log for more details.

This is odd. If you click through to the task you see that koji thinks it actually succeeded. Is pungi detecting things wrong somehow? cc @lsedlar

Overall compose failure is due to pyparted being built with python2 disabled - appliance-creator still needs py2-pyparted. I've poked @dcantrel to re-enable it, if he doesn't get to it in the next hour or two I'll do a build so we can try another compose.

The task indeed succeeded. Pungi detects the success or failure based on exit code of koji cli. According to log it crashed with UnicodeDecodeError. I think the crash is in koji itself.

It was supposed the contents of runroot.log. The offending character is the arrow in [/usr/lib/tmpfiles.d/samba.conf:1] Line references path below legacy directory /var/run/, updating /var/run/samba → /run/samba; please update the tmpfiles.d/ drop-in file accordingly..

I think the crash is in koji itself.

can we open a bug against koji for that?

pyparted-3.11.0-15.fc29 built with python2 enabled again. Sorry for jumping the gun on disabling python2 support in pyparted.

What was the koji (client) version used? Could https://pagure.io/koji/pull-request/828 fix it?

I think this might help us answer that question:

[dustymabe@media koji (upstreammaster=)]$ koji latest-build f27-compose koji
Build                                     Tag                   Built by
----------------------------------------  --------------------  ----------------
koji-1.13.0-4.fc27                        f27                   releng
[dustymabe@media koji (upstreammaster=)]$ koji latest-build f28-compose koji
Build                                     Tag                   Built by
----------------------------------------  --------------------  ----------------
koji-1.15.1-1.fc28                        f28                   puiterwijk

It does not seem like the latest version of koji ( koji-1.16.0) has even been built in koji.

If it is a python3 version (which is now installed as default), it is probably that problem (fix is not released yet, will be part of koji 1.17). But can you try py2 version, if it yields the same problem?

I expect, that it could be reproducible by running 'koji watch-logs' command without running task again.

Interestingly koji watch-logs 28616400 prints everything without problems. That's with koji-1.15.1-1.fc28.noarch on rawhide-composer.phx2.fedoraproject.org.

Interestingly koji watch-logs 28616400 prints everything without problems. That's with koji-1.15.1-1.fc28.noarch on rawhide-composer.phx2.fedoraproject.org.

can you force it to use either py2 or py3 and see if it reproduces with one or the other?

Good idea! It works with python3, but fails with python2.

$ python3 /usr/bin/koji watch-logs 28616400
...
$ echo $?
0
$ python2 /usr/bin/koji watch-logs 28616400
...
DEBUG util.py:439:    spice-glib-0.35-2.fc29.x86_64 (repo-0)
DEBUG util.py:439:    spice-gtk3-0.35-2.fc29.x86_64 (repo-0)
DEBUG util.py:439:    spice-server-0.14.0-5.fc29.x86_64 (repo-0)
DEBUG util.py:439:    spice-vdagent-0.18.0-2.fc29.x86_64 (repo-0)
DEBUG util.py:439:    sqlite-3.24.0-2.fc29.x86_64 (repo-0)
DEBUG util.py:439:    sqlite-libs-3.24.0-2.fc29UnicodeEncodeError: 'ascii' codec can't encode character u'\u2192' in position 10998: ordinal not in range(128)
$ echo $?
1

Good idea! It works with python3, but fails with python2.

are you using the same version of koji on the composer machine? Can someone give us a new build of the koji rpm so that we can test with latest koji?

I tried with koji from git and can still replicate the crash with watch-logs. However applying a similar patch to koji#828 fixes it (while still working on Py3).

diff --git a/cli/koji_cli/lib.py b/cli/koji_cli/lib.py
index 02c6fd2d..328f841c 100644
--- a/cli/koji_cli/lib.py
+++ b/cli/koji_cli/lib.py
@@ -377,7 +377,10 @@ def watch_logs(session, tasklist, opts, poll_interval):
                                 sys.stdout.write("\n")
                             sys.stdout.write("==> %s <==\n" % currlog)
                             lastlog = currlog
-                        sys.stdout.write(contents.decode('utf8'))
+                        if six.PY3:
+                            sys.stdout.buffer.write(contents)
+                        else:
+                            sys.stdout.write(contents)

             if opts.follow:
                 for child in session.getTaskChildren(task_id):

@lsedlar 'From git' you mean master branch? As I'm not able to replicate it with current head (https://copr.fedorainfracloud.org/coprs/tkopecek/koji/build/781549/), both pythons work for me here.

@tkopecek Yup, from master.

$ git describe 
koji-1.16.0-82-ge50cd283
$ python2 --version
Python 2.7.15
$ python3 --version
Python 3.6.6

Login to comment on this ticket.

Metadata