dusty / failed-composes

Clone
Source Code
GIT

#15 Fedora-Rawhide-20180224.n.0 DOOMED
Opened 6 years ago by dustymabe. Modified 5 years ago

Open
Close issue as:
fixed invalid timeout killed known_issue

pungi.global.log

[LIVE_MEDIA      ] [INFO    ] [FAIL] Live media (variant Labs, arch *, subvariant Astronomy_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [INFO    ] Live media task failed: 25278683. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180224.n.0/logs/i386-x86_64/livemedia-Labs-Astronomy_KDE.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [INFO    ] [FAIL] Live media (variant Labs, arch *, subvariant Scientific_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [INFO    ] Live media task failed: 25278693. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180224.n.0/logs/i386-x86_64/livemedia-Labs-Scientific_KDE.i386-x86_64.log for more details.

[LIVE_MEDIA      ] [INFO    ] [FAIL] Live media (variant Labs, arch *, subvariant Robotics) failed, but going on anyway.
[LIVE_MEDIA      ] [INFO    ] Live media task failed: 25278727. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180224.n.0/logs/i386-x86_64/livemedia-Labs-Robotics.i386-x86_64.log for more details.

[LIVE_IMAGES     ] [INFO    ] [FAIL] Live (variant Labs, arch armhfp, subvariant Python_Classroom) failed, but going on anyway.
[LIVE_IMAGES     ] [INFO    ] LiveImage task failed: 25278676. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180224.n.0/logs/armhfp/liveimage-Fedora-Python-Classroom-armhfp-Rawhide-20180224.n.0.armhfp.log for more details.

[LIVE_IMAGES     ] [INFO    ] [FAIL] Live (variant Spins, arch armhfp, subvariant KDE) failed, but going on anyway.
[LIVE_IMAGES     ] [INFO    ] LiveImage task failed: 25278682. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180224.n.0/logs/armhfp/liveimage-Fedora-KDE-armhfp-Rawhide-20180224.n.0.armhfp.log for more details.

[LIVE_IMAGES     ] [INFO    ] [FAIL] Live (variant Spins, arch armhfp, subvariant SoaS) failed, but going on anyway.
[LIVE_IMAGES     ] [INFO    ] LiveImage task failed: 25278711. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180224.n.0/logs/armhfp/liveimage-Fedora-SoaS-armhfp-Rawhide-20180224.n.0.armhfp.log for more details.

[LIVE_IMAGES     ] [INFO    ] [FAIL] Live (variant Spins, arch armhfp, subvariant LXQt) failed, but going on anyway.
[LIVE_IMAGES     ] [INFO    ] LiveImage task failed: 25278721. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180224.n.0/logs/armhfp/liveimage-Fedora-LXQt-armhfp-Rawhide-20180224.n.0.armhfp.log for more details.

[LIVE_IMAGES     ] [INFO    ] [FAIL] Live (variant Spins, arch armhfp, subvariant LXDE) failed, but going on anyway.
[LIVE_IMAGES     ] [INFO    ] LiveImage task failed: 25278687. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180224.n.0/logs/armhfp/liveimage-Fedora-LXDE-armhfp-Rawhide-20180224.n.0.armhfp.log for more details.

[LIVE_IMAGES     ] [INFO    ] [FAIL] Live (variant Spins, arch armhfp, subvariant Mate) failed, but going on anyway.
[LIVE_IMAGES     ] [INFO    ] LiveImage task failed: 25278692. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180224.n.0/logs/armhfp/liveimage-Fedora-Mate-armhfp-Rawhide-20180224.n.0.armhfp.log for more details.

[LIVE_IMAGES     ] [INFO    ] [FAIL] Live (variant Workstation, arch armhfp, subvariant Workstation) failed, but going on anyway.
[LIVE_IMAGES     ] [INFO    ] LiveImage task failed: 25278740. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180224.n.0/logs/armhfp/liveimage-Fedora-Workstation-armhfp-Rawhide-20180224.n.0.armhfp.log for more details.
  • Compose run failed because:
[ERROR   ] Compose run failed: LiveImage task failed: 25278705. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20180224.n.0/logs/armhfp/liveimage-Fedora-Xfce-armhfp-Rawhide-20180224.n.0.armhfp.log for more details.

astronomy scientific robotics all fail because of missing deps.. i'm not going to worry about those right now.

koji task 25278678 Fedora-Python-Classroom-armhfp

DEBUG util.py:439: At least 52MB more space needed on the / filesystem.

koji task: 25278682 Spins armhfp KDE

DEBUG util.py:439: At least 104MB more space needed on the / filesystem.

These two should be fixed by https://pagure.io/fedora-kickstarts/pull-request/343

The rest failed because of some obscure error.. adam/patrick/dennis were looking at it earlier today:

DEBUG util.py:439: imgcreate.errors.MountError: Failed to unmap partitions for '/dev/loop0'

The rest failed because of some obscure error.. adam/patrick/dennis were looking at it earlier today

from dennis:

10:51:40      +dgilmore | the issue is that umounting /sys in the install tree fails
10:51:46      +dgilmore | I can reproduce locally
10:51:57      +dgilmore | but do not get anything indicating why
10:52:08      +dgilmore | and on investigation it is umounted

I don't think that's right. It's not the umount of /sys that's the problem really, because a couple of ARM image composes succeed, and the umount of /sys fails on those:

Note that both have this in their logs:

DEBUG util.py:439:  umount: /var/tmp/imgcreate-Ibmsz1/install_root/sys: target is busy.
DEBUG util.py:439:  Unable to unmount /var/tmp/imgcreate-Ibmsz1/install_root/sys normally, using lazy unmount
DEBUG util.py:439:  lazy umount succeeded on /var/tmp/imgcreate-Ibmsz1/install_root/sys
DEBUG util.py:439:  Unmounting directory /var/tmp/imgcreate-Ibmsz1/install_root/boot/fw
DEBUG util.py:439:  Unmounting directory /var/tmp/imgcreate-Ibmsz1/install_root/boot
DEBUG util.py:439:  Unmounting directory /var/tmp/imgcreate-Ibmsz1/install_root/
DEBUG util.py:439:  Removing compat symlinks
DEBUG util.py:439:  Unmapping /dev/loop1
DEBUG util.py:439:  Losetup remove /dev/loop1
DEBUG util.py:439:  moving disks to stage location
DEBUG util.py:439:  compression successful
DEBUG util.py:439:  moving /var/tmp/imgcreate-Ibmsz1/tmp-0ZofV2/Fedora-Server-armhfp-Rawhide-20180224.n.0-sda.raw.xz to /var/tmp/imgcreate-Ibmsz1/out/Fedora-Server-armhfp-Rawhide-20180224.n.0-sda.raw.xz
DEBUG util.py:439:  writing image XML to /var/tmp/imgcreate-Ibmsz1/out/Fedora-Server-armhfp-Rawhide-20180224.n.0.xml
DEBUG util.py:439:  creating destination dir: app-output/Fedora-Server-armhfp-Rawhide-20180224.n.0
DEBUG util.py:439:  moving /var/tmp/imgcreate-Ibmsz1/out/Fedora-Server-armhfp-Rawhide-20180224.n.0.xml to app-output/Fedora-Server-armhfp-Rawhide-20180224.n.0/Fedora-Server-armhfp-Rawhide-20180224.n.0.xml
DEBUG util.py:439:  moving /var/tmp/imgcreate-Ibmsz1/out/Fedora-Server-armhfp-Rawhide-20180224.n.0-sda.raw.xz to app-output/Fedora-Server-armhfp-Rawhide-20180224.n.0/Fedora-Server-armhfp-Rawhide-20180224.n.0-sda.raw.xz
DEBUG util.py:439:   
DEBUG util.py:439:  lazy umount succeeded on /var/tmp/imgcreate-Ibmsz1/install_root/sys
DEBUG util.py:439:  Finished
DEBUG util.py:577:  Child return code was: 0

The umount of /sys doesn't work (it lazy umounts it instead), but the umount of the other filesystems in the install root succeeds.

In comparison, all the other image builds - all ones with a graphical desktop, which may be significant - fail. The relevant sections of their logs look like this:

DEBUG util.py:439:  umount: /var/tmp/imgcreate-8B0C9r/install_root/sys: target is busy.
DEBUG util.py:439:  Unable to unmount /var/tmp/imgcreate-8B0C9r/install_root/sys normally, using lazy unmount
DEBUG util.py:439:  lazy umount succeeded on /var/tmp/imgcreate-8B0C9r/install_root/sys
DEBUG util.py:439:  Unmounting directory /var/tmp/imgcreate-8B0C9r/install_root/boot/fw
DEBUG util.py:439:  Unmounting directory /var/tmp/imgcreate-8B0C9r/install_root/boot
DEBUG util.py:439:  Unmounting directory /var/tmp/imgcreate-8B0C9r/install_root/
DEBUG util.py:439:  umount: /var/tmp/imgcreate-8B0C9r/install_root/: target is busy.
DEBUG util.py:439:  Unmounting directory /var/tmp/imgcreate-8B0C9r/install_root/ failed, using lazy umount
DEBUG util.py:439:  lazy umount succeeded on /var/tmp/imgcreate-8B0C9r/install_root/
DEBUG util.py:439:  Removing compat symlinks
DEBUG util.py:439:  Unmapping /dev/loop0
DEBUG util.py:439:  Unable to create appliance : Failed to unmap partitions for '/dev/loop0'
DEBUG util.py:439:  Removing compat symlinks
DEBUG util.py:439:  Unmapping /dev/loop0
DEBUG util.py:439:   
DEBUG util.py:439:  lazy umount succeeded on /var/tmp/imgcreate-8B0C9r/install_root/sys
DEBUG util.py:439:  Unmounting directory /var/tmp/imgcreate-8B0C9r/install_root/ failed, using lazy umount
DEBUG util.py:439:  lazy umount succeeded on /var/tmp/imgcreate-8B0C9r/install_root/
DEBUG util.py:439:  Traceback (most recent call last):
DEBUG util.py:439:    File "/usr/bin/appliance-creator", line 173, in <module>
DEBUG util.py:439:      sys.exit(main())
DEBUG util.py:439:    File "/usr/bin/appliance-creator", line 157, in main
DEBUG util.py:439:      creator.cleanup()
DEBUG util.py:439:    File "/usr/lib/python2.7/site-packages/imgcreate/creator.py", line 615, in cleanup
DEBUG util.py:439:      self.unmount()
DEBUG util.py:439:    File "/usr/lib/python2.7/site-packages/imgcreate/creator.py", line 589, in unmount
DEBUG util.py:439:      self._unmount_instroot()
DEBUG util.py:439:    File "/usr/lib/python2.7/site-packages/appcreate/appliance.py", line 539, in _unmount_instroot
DEBUG util.py:439:      self.__instloop.cleanup()
DEBUG util.py:439:    File "/usr/lib/python2.7/site-packages/appcreate/partitionedfs.py", line 220, in cleanup
DEBUG util.py:439:      self.__unmap_partitions()
DEBUG util.py:439:    File "/usr/lib/python2.7/site-packages/appcreate/partitionedfs.py", line 203, in __unmap_partitions
DEBUG util.py:439:      d['disk'].device)
DEBUG util.py:439:  imgcreate.errors.MountError: Failed to unmap partitions for '/dev/loop0'
DEBUG util.py:439:  Removing compat symlinks
DEBUG util.py:439:  Unmapping /dev/loop0
DEBUG util.py:439:  Exception imgcreate.errors.MountError: MountError(u"Failed to unmap partitions for '/dev/loop0'",) in <bound method ApplianceImageCreator.__del__ of <appcreate.appliance.ApplianceImageCreator object at 0xb3db5db0>> ignored
DEBUG util.py:577:  Child return code was: 1

Note not just the umount of /sys fails, but also the umount of / within the install root (/var/tmp/imgcreate-Ibmsz1/install_root/ from outside). That appears to be the one that's actually fatal.

I think we'd need to do the thing where we make it log fuser and stuff to try and figure out why the filesystem is still busy. Or, as @puiterwijk suggested, just identify the group of processes related to the install and kill them all, or something...

This was finally pointed out to me by @davidlt on IRC, and the digging he did indicates it's related to libnss-systemd still being active at the end.

From what we can discern, it's probably caused by this: https://pagure.io/appliance-tools/c/398360b2b5e86072cee058e0e3d2eb9a74eb158e

It was originally added to appliance-tools several years back: https://src.fedoraproject.org/rpms/appliance-tools/c/ed41d49e301ff3e3dcc80222611ab5d245c23db5

Otherwise, I'm not sure what causes nss to be active in the install root...

Do we still need this hack?

Edited 5 years ago by ngompa

I don't think it's that, it looks like it's sssd: see https://bugzilla.redhat.com/show_bug.cgi?id=1591804

Login to comment on this ticket.

Metadata
None
None
None
None
Powered by Pagure 5.13.3
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.