#1997 Fedora-Rawhide-20190612.n.0 DOOMED
Opened 5 months ago by dustymabe. Modified 5 months ago

pungi.global.log

[ERROR   ] [FAIL] Ostree (variant Silverblue, arch x86_64) failed, but going on anyway.
[ERROR   ] Runroot task failed: 35495823. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/x86_64/Silverblue/ostree-1/runroot.log for more details.
[ERROR   ] [FAIL] Ostree (variant Silverblue, arch ppc64le) failed, but going on anyway.
[ERROR   ] Runroot task failed: 35495821. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/ppc64le/Silverblue/ostree-2/runroot.log for more details.
[ERROR   ] [FAIL] Ostree (variant Silverblue, arch aarch64) failed, but going on anyway.
[ERROR   ] Runroot task failed: 35495822. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/aarch64/Silverblue/ostree-3/runroot.log for more details.
[OSTREE_INSTALLER] [ERROR   ] [FAIL] Ostree installer (variant Silverblue, arch ppc64le) failed, but going on anyway.
[OSTREE_INSTALLER] [ERROR   ] Runroot task failed: 35495839. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/ppc64le/Silverblue/ostree_installer-2/runroot.log for more details.
[OSTREE_INSTALLER] [ERROR   ] [FAIL] Ostree installer (variant Silverblue, arch aarch64) failed, but going on anyway.
[OSTREE_INSTALLER] [ERROR   ] Runroot task failed: 35495841. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/aarch64/Silverblue/ostree_installer-1/runroot.log for more details.
[OSTREE_INSTALLER] [ERROR   ] [FAIL] Ostree installer (variant Silverblue, arch x86_64) failed, but going on anyway.
[OSTREE_INSTALLER] [ERROR   ] Runroot task failed: 35495840. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/x86_64/Silverblue/ostree_installer-3/runroot.log for more details.
[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Labs, arch armhfp, subvariant Python_Classroom) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 35500297. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/armhfp/liveimage-Fedora-Python-Classroom-armhfp-Rawhide-20190612.n.0.armhfp.log for more details.
[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Server, arch armhfp, subvariant Server) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 35500299. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/armhfp/liveimage-Fedora-Server-armhfp-Rawhide-20190612.n.0.armhfp.log for more details.
[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant KDE) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 35500302. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/armhfp/liveimage-Fedora-KDE-armhfp-Rawhide-20190612.n.0.armhfp.log for more details.
[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant LXDE) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 35500308. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/armhfp/liveimage-Fedora-LXDE-armhfp-Rawhide-20190612.n.0.armhfp.log for more details.
[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant Mate) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 35500312. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/armhfp/liveimage-Fedora-Mate-armhfp-Rawhide-20190612.n.0.armhfp.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Astronomy_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 35500300. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/i386-x86_64/livemedia-Labs-Astronomy_KDE.i386-x86_64.log for more details.
[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant SoaS) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 35500317. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/armhfp/liveimage-Fedora-SoaS-armhfp-Rawhide-20190612.n.0.armhfp.log for more details.
[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Spins, arch armhfp, subvariant LXQt) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 35500335. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/armhfp/liveimage-Fedora-LXQt-armhfp-Rawhide-20190612.n.0.armhfp.log for more details.
[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Cloud, arch *, subvariant Cloud_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 35500301. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/x86_64/imagebuild-Cloud-Cloud_Base-vagrant-libvirt-vagrant-virtualbox.x86_64.log for more details.
[LIVE_IMAGES     ] [ERROR   ] [FAIL] Live (variant Workstation, arch armhfp, subvariant Workstation) failed, but going on anyway.
[LIVE_IMAGES     ] [ERROR   ] LiveImage task failed: 35500340. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/armhfp/liveimage-Fedora-Workstation-armhfp-Rawhide-20190612.n.0.armhfp.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Design_suite) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 35500307. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/i386-x86_64/livemedia-Labs-Design_suite.i386-x86_64.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Scientific_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 35500310. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/i386-x86_64/livemedia-Labs-Scientific_KDE.i386-x86_64.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Games) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 35500313. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/i386-x86_64/livemedia-Labs-Games.i386-x86_64.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Security) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 35500316. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/i386-x86_64/livemedia-Labs-Security.i386-x86_64.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Robotics) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 35500333. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/i386-x86_64/livemedia-Labs-Robotics.i386-x86_64.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Jam_KDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 35500325. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/i386-x86_64/livemedia-Labs-Jam_KDE.i386-x86_64.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant Xfce) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 35500346. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/i386-x86_64/livemedia-Spins-Xfce.i386-x86_64.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Labs, arch *, subvariant Python_Classroom) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 35500338. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/i386-x86_64/livemedia-Labs-Python_Classroom.i386-x86_64.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant LXDE) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 35500353. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/i386-x86_64/livemedia-Spins-LXDE.i386-x86_64.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant SoaS) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 35500349. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/i386-x86_64/livemedia-Spins-SoaS.i386-x86_64.log for more details.
[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Labs, arch *, subvariant Scientific) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 35500318. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/i386-x86_64/imagebuild-Labs-Scientific-vagrant-libvirt-vagrant-virtualbox.i386-x86_64.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant Cinnamon) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 35500352. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/i386-x86_64/livemedia-Spins-Cinnamon.i386-x86_64.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant Mate) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 35500354. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/i386-x86_64/livemedia-Spins-Mate.i386-x86_64.log for more details.
[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Server, arch *, subvariant Server) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 35500327. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/aarch64-armhfp/imagebuild-Server-Server-raw-xz.aarch64-armhfp.log for more details.
[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Workstation, arch *, subvariant Workstation) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 35500339. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/aarch64-armhfp/imagebuild-Workstation-Workstation-raw-xz.aarch64-armhfp.log for more details.
[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Labs, arch *, subvariant Python_Classroom) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 35500314. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/i386-x86_64/imagebuild-Labs-Python_Classroom-vagrant-libvirt-vagrant-virtualbox.i386-x86_64.log for more details.
[LIVE_MEDIA      ] [ERROR   ] [FAIL] Live media (variant Spins, arch *, subvariant LXQt) failed, but going on anyway.
[LIVE_MEDIA      ] [ERROR   ] Live media task failed: 35500355. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/i386-x86_64/livemedia-Spins-LXQt.i386-x86_64.log for more details.
[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Spins, arch *, subvariant Minimal) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 35500334. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/aarch64-armhfp/imagebuild-Spins-Minimal-raw-xz.aarch64-armhfp.log for more details.
[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Container, arch *, subvariant Container_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 35500309. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/aarch64-armhfp-ppc64le-s390x-x86_64/imagebuild-Container-Container_Base-docker.aarch64-armhfp-ppc64le-s390x-x86_64.log for more details.
[IMAGE_BUILD     ] [ERROR   ] [FAIL] Image build (variant Container, arch *, subvariant Container_Minimal_Base) failed, but going on anyway.
[IMAGE_BUILD     ] [ERROR   ] ImageBuild task failed: 35500311. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/aarch64-armhfp-ppc64le-s390x-x86_64/imagebuild-Container-Container_Minimal_Base-docker.aarch64-armhfp-ppc64le-s390x-x86_64.log for more details.
  • Compose run failed because: - 35500315
[ERROR   ] Compose run failed: LiveImage task failed: 35500315. See /mnt/koji/compose/rawhide/Fedora-Rawhide-20190612.n.0/logs/armhfp/liveimage-Fedora-Minimal-armhfp-Rawhide-20190612.n.0.armhfp.log for more details.

Okay, this is new:

ApplianceError: Appliance functions not available

This is almost for sure my upgrading things. ;(

koji, oz, imagefactory, createrepo_c, mock and others upgraded. koji and friends moved to python3.

I'll try and look, but if anyone else can, that might be quicker.

I think I have fixed this. Should hopefully work on the next run...

@kevin Fedora-Rawhide-20190613.n.0 is also failed due to same reason.

https://koji.fedoraproject.org/koji/taskinfo?taskID=35519017

And something went wrong with the Dusty's compose-tracker, he is debugging it.

Bummer. Well, things moved to python3... perhaps @imcleod could help us?

Otherwise I can debug when I get back... oz and imagefactory are both python3 now as is koji-builder

So, looking at:

https://koji.fedoraproject.org/koji/taskinfo?taskID=35519017

This failed task is actually the "appliance" method which I believe is an entirely different code path, distinct from the bits that call into imagefactory. I'm afraid I'm not familiar with it.

I believe it uses this: https://pagure.io/appliance-tools/tree/master

@kevin we moved imagefactory and oz to Python3 but don't we also need to move imagefactory-plugins to Python3 as well? I believe imagefactory-plugins contains sub-packages like imagefactory-plugins-TinMan , imagefactory-plugins-Docker that are needed to build cloud/container images. It was needed at least when I tried locally running imagefactory from comandline

I tried to port and build imagefactory-plugins to Python3, after few hacks(disabled HyperV to get through mixed use of TAB and spaces error) koji build succeeded. But, imagefactory run (from binary) failed for me due to not getting plugin. I don't have much experience in porting to Python3 and I might have missed something.

So @imcleod should/could assist with the imagefactory-plugins py3 bits too

The problem with appliances was pycdio. It was/is shipped python2 only in Fedora packages, but luckily upstream supports python3 just fine, so I was able to rebuild it and add it to our f29-infra tags.
TODO: move fedora package to python3.

The problem on imagefactory was indeed the python2 plugins package. I have rebuilt and added that.

I'm going to restart builders and fire a new rawhide after that...

so, appliances seem ok now, but cloud images not. ;(

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/koji/daemon.py", line 1295, in runTask
    response = (handler.run(),)
  File "/usr/lib/python3.7/site-packages/koji/tasks.py", line 311, in run
    return koji.util.call_with_argcheck(self.handler, self.params, self.opts)
  File "/usr/lib/python3.7/site-packages/koji/util.py", line 263, in call_with_argcheck
    return func(*args, **kwargs)
  File "/usr/sbin/kojid", line 4172, in handler
    images = self.do_images(ks, template, inst_tree)
  File "/usr/sbin/kojid", line 3807, in do_images
    self.base_img = self._buildBase(template, params)
  File "/usr/sbin/kojid", line 3915, in _buildBase
    self._checkImageState(base)
  File "/usr/sbin/kojid", line 3863, in _checkImageState
    image.os_plugin.abort() # forcibly tear down the VM
AttributeError: 'NoneType' object has no attribute 'abort'

https://koji.fedoraproject.org/koji/taskinfo?taskID=35567136

Everything is python3, so this could be some change due to the python3 move that needs to be made in kojid?

Do we have better logs anywhere? image in this case is an instance of imgfac's Builder class; it looks to me like a plausible scenario where image.os_plugin would be None is if we hit this _build_image_from_template method and something goes wrong; in that case it looks like os_plugin wouldn't get set to anything and would remain as None (which is what it's originally initialized to). But it's hard to tell if we don't have a full log...

Even running imagefactory directly doesn't go well after using Pthon3 packages imagefactory-1.1.13-0.20190527193658gita117084.fc29.infra.noarch.rpm , imagefactory-plugins-TinMan-1.1.13-0.20190528024255gita117084.fc29.infra.noarch.rpm, imagefactory-plugins-1.1.13-0.20190528024255gita117084.fc29.infra.noarch.rpm and oz-0.17.0-4.fc29.infra.noarch.rpm

$ imagefactory --timeout 14440 --debug base_image --file-parameter install_script koji-f29-build-35450104-base.ks --parameter offline_icicle true tdl-ppc64le.xml

...
2019-06-17 06:58:33,025 ERROR imgfac.Builder.Builder thread(908f3b57) Message: Exception encountered in _build_image_from_template thread
2019-06-17 06:58:33,025 ERROR imgfac.Builder.Builder thread(908f3b57) Message: 'NoneType' object has no attribute 'create_base_image'
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/imgfac/Builder.py", line 132, in _build_image_from_template
    self.os_plugin.create_base_image(self, template, parameters)
AttributeError: 'NoneType' object has no attribute 'create_base_image'
2019-06-17 06:58:33,025 DEBUG imgfac.PersistentImage.BaseImage thread(MainThread) Message: Getting metadata in class (<class 'imgfac.BaseImage.BaseImage'>) my metadata is (())
2019-06-17 06:58:33,025 DEBUG imgfac.PersistentImage.BaseImage thread(MainThread) Message: Executing metadata in class (<class 'imgfac.BaseImage.BaseImage'>) my metadata is (('identifier', 'data', 'template', 'icicle', 'status_detail', 'status', 'percent_complete', 'parameters', 'properties'))

Image build FAILED with error: 'NoneType' object has no attribute 'create_base_image'

It seems like after Python3 porting something is not working as expected. [plugin_for_target( https://github.com/redhat-imaging/imagefactory/blob/master/imgfac/Builder.py#L130) function returns None instead of Plugin object. That's why it fails while calling create_base_image() on a NoneType object

Can you please post the entire log including the lines from ...?

while pasting the complete log, noticed another error which was on top

2019-06-17 15:44:11,549 ERROR imgfac.PluginManager.PluginManager thread(MainThread) Message: Exception caught while loading plugin metadata: [Errno 2] No such file or directory: '/etc/imagefactory/plugins.d/TinMan.info'
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/imgfac/PluginManager.py", line 112, in metadata_for_plugin
    fp = open(os.path.join(self.path, info_file), 'r')
FileNotFoundError: [Errno 2] No such file or directory: '/etc/imagefactory/plugins.d/TinMan.info'
2019-06-17 15:44:11,549 ERROR imgfac.PluginManager.PluginManager thread(MainThread) Message: Loading plugin (TinMan) failed with exception: 'NoneType' object is not subscriptable
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/imgfac/PluginManager.py", line 71, in load
    if md['type'].upper() in PLUGIN_TYPES:
TypeError: 'NoneType' object is not subscriptable

When I looked at /etc/imagefactory/plugins.d/TinMan.info, it is a symlink file . From imagefactory-plugins spec file it seems like if a symlink already exists, it does nothing (noticed same in infra f29-infra tag spec too). I already had imagefactory-plugins (Python2 one) installed and hence symlink already existed pointing to /usr/lib/python2.7/site-packages/imagefactory_plugins/TinMan/TinMan.info . I changed symlink to point to /usr/lib/python3.7/site-packages/imagefactory_plugins/TinMan/TinMan.info . After that imagefactory run is progressing so far fine.

In spec file, we may want to update symlink too so that whoever have already installed imagefactory-plugins with Python2 can switch smoothly to Python3. This may also be the cause in our koji image builders.

I'm rebuilding a fixed package. Great investigation @sinnykumari !

ok. I think that fixed things... I have a cloud compose thats been running fine for a while.

I am going to fire another rawhide off and see where we are.

KDE live failed on dep issues in most recent compose :/

Login to comment on this ticket.

Metadata