#9562 cloud/container image creation fails on s390x
Closed: Fixed 3 years ago by mohanboddu. Opened 3 years ago by sharkcz.

  • Describe the issue
    The createImage task in koji fails on the builders in the image channel with a traceback like bellow
traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/koji/daemon.py", line 1340, in runTask
    response = (handler.run(),)
  File "/usr/lib/python3.8/site-packages/koji/tasks.py", line 329, in run
    return koji.util.call_with_argcheck(self.handler, self.params, self.opts)
  File "/usr/lib/python3.8/site-packages/koji/util.py", line 258, in call_with_argcheck
    return func(*args, **kwargs)
  File "/usr/sbin/kojid", line 4359, in handler
    images = self.do_images(ks, template, inst_tree)
  File "/usr/sbin/kojid", line 3986, in do_images
    self.base_img = self._buildBase(template, params)
  File "/usr/sbin/kojid", line 4095, in _buildBase
    self._checkImageState(base)
  File "/usr/sbin/kojid", line 4042, in _checkImageState
    image.os_plugin.abort()  # forcibly tear down the VM
AttributeError: 'NoneType' object has no attribute 'abort'

https://koji.fedoraproject.org/koji/taskinfo?taskID=46373155 = buildvm-s390x-19
https://koji.fedoraproject.org/koji/taskinfo?taskID=46373238 = buildvm-s390x-20

  • When do you need this? (YYYY/MM/DD)
    ASAP

  • When is this no longer needed or useful? (YYYY/MM/DD)
    N/A

  • If we cannot complete your request, what is the impact?
    We cannot deliver cloud and container images for s390x.


Isn't koji and oz/imagefactory out of sync on the builders?

It started when koji was updated to 1.21 and the hub moved to IAD2. So likely one of those caused it.

We think its a packaging bug in imagefactory, so I removed and installed it on all s390x vm's. That should fix the issue. If it works tomorrow, we will close the ticket.

The issue is still there :-(

I wonder whether there is an underlying problem with qemu/libvirt/oz/imagefactory in F-32 and the error in kojid is only a symptom. Trying imagefactory locally ...

running imagefactory locally on F-32 ends with

2020-07-01 12:42:23,934 ERROR imgfac.Builder.Builder thread(ba441d1e) Message: unsupported configuration: ps2 is not supported by this QEMU binary
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/imgfac/Builder.py", line 135, in _build_image_from_template
    self.os_plugin.create_base_image(self, template, parameters)
  File "/usr/lib/python3.8/site-packages/imagefactory_plugins/TinMan/TinMan.py", line 344, in create_base_image
    libvirt_xml = self.guest.install(self.app_config["timeout"])
  File "/usr/lib/python3.8/site-packages/oz/RedHat.py", line 715, in install
    return self._do_install(timeout, force, 0, self.kernelfname,
  File "/usr/lib/python3.8/site-packages/oz/Guest.py", line 1606, in _do_install
    self._wait_for_install_finish(xml, timeout)
  File "/usr/lib/python3.8/site-packages/oz/Guest.py", line 778, in _wait_for_install_finish
    libvirt_dom = self.libvirt_conn.createXML(xml, 0)
  File "/usr/lib64/python3.8/site-packages/libvirt.py", line 4035, in createXML
    if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
libvirt.libvirtError: unsupported configuration: ps2 is not supported by this QEMU binary

Image build FAILED with error: unsupported configuration: ps2 is not supported by this QEMU binary

and after patching oz/Guest.py to omit mouse on s390x, it finished OK with an image produced.

Hm, so we get the same error even with fixed/updated oz.

CC @tkopecek, could you look at the bug from a koji perspective, please? Looks to me that the image.os_plugin member is None for some reason.

@sharkcz Is imagefactory loading ok on f32? I've tried to start kojid with that now and I'm getting a lot of problems with plugins in the beggining (investigating now).

Hmm, upgrade seems to break symlinks in /etc/imagefactory/plugins.d/ - they are not correctly replaced with newer python (3.8) paths. Not sure if it is also your case.

@tkopecek, standalone imagefactoryworked well for me on F-32.

The plugin upgrade issue is known and should be already fixed on the koji builders.

@breilly Is it something you've seen?

We have updated oz to include https://github.com/clalancette/oz/pull/282, but we still see failures. But what's interesting, there are occasions when the tasks succeed, same builder has good and bad runs ...

Not sure what changed, but now we are getting the images.

Closing the ticket.

Metadata Update from @mohanboddu:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

3 years ago

Login to comment on this ticket.

Metadata