A fix to 390x Koji builders was applied in 1b9567b, please test if the build works now.

This seems to be working ok now to me:

https://koji.fedoraproject.org/koji/taskinfo?taskID=32993152

Must have been a transitory failure?

:dragon:

It's here again:
https://koji.fedoraproject.org/koji/taskinfo?taskID=33115191
https://koji.fedoraproject.org/koji/taskinfo?taskID=33115228

This is a different issue.

I think I have fixed it, can you resubmit and confirm?

So, I did a test here and it failed, but looks like compiler issues in f31 perhaps?
https://koji.fedoraproject.org/koji/taskinfo?taskID=33491398

I think I have isolated that unpacking error you were seeing to mbs using new style chroots (systemd-nspawn) and something not cleaning them up, so they fill up resources as they pile up.

mbs is going to change that soon and I am manually cleaning them, so IMHO this issue should be fixed.

:black_square_button:

I'm going re-enable s390x for the next builds.

It's here again - Fedora 28:
https://kojipkgs.fedoraproject.org//work/tasks/711/33600711/root.log

My theory as to what was causing the unpack errors I guess was incorrect.

I have no other ideas what this could be.

@sharkcz any s390x specific ideas?

It sounds like communication error from the network. Wonder if the proxies could have any effect or if there is anything in the web server log on the hub (where the srpm is downloaded from). I suppose the builders look good otherwise - enough free disk space and so on.

I don't think it's a network issue. It looks like it downloaded and generated the src.rpm, then when it went to install it to build it, it got the error. So, some sort of unpacking error.

Wrote: /builddir/build/SRPMS/firefox-66.0-4.fc28.src.rpm
Child return code was: 0
...
DEBUG util.py:643: Executing command: ['/bin/rpm', '-Uvh', '--nodeps', '/builddir/build/originals/firefox-66.0-4.fc28.src.rpm'] with env {'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOME': '/builddir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'PROMPT_COMMAND': 'printf "\033]0;<mock-chroot>\007"', 'PS1': '<mock-chroot> \s-\v\$ ', 'LANG': 'C.UTF-8'} and shell False
DEBUG util.py:389: Unsharing. Flags: 134217728
DEBUG util.py:556: Updating / installing...
DEBUG util.py:556: firefox-66.0-4.fc28 ########################################
DEBUG util.py:554: BUILDSTDERR: error: unpacking of archive failed on file /builddir/build/SOURCES/firefox-66.0.source.tar.xz;5c8f5fbd: cpio: read failed - Inappropriate ioctl for device
DEBUG util.py:554: BUILDSTDERR: error: /builddir/build/originals/firefox-66.0-4.fc28.src.rpm cannot be installed

It is a 313MB src.rpm, but still, it was all local...

AFAIK the srpm got created in https://koji.fedoraproject.org/koji/taskinfo?taskID=33600375 (buildSRPMFromSCM), uploaded to the hub, the s390x builder downloads it from the hub and installs, where it fails. IMHO the srpm got corrupted/truncated/... during the transfer.

For confirmation we would need access to the chroot with the failed build, IIRC there is 4 hour window defined in koji before the chroot will be removed.

Ah, right you are.

Odd then that it didn't error on the download?

Yeah, we need to get to a root where this fails before the timout and it's cleaned up.

I think a cache can mask an incomplete transfer, IIRC I saw that with squid and forced reload of given URL was required then. It could be a bit flipped in otherwise complete file too.

We have done some additional work now (lowered the weight on s390x builders so they don't take on multiple jobs at once) and adjusted the caching more.

@stransky can you try a build with s390x re-enabled again now?

It's still broken:
https://koji.fedoraproject.org/koji/taskinfo?taskID=34573603
https://koji.fedoraproject.org/koji/taskinfo?taskID=34573599

I've filed a koji issue on this:

https://pagure.io/koji/issue/1418

Hopefully they can come up with some mitigation/fix for this. Please follow along on that ticket...

Thanks, and sorry for all the hassle.