#7581 Firefox build failure on s390x/F28 only - can't get sources
Closed: Upstream 3 months ago by kevin. Opened 6 months ago by stransky.

Log:
https://kojipkgs.fedoraproject.org//work/tasks/2573/32902573/root.log

Failure:
error: unpacking of archive failed on file /builddir/build/SOURCES/firefox-65.0.1.source.tar.xz;5c6bca60: cpio: read failed - Inappropriate ioctl for device

  • If we cannot complete your request, what is the impact?
    Firefox/s390x builds won't be available for Fedora 28

A fix to 390x Koji builders was applied in 1b9567b, please test if the build works now.

Metadata Update from @mizdebsk:
- Issue priority set to: Waiting on Reporter (was: Needs Review)

6 months ago

Metadata Update from @mizdebsk:
- Issue tagged with: koji

6 months ago

This seems to be working ok now to me:

https://koji.fedoraproject.org/koji/taskinfo?taskID=32993152

Must have been a transitory failure?

:dragon:

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

6 months ago

Metadata Update from @stransky:
- Issue status updated to: Open (was: Closed)

5 months ago

This is a different issue.

I think I have fixed it, can you resubmit and confirm?

So, I did a test here and it failed, but looks like compiler issues in f31 perhaps?
https://koji.fedoraproject.org/koji/taskinfo?taskID=33491398

I think I have isolated that unpacking error you were seeing to mbs using new style chroots (systemd-nspawn) and something not cleaning them up, so they fill up resources as they pile up.

mbs is going to change that soon and I am manually cleaning them, so IMHO this issue should be fixed.

:black_square_button:

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

5 months ago

I'm going re-enable s390x for the next builds.

Metadata Update from @stransky:
- Issue status updated to: Open (was: Closed)

5 months ago

My theory as to what was causing the unpack errors I guess was incorrect.

I have no other ideas what this could be.

@sharkcz any s390x specific ideas?

It sounds like communication error from the network. Wonder if the proxies could have any effect or if there is anything in the web server log on the hub (where the srpm is downloaded from). I suppose the builders look good otherwise - enough free disk space and so on.

I don't think it's a network issue. It looks like it downloaded and generated the src.rpm, then when it went to install it to build it, it got the error. So, some sort of unpacking error.

Wrote: /builddir/build/SRPMS/firefox-66.0-4.fc28.src.rpm
Child return code was: 0
...
DEBUG util.py:643: Executing command: ['/bin/rpm', '-Uvh', '--nodeps', '/builddir/build/originals/firefox-66.0-4.fc28.src.rpm'] with env {'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOME': '/builddir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'PROMPT_COMMAND': 'printf "\033]0;<mock-chroot>\007"', 'PS1': '<mock-chroot> \s-\v\$ ', 'LANG': 'C.UTF-8'} and shell False
DEBUG util.py:389: Unsharing. Flags: 134217728
DEBUG util.py:556: Updating / installing...
DEBUG util.py:556: firefox-66.0-4.fc28 ########################################
DEBUG util.py:554: BUILDSTDERR: error: unpacking of archive failed on file /builddir/build/SOURCES/firefox-66.0.source.tar.xz;5c8f5fbd: cpio: read failed - Inappropriate ioctl for device
DEBUG util.py:554: BUILDSTDERR: error: /builddir/build/originals/firefox-66.0-4.fc28.src.rpm cannot be installed

It is a 313MB src.rpm, but still, it was all local...

AFAIK the srpm got created in https://koji.fedoraproject.org/koji/taskinfo?taskID=33600375 (buildSRPMFromSCM), uploaded to the hub, the s390x builder downloads it from the hub and installs, where it fails. IMHO the srpm got corrupted/truncated/... during the transfer.

For confirmation we would need access to the chroot with the failed build, IIRC there is 4 hour window defined in koji before the chroot will be removed.

Ah, right you are.

Odd then that it didn't error on the download?

Yeah, we need to get to a root where this fails before the timout and it's cleaned up.

I think a cache can mask an incomplete transfer, IIRC I saw that with squid and forced reload of given URL was required then. It could be a bit flipped in otherwise complete file too.

We have done some additional work now (lowered the weight on s390x builders so they don't take on multiple jobs at once) and adjusted the caching more.

@stransky can you try a build with s390x re-enabled again now?

I've filed a koji issue on this:

https://pagure.io/koji/issue/1418

Hopefully they can come up with some mitigation/fix for this. Please follow along on that ticket...

Thanks, and sorry for all the hassle.

Metadata Update from @kevin:
- Issue close_status updated to: Upstream
- Issue status updated to: Closed (was: Open)

3 months ago

Login to comment on this ticket.

Metadata