Similar to https://pagure.io/fedora-infrastructure/issue/10347

Another strange s390x failure: https://koji.fedoraproject.org/koji/taskinfo?taskID=79179721
but on buildvm-s390x-23.s390.fedoraproject.org

I have re-installed the affected builds with different cache settings. I think the new z15 mainframe might expose some issues with the caching settings we had on the vm's.

Please report any issues you hit after now ( 2021-11-23 20:30UTC)

Don't know if this is related, but s390x specific failure

/var/tmp/rpm-tmp.7MWy6j: line 105: /builddir/build/BUILDROOT/php-8.1.0-1.fc36.s390x/usr/bin/php: cannot execute binary file: Exec format error

=> https://koji.fedoraproject.org/koji/taskinfo?taskID=79218730
=> https://koji.fedoraproject.org/koji/taskinfo?taskID=79219016

(on different executable, one on /usr/bin/php, other with /usr/bin/zts-php)

Don't know if this is related, but s390x specific failure

We'll need either Koji task ID or at least builder name so that we can check what is going on with the host.

@mizdebsk sorry for paste issue, previous comment updated with taskID

Thanks for the report. Both builders (-20 and -21) have I/O errors in dmesg. So it looks like changing KVM cache settings did not help.

Another one:

[ 28%] Building Java object Wrapping/Java/CMakeFiles/vtkjava.dir/vtk/CellType.class
cd /builddir/build/BUILD/VTK-9.1.0/build/Wrapping/Java && /usr/bin/javac -classpath /builddir/build/BUILD/VTK-9.1.0/build/Wrapping/Java -source 1.7 -target 1.7 /builddir/build/BUILD/VTK-9.1.0/build/Wrapping/Java/vtk/CellType.java -d CMakeFiles/vtkjava.dir
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
gmake[2]: *** [Wrapping/Java/CMakeFiles/vtkjava.dir/build.make:18704: Wrapping/Java/CMakeFiles/vtkjava.dir/vtk/CellType.class] Error 1

https://koji.fedoraproject.org/koji/taskinfo?taskID=79217763
buildvm-s390x-23.s390.fedoraproject.org

Another one:

+ /usr/lib/rpm/brp-strip-static-archive /usr/bin/strip
/usr/bin/strip: unable to copy file '/builddir/build/BUILDROOT/glib2-2.70.1-2.fc35.s390x/usr/lib64/libglib-2.0.a'; reason: Input/output error
/usr/bin/strip: unable to copy file '/builddir/build/BUILDROOT/glib2-2.70.1-2.fc35.s390x/usr/lib64/libgio-2.0.a'; reason: Input/output error

https://kojipkgs.fedoraproject.org//work/tasks/8434/79228434/build.log

:(

So, nothing at all on the host here, so I suspect this is a qemu problem.

I tried to change the builders to use s390-ccw-virtio-4.1 instead of s390-ccw-virtio-6.1, but that had no effect.

So, I have now downgraded qemu to the f34 version that was formerly working.

If this makes things stable, I'll file a qemu bug.

So, do let me know if you see this again now (and I will also try and watch)

Seems to have worked so far... I see 0 errors on the guests I've rebooted with the f34 qemu.

I have filed:

https://bugzilla.redhat.com/show_bug.cgi?id=2026747