Here's the list for mass rebuild:

https://pagure.io/releng/issue/8501
https://pagure.io/releng/issue/8482
https://pagure.io/releng/issue/8462
https://pagure.io/releng/issue/8395
https://pagure.io/releng/issue/8340
https://pagure.io/releng/issue/7575

https://pagure.io/releng/issue/8481 is missing on the list. I'm work atm on getting the prerelease version in.

There might be expected FTBFS on golang side of things due to the on going work on new macros stack. Following packages might fail to build(with broken deps):

arduino-builder
buildah
caddy
consul
containernetworking-plugins
cri-tools
deepin-api
deepin-daemon
delve
dep
direnv
dnscrypt-proxy
docker-distribution
git-octopus
glusterd2
go-bindata
godep
gomtree
gotun
grafana
heketi
hugo
ignition
jid
kata-ksm-throttler
kata-osbuilder
kata-proxy
kata-shim
kompose
kubernetes
manifest-tool
mantle
micro
moby-engine
obfs4
oci-kvm-hook
oci-register-machine
ocitools
openvswitch-ovn-kubernetes
origin
oshinko-cli
podman
popub
reg
restic
rkt
runc
singularity
skopeo
snapd
startdde
source-to-image
swig
xe-guest-utilities

If they fail due to the broken deps they will be rebuild by the owners of https://fedoraproject.org/wiki/Changes/Adopt_new_Go_Packaging_Guidelines in collaboration with the respective maintainers. I have just spoke about it with @eclipseo

https://pagure.io/releng/issue/8481 is missing on the list. I'm work atm on getting the prerelease version in.

Ready in koji https://koji.fedoraproject.org/koji/buildinfo?buildID=1319666

https://pagure.io/releng/issue/8481 is missing on the list. I'm work atm on getting the prerelease version in.

Ready in koji https://koji.fedoraproject.org/koji/buildinfo?buildID=1319666

Actually I have omitted one patch this is correct build https://koji.fedoraproject.org/koji/taskinfo?taskID=36450997

@jcajka Thanks for the update. We will use golang-1.13-0.beta1.2.fc31 for mass rebuild.

We are not including .plt.got isolation in this mass rebuild as it was not accepted upstream.

Are the s390x builders OK?

DEBUG util.py:587:  CuraEngine-lulzbot-1:3.6.12-3.fc31    ########################################
DEBUG util.py:585:  BUILDSTDERR: error: unpacking of archive failed on file /builddir/build/SOURCES/CuraEngine-lulzbot-3.6.12.tar.gz;5d387e80: cpio: read failed - Inappropriate ioctl for device
DEBUG util.py:585:  BUILDSTDERR: error: /builddir/build/originals/CuraEngine-lulzbot-3.6.12-3.fc31.src.rpm cannot be installed

https://koji.fedoraproject.org/koji/taskinfo?taskID=36462135

(Seem like an isolated incident.)

We have seen this very sporadically on those builders when they are under high load. ;(

I've also noticed the ioctl error on dontpanic package and reported it https://pagure.io/fedora-infrastructure/issue/8029 as I could not find it in the issues.

More importantly, are relengs going to retry these failures? Or should do that packagers instead. If packagers, can they resubmit the built into the side tag right now? F30 rebuild suffered from dist-git failures (repository not found) and I had to resubmit tens of these builds.

If you do a regular rawhide build and it succeeds, the package goes away from https://kojipkgs.fedoraproject.org/mass-rebuild/f31-failures.html

That solves the symptoms (the package is not enumerated as failed) but it produces a package built in a different build root with a possibly different compiler, RPM macros etc.

Is somewhere documented whether this side tag enables some of these changes? Or is this time pretty safe to rebuild packages in whatever build root?

I've also noticed the ioctl error on dontpanic package and reported it https://pagure.io/fedora-infrastructure/issue/8029 as I could not find it in the issues.
More importantly, are relengs going to retry these failures? Or should do that packagers instead. If packagers, can they resubmit the built into the side tag right now? F30 rebuild suffered from dist-git failures (repository not found) and I had to resubmit tens of these builds.

The rel-eng meeting yesterday touched the idea of a re-rebuild to workaround the infra glitches.

That solves the symptoms (the package is not enumerated as failed) but it produces a package built in a different build root with a possibly different compiler, RPM macros etc.

Mass rebuild uses the standard rawhide buildroot - f31-build.

Yeah, I think we should wait until the mass rebuild is over, then just resubmit all the failed builds again. That won't get all of them, but should help, and perhaps we can target more from that final pool of failures.

Note that a duplicated F31FTBFS bugzilla was created: https://bugzilla.redhat.com/show_bug.cgi?id=1732841

I've marked it as duplicate. Will check if it is hardcoded anywhere in this repo and update the number.

https://pagure.io/releng/pull-request/8574

This is done.

:christmas_tree: