Planned Outage - s390x builders - 2021-06-04 22:00 UTC
There will be an outage starting at 2021-06-04 22:00 UTC, which will last approximately 36 hours.
To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run:
date -d '2021-06-04 22:00UTC'
Reason for outage:
The facility hosting the fedora s390x builders has a site wide power outage scheduled for June 4th to June 6th. We will be powering off these builders at 22UTC on 2021-06-04. It's likely that they will be back sometime late in the day of 2021-06-05, but it could be they are not back until 2021-06-06. During this time any builds that build against s390x will stall in 'free' state until the builders are back. No rawhide composes will be done in the outage window.
Affected Services:
koji builds needing to build on s390x will queue up and wait for the builders to return.
Ticket Link:
https://pagure.io/fedora-infrastructure/issue/9960
Please join #fedora-admin or #fedora-noc on irc.freenode.net or add comments to the ticket for this outage above.
+1 looks good.
LGTM :thumbsup:
Metadata Update from @smooge: - Issue priority set to: Waiting on Assignee (was: Needs Review)
Mailed devel-announce and infrastructure lists and updated status with planned outage.
This outage is over.
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Now that s390x builders are coming back online, a lot of builds are failing with what looks like networking issues there ... is this expected?
DEBUG util.py:444: Errors during downloading metadata for repository 'build': DEBUG util.py:444: - Curl error (6): Couldn't resolve host name for http://kojipkgs01.fedoraproject.org/repos/f35-build/3664786/s390x/repodata/repomd.xml [Could not resolve host: kojipkgs01.fedoraproject.org] DEBUG util.py:444: - Curl error (6): Couldn't resolve host name for http://kojipkgs02.fedoraproject.org/repos/f35-build/3664786/s390x/repodata/repomd.xml [Could not resolve host: kojipkgs02.fedoraproject.org] DEBUG util.py:444: - Curl error (6): Couldn't resolve host name for http://kojipkgs-cache01.s390.fedoraproject.org/repos/f35-build/3664786/s390x/repodata/repomd.xml [Could not resolve host: kojipkgs-cache01.s390.fedoraproject.org] DEBUG util.py:444: Error: Failed to download metadata for repo 'build': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried DEBUG util.py:598: Child return code was: 1
It doesn't happen every time, but about 50% of my builds failed with this issue, and resubmitting them was a coin toss again ...
No, it's not. Looking...
6 of the builders had incorrect /etc/resolv.conf... fixed those.
Let me know if you see any more of these now. :(
Thanks, I'll keep my eyes peeled :)
Login to comment on this ticket.