#9960 Planned Outage - s390x builders - 2021-06-04 22:00 UTC
Closed: Fixed 10 days ago by kevin. Opened a month ago by kevin.

Planned Outage - s390x builders - 2021-06-04 22:00 UTC

There will be an outage starting at 2021-06-04 22:00 UTC,
which will last approximately 36 hours.

To convert UTC to your local time, take a look at
http://fedoraproject.org/wiki/Infrastructure/UTCHowto
or run:

date -d '2021-06-04 22:00UTC'

Reason for outage:

The facility hosting the fedora s390x builders has a site wide power outage scheduled for June 4th to June 6th. We will be powering off these builders at 22UTC on 2021-06-04. It's likely that they will be back sometime late in the day of 2021-06-05, but it could be they are not back until 2021-06-06. During this time any builds that build against s390x will stall in 'free' state until the builders are back. No rawhide composes will be done in the outage window.

Affected Services:

koji builds needing to build on s390x will queue up and wait for the builders to return.

Ticket Link:

https://pagure.io/fedora-infrastructure/issue/9960

Please join #fedora-admin or #fedora-noc on irc.freenode.net
or add comments to the ticket for this outage above.


Metadata Update from @smooge:
- Issue priority set to: Waiting on Assignee (was: Needs Review)

a month ago

Mailed devel-announce and infrastructure lists and updated status with planned outage.

This outage is over.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

10 days ago

Now that s390x builders are coming back online, a lot of builds are failing with what looks like networking issues there ... is this expected?

DEBUG util.py:444:  Errors during downloading metadata for repository 'build':
DEBUG util.py:444:    - Curl error (6): Couldn't resolve host name for http://kojipkgs01.fedoraproject.org/repos/f35-build/3664786/s390x/repodata/repomd.xml [Could not resolve host: kojipkgs01.fedoraproject.org]
DEBUG util.py:444:    - Curl error (6): Couldn't resolve host name for http://kojipkgs02.fedoraproject.org/repos/f35-build/3664786/s390x/repodata/repomd.xml [Could not resolve host: kojipkgs02.fedoraproject.org]
DEBUG util.py:444:    - Curl error (6): Couldn't resolve host name for http://kojipkgs-cache01.s390.fedoraproject.org/repos/f35-build/3664786/s390x/repodata/repomd.xml [Could not resolve host: kojipkgs-cache01.s390.fedoraproject.org]
DEBUG util.py:444:  Error: Failed to download metadata for repo 'build': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
DEBUG util.py:598:  Child return code was: 1

It doesn't happen every time, but about 50% of my builds failed with this issue, and resubmitting them was a coin toss again ...

No, it's not. Looking...

6 of the builders had incorrect /etc/resolv.conf... fixed those.

Let me know if you see any more of these now. :(

Thanks, I'll keep my eyes peeled :)

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog