#10302 Planned Outage - build systems ( koji, osbs, mbs, src, pdc, kojipkgs, odcs, registry) - 2021-11-09 17:00 UTC
Closed: Fixed 2 years ago by kevin. Opened 2 years ago by kevin.

Planned Outage - build systems ( koji, osbs, mbs, src, pdc, kojipkgs, odcs, registriy) - 2021-11-09 17:00 UTC

There will be an outage starting at 2021-11-09 17:00 UTC,
which will last approximately 4 hours.

To convert UTC to your local time, take a look at
http://fedoraproject.org/wiki/Infrastructure/UTCHowto
or run:

date -d '2021-11-09 17:00UTC'

Reason for outage:

We will be doing several maint tasks during this outage:

  1. All the s390x builders will be moving from the current z13 maintframe to a z15 mainframe.

  2. koji hub and builders will be updated from 1.25.1 to 1.26.1

  3. Updates will be applied to all build servers and reboots done to the latest kernel.

Maintainers are advised to avoid starting builds before the outage that won't complete before the outage is over. Some builds may restart or need to be resubmitted if they are running during the maint window.

Affected Services:

koji (both hub and builders)
mbs - module build service
odcs - on demand compose service
osbs - openshift build service
pdc - product def center
src - dist-git/pagure
kojipkgs - pkgs stored on koji
registry - container registries

Ticket Link:

https://pagure.io/fedora-infrastructure/issue/10302

Please join #fedora-admin or #fedora-noc on irc.libera.chat
or add comments to the ticket for this outage above.


Metadata Update from @kevin:
- Issue priority set to: Waiting on External (was: Needs Review)

2 years ago

Mailed devel-announce and added to fedorastatus as pending.

ok, this did not go great. ;(

  • We tried to move the z/vm lpar to the z15 mainframe, but it wouldn't run there, it needs some patches to be added to the z15 to get it working. :( So, we moved it back.
  • We tried to move the kvm lpar to the z15 mainframe, and it came up fine, but it couldn't get up on the network. There appears to be a faulty card on the z15 that wasn't working. :(

Mainframe folks are going to try and address both those issues somehow and we are hopefully going to try again with more success next week. ;(

I did manage to:

  • update all the builders (still f34 tho)
  • update koji on hubs and builders to the latest ( 1.16.1)
  • update/reboot all the bvmhost-x86 virthosts and their guests.

buildvm-s390x-07 is still having a bit of issue, but I will deal with it when I get time.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog