Planned Outage - Build/Production - 2018-10-03 21:00 UTC
There will be an outage starting at 2018-10-03 21:00 UTC, which will last approximately 4 hours.
To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run:
date -d '2018-10-03 21:00UTC'
Reason for outage:
Various kernel, library, and general security updates need to be applied to all servers. Due to the kernel, glibc, and other updates.. all systems will be rebooted afterwords
Affected Services:
All build, production and related services will be affected.
Ticket Link:
https://pagure.io/fedora-infrastructure/issue/7272
Please join #fedora-admin or #fedora-noc on irc.freenode.net or add comments to the ticket for this outage above.
Machines with drives that need re-adding on reboot:
? Tasks needing to be done during outage: Drives need to be added during reboots ppc8-02, qa12, qa14 (2 drives), virthost17, virthost18, download03 ?
Updates will be mass applied to 'all' in ansible before outage begins. So, we will just need to do reboots and checking things, then a master ansible run after everything is back up.
Monday Hosts to Update/Reboot
bvirthost01.stg.phx2 - virthost01.stg.phx2 - virthost02.stg.phx2 - virthost03.stg.phx2 - virthost04.stg.phx2 - virthost05.stg.phx2 -
Tuesday Hosts to Update/Reboot data-analysis01.phx2 - done - smooge download01.phx2 - download02.phx2 - download03.phx2 - download04.phx2 - download05.phx2 - download-rdu01 - coloamer01 - dedicatedsolutions01 - internetx01 - tummy01 - virthost-rdu01 - virthost-rdu02 - ibiblio01 - ibiblio04 - osuosl01 - osuosl02 - virthost-cc-rdu01 - virthost-cc-rdu02 - virthost-cc-rdu03 -
Wednesday Hosts to Update/Reboot: buildhw-01.phx2 - buildhw-02.phx2 - buildhw-03.phx2 - buildhw-04.phx2 - buildhw-05.phx2 - buildhw-06.phx2 - buildhw-07.phx2 - buildhw-08.phx2 - buildhw-09.phx2 - buildhw-10.phx2 - buildvmhost-01.phx2 - buildvmhost-02.phx2 - buildvmhost-03.phx2 - buildvmhost-04.phx2 - autocloud-backend-libvirt2.phx2 - autocloud-backend-vbox2.phx2 - bvirthost01.phx2 - bvirthost04.phx2 - bvirthost05.phx2 - bvirthost08.phx2 - bvirthost12.phx2 - bvirthost13.phx2 - bvirthost14.phx2 - bvirthost15.phx2 - ppc8-01.ppc - ppc8-02.ppc - ppc8-03.ppc - ppc8-04.ppc - aarch64-c01n1.arm - aarch64-c02n1.arm - aarch64-c03n1.arm - aarch64-c04n1.arm - aarch64-c05n1.arm - aarch64-c06n1.arm - aarch64-c07n1.arm - aarch64-c08n1.arm - aarch64-c09n1.arm - aarch64-c10n1.arm - aarch64-c11n1.arm - aarch64-c12n1.arm - aarch64-c13n1.arm - aarch64-c14n1.arm - aarch64-c15n1.arm - aarch64-c16n1.arm - aarch64-c17n1.arm - aarch64-c18n1.arm - aarch64-c19n1.arm - aarch64-c20n1.arm - aarch64-c21n1.arm - aarch64-c22n1.arm - aarch64-c23n1.arm - aarch64-c24n1.arm - aarch64-c25n1.arm - buildhw-aarch64-01.arm - buildhw-aarch64-02.arm - buildhw-aarch64-03.arm - buildhw-aarch64-04.arm - buildhw-aarch64-05.arm - buildhw-aarch64-06.arm - buildhw-aarch64-07.arm - buildhw-aarch64-08.arm - buildhw-aarch64-10.arm -
buildvm-s390x-01.s390 - buildvm-s390x-01.stg.s390 - buildvm-s390x-02.s390 - buildvm-s390x-03.s390 - buildvm-s390x-04.s390 - buildvm-s390x-05.s390 - buildvm-s390x-06.s390 - buildvm-s390x-07.s390 - buildvm-s390x-08.s390 - buildvm-s390x-09.s390 - buildvm-s390x-10.s390 - buildvm-s390x-11.s390 - buildvm-s390x-12.s390 - buildvm-s390x-13.s390 - buildvm-s390x-14.s390 -
sign-vault03.phx2 - sign-vault04.phx2 - sign-vault05.phx2 - sign-vault06.phx2 - bkernel01.phx2 - bkernel02.phx2 - bkernel03.phx2 - bkernel04.phx2 -
Thursday? Systems to Update/Reboot
virthost01.phx2 - virthost02.phx2 - virthost03.phx2 - virthost12.phx2 - virthost14.phx2 - virthost15.phx2 - virthost17.phx2 - virthost18.phx2 - virthost19.phx2 - virthost21.phx2 - virthost22.phx2 -
ibiblio05 - osuosl03 -
qa05.qa - qa07.qa - qa09.qa - qa10.qa - qa11.qa - qa12.qa - qa13.qa - qa14.qa - virthost-comm01.qa - virthost-comm03.qa - virthost-comm04.qa - aarch64-c26n1-oqa.arm - aarch64-c27n1-oqa.arm - aarch64-c28n1-oqa.arm - aarch64-c29n1-oqa.arm - aarch64-c30n1-oqa.arm - kernel01.qa - kernel02.qa - retrace01.qa - retrace02.qa -
NEWCLOUD DOES ITS OWN TING OLDCLOUD DO NOT TOUCH cloud-noc01.cloud cn-x86-64-01-03.cloud cn-x86-64-01-04.cloud cn-x86-64-01-05.cloud cn-x86-64-01-06.cloud cn-x86-64-01-07.cloud cn-x86-64-01-08.cloud cn-x86-64-02-01.cloud cn-x86-64-02-02.cloud control01.cloud control02.cloud control03.cloud fed-cloud03.cloud fed-cloud04.cloud fed-cloud05.cloud fed-cloud06.cloud fed-cloud07.cloud fed-cloud08.cloud fed-cloud09.cloud fed-cloud10.cloud fed-cloud11.cloud fed-cloud12.cloud fed-cloud13.cloud fed-cloud14.cloud fed-cloud15.cloud arm03-packager00.cloud arm03-packager01.cloud arm03-qa00.cloud arm03-qa01.cloud
This outage is over, with a few exceptions:
Something is wrong in the cloud, so copr cannot spin up new builders, hopefully we can fix it.
A few cloud instances are also down because they wouldn't reboot.
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.