#7272 Fedora Infrastructure Service Outages (2018-10-03)
Closed: Fixed 9 months ago Opened 9 months ago by smooge.

Planned Outage - Build/Production - 2018-10-03 21:00 UTC

There will be an outage starting at 2018-10-03 21:00 UTC,
which will last approximately 4 hours.

To convert UTC to your local time, take a look at
http://fedoraproject.org/wiki/Infrastructure/UTCHowto
or run:

date -d '2018-10-03 21:00UTC'

Reason for outage:

Various kernel, library, and general security updates need to be applied to all servers. Due to the kernel, glibc, and other updates.. all systems will be rebooted afterwords

Affected Services:

All build, production and related services will be affected.

Ticket Link:

https://pagure.io/fedora-infrastructure/issue/7272

Please join #fedora-admin or #fedora-noc on irc.freenode.net
or add comments to the ticket for this outage above.


Machines with drives that need re-adding on reboot:

?
Tasks needing to be done during outage:
Drives need to be added during reboots
ppc8-02,
qa12,
qa14 (2 drives),
virthost17,
virthost18,
download03
?

Updates will be mass applied to 'all' in ansible before outage begins.
So, we will just need to do reboots and checking things, then
a master ansible run after everything is back up.

Monday Hosts to Update/Reboot

bvirthost01.stg.phx2 -
virthost01.stg.phx2 -
virthost02.stg.phx2 -
virthost03.stg.phx2 -
virthost04.stg.phx2 -
virthost05.stg.phx2 -

Tuesday Hosts to Update/Reboot
data-analysis01.phx2 - done - smooge
download01.phx2 -
download02.phx2 -
download03.phx2 -
download04.phx2 -
download05.phx2 -
download-rdu01 -
coloamer01 -
dedicatedsolutions01 -
internetx01 -
tummy01 -
virthost-rdu01 -
virthost-rdu02 -
ibiblio01 -
ibiblio04 -
osuosl01 -
osuosl02 -
virthost-cc-rdu01 -
virthost-cc-rdu02 -
virthost-cc-rdu03 -

Wednesday Hosts to Update/Reboot:
buildhw-01.phx2 -
buildhw-02.phx2 -
buildhw-03.phx2 -
buildhw-04.phx2 -
buildhw-05.phx2 -
buildhw-06.phx2 -
buildhw-07.phx2 -
buildhw-08.phx2 -
buildhw-09.phx2 -
buildhw-10.phx2 -
buildvmhost-01.phx2 -
buildvmhost-02.phx2 -
buildvmhost-03.phx2 -
buildvmhost-04.phx2 -
autocloud-backend-libvirt2.phx2 -
autocloud-backend-vbox2.phx2 -
bvirthost01.phx2 -
bvirthost04.phx2 -
bvirthost05.phx2 -
bvirthost08.phx2 -
bvirthost12.phx2 -
bvirthost13.phx2 -
bvirthost14.phx2 -
bvirthost15.phx2 -
ppc8-01.ppc -
ppc8-02.ppc -
ppc8-03.ppc -
ppc8-04.ppc -
aarch64-c01n1.arm -
aarch64-c02n1.arm -
aarch64-c03n1.arm -
aarch64-c04n1.arm -
aarch64-c05n1.arm -
aarch64-c06n1.arm -
aarch64-c07n1.arm -
aarch64-c08n1.arm -
aarch64-c09n1.arm -
aarch64-c10n1.arm -
aarch64-c11n1.arm -
aarch64-c12n1.arm -
aarch64-c13n1.arm -
aarch64-c14n1.arm -
aarch64-c15n1.arm -
aarch64-c16n1.arm -
aarch64-c17n1.arm -
aarch64-c18n1.arm -
aarch64-c19n1.arm -
aarch64-c20n1.arm -
aarch64-c21n1.arm -
aarch64-c22n1.arm -
aarch64-c23n1.arm -
aarch64-c24n1.arm -
aarch64-c25n1.arm -
buildhw-aarch64-01.arm -
buildhw-aarch64-02.arm -
buildhw-aarch64-03.arm -
buildhw-aarch64-04.arm -
buildhw-aarch64-05.arm -
buildhw-aarch64-06.arm -
buildhw-aarch64-07.arm -
buildhw-aarch64-08.arm -
buildhw-aarch64-10.arm -

buildvm-s390x-01.s390 -
buildvm-s390x-01.stg.s390 -
buildvm-s390x-02.s390 -
buildvm-s390x-03.s390 -
buildvm-s390x-04.s390 -
buildvm-s390x-05.s390 -
buildvm-s390x-06.s390 -
buildvm-s390x-07.s390 -
buildvm-s390x-08.s390 -
buildvm-s390x-09.s390 -
buildvm-s390x-10.s390 -
buildvm-s390x-11.s390 -
buildvm-s390x-12.s390 -
buildvm-s390x-13.s390 -
buildvm-s390x-14.s390 -

sign-vault03.phx2 -
sign-vault04.phx2 -
sign-vault05.phx2 -
sign-vault06.phx2 -
bkernel01.phx2 -
bkernel02.phx2 -
bkernel03.phx2 -
bkernel04.phx2 -

Thursday? Systems to Update/Reboot

virthost01.phx2 -
virthost02.phx2 -
virthost03.phx2 -
virthost12.phx2 -
virthost14.phx2 -
virthost15.phx2 -
virthost17.phx2 -
virthost18.phx2 -
virthost19.phx2 -
virthost21.phx2 -
virthost22.phx2 -

ibiblio05 -
osuosl03 -

qa05.qa -
qa07.qa -
qa09.qa -
qa10.qa -
qa11.qa -
qa12.qa -
qa13.qa -
qa14.qa -
virthost-comm01.qa -
virthost-comm03.qa -
virthost-comm04.qa -
aarch64-c26n1-oqa.arm -
aarch64-c27n1-oqa.arm -
aarch64-c28n1-oqa.arm -
aarch64-c29n1-oqa.arm -
aarch64-c30n1-oqa.arm -
kernel01.qa -
kernel02.qa -
retrace01.qa -
retrace02.qa -

NEWCLOUD DOES ITS OWN TING
OLDCLOUD DO NOT TOUCH
cloud-noc01.cloud
cn-x86-64-01-03.cloud
cn-x86-64-01-04.cloud
cn-x86-64-01-05.cloud
cn-x86-64-01-06.cloud
cn-x86-64-01-07.cloud
cn-x86-64-01-08.cloud
cn-x86-64-02-01.cloud
cn-x86-64-02-02.cloud
control01.cloud
control02.cloud
control03.cloud
fed-cloud03.cloud
fed-cloud04.cloud
fed-cloud05.cloud
fed-cloud06.cloud
fed-cloud07.cloud
fed-cloud08.cloud
fed-cloud09.cloud
fed-cloud10.cloud
fed-cloud11.cloud
fed-cloud12.cloud
fed-cloud13.cloud
fed-cloud14.cloud
fed-cloud15.cloud
arm03-packager00.cloud
arm03-packager01.cloud
arm03-qa00.cloud
arm03-qa01.cloud

This outage is over, with a few exceptions:

Something is wrong in the cloud, so copr cannot spin up new builders, hopefully we can fix it.

A few cloud instances are also down because they wouldn't reboot.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

9 months ago

Login to comment on this ticket.

Metadata