#10951 Outage: Moving the Fedora Copr backend storage, 2022-10-28 -- 2022-10-30
Closed: Fixed a year ago by praiskup. Opened 2 years ago by praiskup.

We need to replace the volume hosting our RPM/YUM repo data with a larger one:

  • 16TB => 24TB
  • one EBS EC2 volume => RAID 10 of EBS EC2 volumes
  • LVM on top of that, for more comfortable size increase in the future

As a consequence, we'll have to stop the build queue processing for about 20 hours, and we'll have two "full" (but short) outages (2x5 minutes) when the yum repositories will be unavailable.

The anticipated start of the outage (queue stops processing) is Saturday 2022-10-29:

$ date --date "2022-10-29 17:00 UTC"
Sat Oct 29 07:00:00 PM CEST 2022

Even a few hours before, we'll lower the build throughput a bit to prioritize the I/O we have on the data migration task.

We'd like to kindly ask you to minimize the build throughput in the Fedora Copr build service during the weekend in question to allow us to make the transition as smooth as possible. It would be nice if you could delay your mass rebuilding and similar other large data modifications until the storage movement is over. That said, we don't plan to turn the frontend machine (API and web-UI) off during the whole outage period.

For more info, see the more detailed schedule.

Affected services:
Copr (frontend):
https://copr.fedorainfracloud.org
Copr repositories:
https://download.copr.fedorainfracloud.org/
https://copr-be.cloud.fedoraproject.org/


Metadata Update from @zlopez:
- Issue tagged with: outage

2 years ago

Metadata Update from @phsmoura:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: medium-gain, medium-trouble

2 years ago

Aaand the outage is over. Happy building!

Metadata Update from @praiskup:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

a year ago

Login to comment on this ticket.

Metadata