#7918 ODCS systems moved to NFS volumes
Closed: Fixed a month ago by kevin. Opened 3 months ago by smooge.

Currently the ODCS systems are running out of space. They store a lot of data on a 2 node gluster which acts like a failover brick. This is not suported by the gluster upstream and causes various problems over time. We would want to move the data out of the brick into a shared NFS storage node.

  1. Work with odcs people on downtime
  2. Set up 2 25 GB NFS shares (prod/share) on netapp. Export to the odcs boxes
  3. Mount on odcs boxes
  4. Copy data out of bricks onto netapp nfs
  5. Unmount bricks and remount netapp in place
  6. Test and fix
  7. Repeat.

Metadata Update from @mizdebsk:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: odcs

3 months ago

Can someone provide some sort of timeline of when this needs to be done by?

The real issue behind this is that ODCS does not remove old composes in Fedora infra. I'm going to update ODCS to new version which should fix this. But because the ODCS version used in Fedora infra is very old, we need this to happen first:

https://pagure.io/fedora-infrastructure/issue/7597

So, as I feared... this reached 100% full recently. We didn't get an alert because we got on when it went 90% full and acked it... :(

So, I just went and did it. Both prod and stg are now using nfs volumes. Please check permissions or the like...

I will go deal with the rabbitmq ticket here as soon as I can.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

a month ago

Login to comment on this ticket.

Metadata