Due to some storage issue recently , we needed to moved the PersistentVolumes used on OCP openshift cluster in CI env to another storage box. As now all volumes are moved (including some that weren't related to openshift/CI) we'll do some maintenance on storage02 box, optimize disk layout and then schedule a maintenance window to rebalance openshift PVs on that node.
Metadata Update from @arrfab: - Issue assigned to arrfab
Metadata Update from @arrfab: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: centos-ci-infra, doc, high-gain, medium-trouble
Already discussed with @dkirwan and we'll do that together, also tagged as "doc" as we'll also write the SOP for such kind of storage reconfiguration and optimization
Status update : storage is being reconfigured and also with some fio/ioperf tests ran before and after md device layout change Also some underlying maintenance on HBA firmware update that seems to have solved the issue. Machine was under constant random read/write tests for 48h+ without any issue appearing anymore.
We'll post more details when storage/NFS filer will be ready to be used back and we'll announce maintenance window to move openshift PersistentVolumes on reconfigured box
storage02 is now back in action, with array reconfigured as raid10 and some devices added as spare disks. That should give use more reliability and also able to rebuild array with existing spare disks (hard to find these days)
spare
Metadata Update from @arrfab: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.