We need to temporarily change the type of the EC2 instance to a larger one (probably c5.18xlarge) with much more memory, to increase the kernel dentry caching possibilities (while still being able to handle the current load).

ATM we are trying to rsync our data from one volume to another (with raid). We have 42M inodes and 15T of data. The problem is not to copy the increments (hundreds of GBs), but to traverse the directory structure (>= 30 hours). The plan is to have much more memory, let the kernel cache ideally all dentries (perhaps tweak the vfs_cache_pressure configuration), and thus have relatively fast rsync runs (ideally a few hours).

Then, we'll need a next major outage - multi-hour - but data read-only. We'll do the final rsync, and move the instance back to what we have now (c5.4xlarge).

The outage window for increasing the power should be short, but copr-backend data (RPM repositories) will be down. We expect an outage in a matter of minutes till we reboot:

$ date --date "2022-10-17 13:00 UTC"
Mon Oct 17 03:00:00 PM CEST 2022

Sorry for any inconvenience.