Should be fixed in about an hour.

Unfortunately it is not clear what has happened here. The primary mirror scanner did not pick up the changes on the primary mirror correctly. Not sure why. I have not seen this with Fedora and EPEL yet.

CC: @carlgeorge Something is not yet working correctly. Not sure what.

I have enabled debug output. Maybe that can help us if it happens again.

Indeed, seems fixed, thank you!

@adrian , @carlgeorge wondering if we can keep this one open to be sure we know what's the problem

@praiskup : is copr still running from EC2 ? if so it should always get internal cloudfront mirror, so not even using mirror.stream.centos.org as official mirror , nor any of the existing/validated mirrors

@adrian : I tried a curl 'https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64' from an EC2 instance and mirrormanager doesn't pick the correct cloudfront mirror as it should, so yeah, that would be a problem to solve too : the metalink I get is a "normal" one but not with the cloudfront mirror I should have in the returned list (and it should have higher pref always) : do you want me to create a ticket somewhere at fedora-infra to fix that ?

@adrian : I tried a curl 'https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64' from an EC2 instance and mirrormanager doesn't pick the correct cloudfront mirror as it should, so yeah, that would be a problem to solve too : the metalink I get is a "normal" one but not with the cloudfront mirror I should have in the returned list (and it should have higher pref always) : do you want me to create a ticket somewhere at fedora-infra to fix that ?

We never configured the cloudfront mirror correctly for EC2 netblocks. I have taken the netblock list from Fedora's cloudfront mirror and added it to the stream cloudfront mirror. This should be solved in about an hour.

@praiskup : is copr still running from EC2 ? if so it should always get internal cloudfront mirror, so not even using mirror.stream.centos.org as official mirror , nor any of the existing/validated mirrors

The infrastructure runs in EC2 (isn't related to this report), and a subset of builders too. So, regardless of where the builders are, they aim to use the closest mirrors. As defined in mock-core-configs.rpm package.

The problem I've seen yesterday was though affecting even my own local box, location Czech, Brno.

Now the cloudfront redirection seems to work. Trying following command in EC2 (US) I get:

# curl 'https://mirrors.centos.org/mirrorlist?repo=centos-crb-9-stream&arch=x86_64'
# repo = centos-crb-9-stream arch = x86_64 Using preferred netblock country = US country = CA 

Using the mirrorlist interface for debugging tells me that it is not just using country/continent to assign the mirror, but for this client it first uses preferred netblocks.

@praiskup : is copr still running from EC2 ? if so it should always get internal cloudfront mirror, so not even using mirror.stream.centos.org as official mirror , nor any of the existing/validated mirrors

The infrastructure runs in EC2 (isn't related to this report), and a subset of builders too. So, regardless of where the builders are, they aim to use the closest mirrors. As defined in mock-core-configs.rpm package.

The problem I've seen yesterday was though affecting even my own local box, location Czech, Brno.

Ok, so still something to investigate at the mirrormanager side (and so for @adrian maybe)

I also confirms that (tested from a ec2 instance) I get preference="100" (so higher one) for the cloudfront mirror, so always fastest for all aws

Ok, so still something to investigate at the mirrormanager side (and so for @adrian maybe)

Yes. I need to to understand what has happened.

I understand the problem now.

Our scanner is going over all directories and if any directory has a newer ctime than in the database we look closer at it. If it has a new repomd.xml we update the information in the database with the new details about repomd.xml. Size, time and checksums.

With the centos scanning following happens. We detect that the directory changed and look at the directory:

$ rsync://mirror.stream.centos.org/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/repodata/
drwxr-xr-x          4,096 2021/10/14 16:48:32 .
-rw-r--r--        598,368 2021/10/08 14:56:15 649fb2839990a306cd3139a4ceb307cb65d8d83016a16de1e5d06d07b66df3a8-filelists.xml.gz
-rw-r--r--        585,636 2021/10/08 14:56:16 8ec258d2cc218a8a155921523462b441ceea1df980026f23eba2cc0899ff61ab-primary.sqlite.xz
-rw-r--r--        243,888 2021/10/08 14:56:16 94159cbd44cd2b9660adb90ea596f1b49d19d8488f141dd2bcdc6acc30abf9a4-other.sqlite.xz
-rw-r--r--        610,708 2021/10/08 14:56:16 a339fd8ad8328f6f9cb3055c095f8af9deb399bf86bcd9dee095d3223b88db81-filelists.sqlite.xz
-rw-r--r--         39,732 2021/10/08 14:56:15 b110f5448750724b81bc70c780bc954e0bd7f454b08b3f5ab7ef0e91cf3c7de5-comps-BaseOS.x86_64.xml
-rw-r--r--          4,660 2021/10/08 14:56:15 c4967e93e26112f55ac2f6ee402dee9d892450e5012b216e3708898464a81409-comps-BaseOS.x86_64.xml.xz
-rw-r--r--        491,555 2021/10/08 14:56:15 d9c4b7d2747a45707e77a0b9288d3b8b05d2da901d90b66ca27ef62fb8df74dd-other.xml.gz
-rw-r--r--        362,545 2021/10/08 14:56:15 f7ce4f5b04ce882073b35d65aab8a1af676ac7073d3f634d9d25e125a7c45c34-primary.xml.gz
-rw-r--r--          3,986 2021/10/08 14:56:16 repomd.xml
-rw-r--r--            811 2021/10/08 15:13:04 repomd.xml.asc

The time of repodata/ has changed but none of the files have been updated. A couple of hours later I see (on some mirrors):

$ rsync rsync://mirror.stream.centos.org/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/repodata/
drwxr-xr-x          4,096 2021/10/14 16:48:32 .
-rw-r--r--        702,727 2021/10/14 14:13:01 016d0f8915c12685b1072972f98b22c903bad15662e9e903519261f51aea3e15-filelists.xml.gz
-rw-r--r--        508,055 2021/10/14 14:13:01 2d73559de4929fdd311c333289b2c11e9781b904dfa04a9962507de32f47e7fe-other.xml.gz
-rw-r--r--        671,976 2021/10/14 14:13:02 6fe0a8d17091999d3a0c8290e76ec079d1051663a0fb8183e026604ff40b1bb4-primary.sqlite.xz
-rw-r--r--        432,966 2021/10/14 14:13:01 79e36aca0e07526111559bcbd9964df8b66f116d734de202370d4eb2249e11f9-primary.xml.gz
-rw-r--r--         39,732 2021/10/14 14:13:00 b110f5448750724b81bc70c780bc954e0bd7f454b08b3f5ab7ef0e91cf3c7de5-comps-BaseOS.x86_64.xml
-rw-r--r--        251,456 2021/10/14 14:13:02 b910ed5ab7ce1f8d395126723576f55f70182e7718b4a6edfbe573455af4ceeb-other.sqlite.xz
-rw-r--r--          4,660 2021/10/14 14:13:01 c4967e93e26112f55ac2f6ee402dee9d892450e5012b216e3708898464a81409-comps-BaseOS.x86_64.xml.xz
-rw-r--r--        682,288 2021/10/14 14:13:02 f362c298420df0470fd7873e974c43c7aa9afa0732befbd76f31ee508bb53ea5-filelists.sqlite.xz
-rw-r--r--          3,986 2021/10/14 14:13:02 repomd.xml
-rw-rw-r--            811 2021/10/14 16:48:34 repomd.xml.asc

But not on all systems which are backing mirror.stream.centos.org. In this output the ctime of the directory is still the same but the files are much newer. Our scanner will not pick up the changes because the files should have only changed if the ctime of the directory has changed.

I think this is related to the fact that we are not scanning the real primary system but we are scanning multiple systems which are syncing the content at different times.

We need to be able to scan the real primary mirror and not some replication which is different depending on the result of DNS and to which supposedly primary mirror we are redirected.

The current setup will break probably every time an update is pushed to the mirrors.

OK can we NFS mount master-2.iad2 onto the mirrormanager master system and scan from there?
Or can we make a Netapp partitition and put the centos content on it?

rsync scanning works for RPM Fusion. It is just that we need to be able to really access the real primary mirror and not a random copy.

Mounting via NFS would bring us fullfiletimelist-* scanning (which basically does not do any I/O besides reading the file). That's what we do for Fedora/EPEL.

I could also implement fullfiletimelist-* scanning via rsync. That would also work.

if mirrormanager is in IAD2, it makes sense to use one node we have there too and if you can give me an IP that will be used for this, I can create a specific rsync module/target and hand it over to you, and then you can change upstream/origin in mirrormanager to always consider that one to be real upstream.

if mirrormanager is in IAD2, it makes sense to use one node we have there too and if you can give me an IP that will be used for this, I can create a specific rsync module/target and hand it over to you, and then you can change upstream/origin in mirrormanager to always consider that one to be real upstream.

Do you need the internal IP or the external IP? The connection will be coming from mm-backend01.iad2.fedoraproject.org

Internal IP would work and I just pushed in git/ansible the needed new module for this.
Sent details (by email) about ip/rsync module to use from that internal node in fedora infra.

Closing as #495 was also worked on and closed/fixed