Hello,
we recently hit https://bugzilla.redhat.com/show_bug.cgi?id=2234711 which seems like it is caused by some CentOS 8 Stream mirrors having over 40 days old content:
$ curl -s 'http://mirrorlist.centos.org/?release=8-stream&arch=x86_64&repo=BaseOS&infra=container' \ | while read i ; do echo $i ; curl -si $i/repodata/repomd.xml | grep -i Last-Modified ; done http://centos2.hti.pl/8-stream/BaseOS/x86_64/os/ Last-Modified: Mon, 10 Jul 2023 04:01:14 GMT http://centosmirror.netcup.net/centos/8-stream/BaseOS/x86_64/os/ Last-Modified: Tue, 22 Aug 2023 08:14:22 GMT http://centos.anexia.at/centos/8-stream/BaseOS/x86_64/os/ Last-Modified: Mon, 10 Jul 2023 04:01:14 GMT http://centos.anexia.at/centos/8-stream/BaseOS/x86_64/os/ Last-Modified: Mon, 10 Jul 2023 04:01:14 GMT http://distrib-coffee.ipsl.jussieu.fr/pub/linux/centos/8-stream/BaseOS/x86_64/os/ Last-Modified: Mon, 10 Jul 2023 04:01:14 GMT http://mirror.radoreservers.com/centos/8-stream/BaseOS/x86_64/os/ Last-Modified: Mon, 10 Jul 2023 04:01:14 GMT http://mirror.mangohost.net/centos/8-stream/BaseOS/x86_64/os/ Last-Modified: Mon, 10 Jul 2023 04:01:14 GMT http://mirrors.xservers.ro/centos/8-stream/BaseOS/x86_64/os/ Last-Modified: Mon, 10 Jul 2023 04:01:14 GMT http://ro.mirrors.virtono.com/centos/8-stream/BaseOS/x86_64/os/ Last-Modified: Mon, 10 Jul 2023 04:01:14 GMT http://mirrors.evoluso.com/centos/8-stream/BaseOS/x86_64/os/ Last-Modified: Mon, 10 Jul 2023 04:01:14 GMT
What is the quality standard that we expect from the mirrors to be listed on mirrorlist.centos.org?
mirrorlist.centos.org
An interesting datapoint is, I get this list of mirrors of mostly all stale when I run that curl -s 'http://mirrorlist.centos.org/?release=8-stream&arch=x86_64&repo=AppStream&infra=container' in a container. On a host, I get much "better" mirrors.
curl -s 'http://mirrorlist.centos.org/?release=8-stream&arch=x86_64&repo=AppStream&infra=container'
Metadata Update from @arrfab: - Issue assigned to arrfab
Metadata Update from @arrfab: - Issue tagged with: centos-build-pipeline, centos-common-infra, investigation, medium-gain, medium-trouble, mirror-linux
Hi, Sorry to just see your ticket now but realizing that gmail (Red Hat chosen email platform) considered it was spam so not getting the pagure.io notification ...
Anyway, I had a quick look at this and there was indeed no update for CentOS Stream 8 for ~40 days but suddenly this week a massive push was done so all mirrors had to pick the changes (including .iso images, etc).
The mirrorlist crawler (that will disappear when 8-stream and 7 will be EOL so in some months) logic is this : verify timestamp for each repomd.xml file and keeping previous one in cache (in DB). So two things happened at the same time : temporary network issue from the crawler to the main machine used as "reference" (valid timestamp) and then third-party mirrors having to still sync (and they should have been invalidated if not up2date already after ~24h)
Verifying mirror crawler logs and current crawl/verification operation should (in)validate third-party mirrors. Worth knowing also that because of all the repodata for 8-stream, including SIGs content, a typical crawl for the 520 external mirrors takes some hours. Keeping ticket open until we can confirm in some hours that it's back to normal
follow-up : confirmed through crawler log that (for example) first mirror was invalidated so should disappear from mirrorlist soon:
http://centos2.hti.pl/8-stream/BaseOS/ppc64le/os/ - Bad ts 1688961305
It seems the x86_64 BaseOS repos have now recovered as well, and I was able to rebuild the CentOS 8 Stream-based FreeIPA container image.
Thanks for the investigation and getting the tooling fix the setup.
Thanks for the feedback, really appreciated. Closing ticket as it's now ok
Metadata Update from @arrfab: - Issue close_status updated to: Fixed with Explanation - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.