Issue #812: Broken Cs9 repositories - centos-infra

centos-infra

#812 Broken Cs9 repositories

Closed: Fixed with Explanation 2 years ago by arrfab. Opened 2 years ago by mvadkert.

We are just getting reports on Cs9 tests failing on

STDOUT:

CentOS Stream 9 - BaseOS                        642  B/s | 3.9 kB     00:06    


STDERR:

Errors during downloading metadata for repository 'baseos':
  - Downloading successful, but checksum doesn't match. Calculated: a627c683cb157357f45892c00bfeba9a5a0d7e99fb791ac1f2a2b523fdf3cfcf3329498ac2662fd9d278a1d7c1612b635eeff6dc5ca003a42e5587839aa9abd8(sha512)  Expected: 3357f925710b07526b1bc68fa06c48e4a483f823e86f29dfc23d963689e0933c1dff32d4bd55124a1b43ff1bfc18a3ad57a9b56622e21142bce6dec2f2b2d2e1(sha512) 
Error: Failed to download metadata for repo 'baseos': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

What is this caused by? Invalid mirror? Or could it be a more general problem in mirroring? We are trying to find out which repository mirror is affected now.

Is there a DNF problem that it does not retry other mirror on this occasion.

Metadata Update from @arrfab:
- Issue assigned to arrfab

2 years ago

arrfab commented 2 years ago

We don't control mirrormanager directly but it pulls from a host to have files in cache at mirrormanager side. We suffered from a nasty issue on that infra (see similar BZ ) and so that means that while third-party mirrors , and mirror.stream.centos.org are current and up2date, MirrorManager itself is not and so considering that all mirrors are invalid (while mirrormanager is the invalid one, and "living on cache").

I'd like to see with @adrian if there is a solution to quickly update from where mirrormanager (hosted in Fedora Infra) is pulling from , or at least have a fallback node in case it can't update from its origin (in this case the one with underlying issue)

FWIW, the initial root cause should be mitigated, so mirrormanager (but don't know when) should pick up the changes and so slowly working again .

Metadata Update from @arrfab:
- Issue tagged with: centos-stream, high-gain, medium-trouble, mirrormanager

2 years ago

arrfab commented 2 years ago

Based on discussion with @adrian (on irc.libera.chat, in #centos-devel), it should be fixed :100:

(11:34:30) adrian: arrfab: MM DB is now in sync with the primary mirror, fixes should go live 20 minutes after the next hour

So closing for now but maybe a RFE for mirrormanager would be good to ensure it has itself a fallback origin node to pull from ?

Metadata Update from @arrfab:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)

2 years ago

siwalter commented 2 years ago

It looks like there are still broken mirrors:

dnf.exceptions.RepoError: Failed to download metadata for repo 'appstream': Yum repo downloading error: Downloading error(s): repodata/2db88dbedaf65b54991c82fec49562c00e7e90983a2c47f7f70cf08ca2590a0d-filelists.xml.gz - Cannot download, all mirrors were already tried without success
make: *** [Makefile:128: _build/cs9-qemu-qa-ostree.aarch64.json] Error 1

siwalter commented 2 years ago

@arrfab Can you delete my first comment since I don't have permission to? Thanks.

adrian commented 2 years ago

@siwalter A quick look seems to indicate everything is correct. Can you provide an easy reproducer? The appstream repository looks correct and the file which was mentioned in the now deleted comment exists on mirrors. Hard to tell what problem you are seeing.

siwalter commented 2 years ago

Can you provide an easy reproducer?

If it continues today, I will post another comment with something that should be easy to reproduce. Thanks!

siwalter commented 2 years ago

@adrian, finally, it's all working in our pipeline. Sorry for the noise! m( )m

weatherchnl commented 2 years ago

@adrian Hi, I think I found another mirror still with corrupted repodata (I work with @siwalter and was checking the same errors).

https://ftp.riken.jp/Linux/centos-stream/9-stream/AppStream/aarch64/os/repodata/2db88dbedaf65b54991c82fec49562c00e7e90983a2c47f7f70cf08ca2590a0d-filelists.xml.gz

This file has the checksum 82b706b7a035a6ac642677987a4a31ba5b6a2e967147ef09ef37669613cbca96 after downloading it (at least on my machine). But should have the 2db88d checksum.

Do you see the same if you download it?

weatherchnl commented 2 years ago

Additional mirrors with the same corrupted checksum:

https://mirror.shastacoe.net/centos-stream/9-stream/AppStream/aarch64/os/repodata/2db88dbedaf65b54991c82fec49562c00e7e90983a2c47f7f70cf08ca2590a0d-filelists.xml.gz

https://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/AppStream/aarch64/os/repodata/2db88dbedaf65b54991c82fec49562c00e7e90983a2c47f7f70cf08ca2590a0d-filelists.xml.gz

https://ftp-chi.osuosl.org/pub/centos-stream/9-stream/AppStream/aarch64/os/repodata/2db88dbedaf65b54991c82fec49562c00e7e90983a2c47f7f70cf08ca2590a0d-filelists.xml.gz

https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/aarch64/os/repodata/2db88dbedaf65b54991c82fec49562c00e7e90983a2c47f7f70cf08ca2590a0d-filelists.xml.gz

adrian commented 2 years ago

This is a different problem than this ticket was originally about. Not sure what to do about it.

@arrfab There was recently another ticket with a corrupted file and rsync not correctly fixing it if size and timestamp are correct. Can somebody update the timestamp on the primary mirror?