#10279 mirrormanager doesn't return correct metalink file for 9-stream
Closed: Fixed a month ago by kevin. Opened 2 months ago by arrfab.

Describe what you would like us to do:


When deploying 9-stream in the CI infra , I noticed an issue :

DNF version: 4.7.0
Command: dnf -d9 makecache 
Installroot: /
Releasever: 9
cachedir: /var/cache/dnf
Base command: makecache
Extra commands: ['-d9', 'makecache']
Making cache files for all metadata files.
baseos: has expired and will be refreshed.
appstream: has expired and will be refreshed.
repo: downloading from remote: baseos
countme: no event for baseos: window already counted
error: Downloading successful, but checksum doesn't match. Calculated: dd496ae5f01df5b27da7fa98cf7509c03f8db34b982e79d320cd9ecbacd4504a80e1fe5cc893447d2d5b66fe624a4d1a06b2eeeb39cddb5be9a7c1e824e8fd4b(sha512)  Expected: 8cbc4c85dca2f889c3df2d8efe73a1f287618afed2ce9b18c560d6bd9ca53a231f07a73ac70100578e7da0a7d23bbec30e48cf52cad7b2b0ad8014df3bc29d0d(sha512)  (http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml).
error: Downloading successful, but checksum doesn't match. Calculated: dd496ae5f01df5b27da7fa98cf7509c03f8db34b982e79d320cd9ecbacd4504a80e1fe5cc893447d2d5b66fe624a4d1a06b2eeeb39cddb5be9a7c1e824e8fd4b(sha512)  Expected: 8cbc4c85dca2f889c3df2d8efe73a1f287618afed2ce9b18c560d6bd9ca53a231f07a73ac70100578e7da0a7d23bbec30e48cf52cad7b2b0ad8014df3bc29d0d(sha512)  (https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml).
error: Downloading successful, but checksum doesn't match. Calculated: dd496ae5f01df5b27da7fa98cf7509c03f8db34b982e79d320cd9ecbacd4504a80e1fe5cc893447d2d5b66fe624a4d1a06b2eeeb39cddb5be9a7c1e824e8fd4b(sha512)  Expected: 8cbc4c85dca2f889c3df2d8efe73a1f287618afed2ce9b18c560d6bd9ca53a231f07a73ac70100578e7da0a7d23bbec30e48cf52cad7b2b0ad8014df3bc29d0d(sha512)  (https://download.cf.centos.org/9-stream/BaseOS/x86_64/os/repodata/repomd.xml).
error: Downloading successful, but checksum doesn't match. Calculated: dd496ae5f01df5b27da7fa98cf7509c03f8db34b982e79d320cd9ecbacd4504a80e1fe5cc893447d2d5b66fe624a4d1a06b2eeeb39cddb5be9a7c1e824e8fd4b(sha512)  Expected: 8cbc4c85dca2f889c3df2d8efe73a1f287618afed2ce9b18c560d6bd9ca53a231f07a73ac70100578e7da0a7d23bbec30e48cf52cad7b2b0ad8014df3bc29d0d(sha512)  (http://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/repodata/repomd.xml).
error: Downloading successful, but checksum doesn't match. Calculated: dd496ae5f01df5b27da7fa98cf7509c03f8db34b982e79d320cd9ecbacd4504a80e1fe5cc893447d2d5b66fe624a4d1a06b2eeeb39cddb5be9a7c1e824e8fd4b(sha512)  Expected: 8cbc4c85dca2f889c3df2d8efe73a1f287618afed2ce9b18c560d6bd9ca53a231f07a73ac70100578e7da0a7d23bbec30e48cf52cad7b2b0ad8014df3bc29d0d(sha512)  (http://mirror.0xem.ma/stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml).
error: Downloading successful, but checksum doesn't match. Calculated: dd496ae5f01df5b27da7fa98cf7509c03f8db34b982e79d320cd9ecbacd4504a80e1fe5cc893447d2d5b66fe624a4d1a06b2eeeb39cddb5be9a7c1e824e8fd4b(sha512)  Expected: 8cbc4c85dca2f889c3df2d8efe73a1f287618afed2ce9b18c560d6bd9ca53a231f07a73ac70100578e7da0a7d23bbec30e48cf52cad7b2b0ad8014df3bc29d0d(sha512)  (https://mirror.0xem.ma/stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml).
error: Downloading successful, but checksum doesn't match. Calculated: dd496ae5f01df5b27da7fa98cf7509c03f8db34b982e79d320cd9ecbacd4504a80e1fe5cc893447d2d5b66fe624a4d1a06b2eeeb39cddb5be9a7c1e824e8fd4b(sha512)  Expected: 8cbc4c85dca2f889c3df2d8efe73a1f287618afed2ce9b18c560d6bd9ca53a231f07a73ac70100578e7da0a7d23bbec30e48cf52cad7b2b0ad8014df3bc29d0d(sha512)  (http://mirrors.up.pt/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml).
error: Downloading successful, but checksum doesn't match. Calculated: dd496ae5f01df5b27da7fa98cf7509c03f8db34b982e79d320cd9ecbacd4504a80e1fe5cc893447d2d5b66fe624a4d1a06b2eeeb39cddb5be9a7c1e824e8fd4b(sha512)  Expected: 8cbc4c85dca2f889c3df2d8efe73a1f287618afed2ce9b18c560d6bd9ca53a231f07a73ac70100578e7da0a7d23bbec30e48cf52cad7b2b0ad8014df3bc29d0d(sha512)  (https://mirrors.up.pt/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml).
error: Downloading successful, but checksum doesn't match. Calculated: dd496ae5f01df5b27da7fa98cf7509c03f8db34b982e79d320cd9ecbacd4504a80e1fe5cc893447d2d5b66fe624a4d1a06b2eeeb39cddb5be9a7c1e824e8fd4b(sha512)  Expected: 8cbc4c85dca2f889c3df2d8efe73a1f287618afed2ce9b18c560d6bd9ca53a231f07a73ac70100578e7da0a7d23bbec30e48cf52cad7b2b0ad8014df3bc29d0d(sha512)  (https://mirror1.hs-esslingen.de/pub/Mirrors/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml).
error: Downloading successful, but checksum doesn't match. Calculated: dd496ae5f01df5b27da7fa98cf7509c03f8db34b982e79d320cd9ecbacd4504a80e1fe5cc893447d2d5b66fe624a4d1a06b2eeeb39cddb5be9a7c1e824e8fd4b(sha512)  Expected: 8cbc4c85dca2f889c3df2d8efe73a1f287618afed2ce9b18c560d6bd9ca53a231f07a73ac70100578e7da0a7d23bbec30e48cf52cad7b2b0ad8014df3bc29d0d(sha512)  (http://mirror1.hs-esslingen.de/pub/Mirrors/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml).
error: Downloading successful, but checksum doesn't match. Calculated: dd496ae5f01df5b27da7fa98cf7509c03f8db34b982e79d320cd9ecbacd4504a80e1fe5cc893447d2d5b66fe624a4d1a06b2eeeb39cddb5be9a7c1e824e8fd4b(sha512)  Expected: 8cbc4c85dca2f889c3df2d8efe73a1f287618afed2ce9b18c560d6bd9ca53a231f07a73ac70100578e7da0a7d23bbec30e48cf52cad7b2b0ad8014df3bc29d0d(sha512)  (http://mirror.netsite.dk/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml).
error: Downloading successful, but checksum doesn't match. Calculated: dd496ae5f01df5b27da7fa98cf7509c03f8db34b982e79d320cd9ecbacd4504a80e1fe5cc893447d2d5b66fe624a4d1a06b2eeeb39cddb5be9a7c1e824e8fd4b(sha512)  Expected: 8cbc4c85dca2f889c3df2d8efe73a1f287618afed2ce9b18c560d6bd9ca53a231f07a73ac70100578e7da0a7d23bbec30e48cf52cad7b2b0ad8014df3bc29d0d(sha512)  (https://mirror.netsite.dk/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml).
error: Downloading successful, but checksum doesn't match. Calculated: dd496ae5f01df5b27da7fa98cf7509c03f8db34b982e79d320cd9ecbacd4504a80e1fe5cc893447d2d5b66fe624a4d1a06b2eeeb39cddb5be9a7c1e824e8fd4b(sha512)  Expected: 8cbc4c85dca2f889c3df2d8efe73a1f287618afed2ce9b18c560d6bd9ca53a231f07a73ac70100578e7da0a7d23bbec30e48cf52cad7b2b0ad8014df3bc29d0d(sha512)  (http://mirror1.cl.netactuate.com/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml).
error: Downloading successful, but checksum doesn't match. Calculated: dd496ae5f01df5b27da7fa98cf7509c03f8db34b982e79d320cd9ecbacd4504a80e1fe5cc893447d2d5b66fe624a4d1a06b2eeeb39cddb5be9a7c1e824e8fd4b(sha512)  Expected: 8cbc4c85dca2f889c3df2d8efe73a1f287618afed2ce9b18c560d6bd9ca53a231f07a73ac70100578e7da0a7d23bbec30e48cf52cad7b2b0ad8014df3bc29d0d(sha512)  (https://mirror1.cl.netactuate.com/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml).
error: Downloading successful, but checksum doesn't match. Calculated: dd496ae5f01df5b27da7fa98cf7509c03f8db34b982e79d320cd9ecbacd4504a80e1fe5cc893447d2d5b66fe624a4d1a06b2eeeb39cddb5be9a7c1e824e8fd4b(sha512)  Expected: 8cbc4c85dca2f889c3df2d8efe73a1f287618afed2ce9b18c560d6bd9ca53a231f07a73ac70100578e7da0a7d23bbec30e48cf52cad7b2b0ad8014df3bc29d0d(sha512)  (http://linuxsoft.cern.ch/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml).
error: Downloading successful, but checksum doesn't match. Calculated: dd496ae5f01df5b27da7fa98cf7509c03f8db34b982e79d320cd9ecbacd4504a80e1fe5cc893447d2d5b66fe624a4d1a06b2eeeb39cddb5be9a7c1e824e8fd4b(sha512)  Expected: 8cbc4c85dca2f889c3df2d8efe73a1f287618afed2ce9b18c560d6bd9ca53a231f07a73ac70100578e7da0a7d23bbec30e48cf52cad7b2b0ad8014df3bc29d0d(sha512)  (https://linuxsoft.cern.ch/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml).
error: Downloading successful, but checksum doesn't match. Calculated: dd496ae5f01df5b27da7fa98cf7509c03f8db34b982e79d320cd9ecbacd4504a80e1fe5cc893447d2d5b66fe624a4d1a06b2eeeb39cddb5be9a7c1e824e8fd4b(sha512)  Expected: 8cbc4c85dca2f889c3df2d8efe73a1f287618afed2ce9b18c560d6bd9ca53a231f07a73ac70100578e7da0a7d23bbec30e48cf52cad7b2b0ad8014df3bc29d0d(sha512)  (http://mirror.2degrees.nz/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml).
error: Downloading successful, but checksum doesn't match. Calculated: dd496ae5f01df5b27da7fa98cf7509c03f8db34b982e79d320cd9ecbacd4504a80e1fe5cc893447d2d5b66fe624a4d1a06b2eeeb39cddb5be9a7c1e824e8fd4b(sha512)  Expected: 8cbc4c85dca2f889c3df2d8efe73a1f287618afed2ce9b18c560d6bd9ca53a231f07a73ac70100578e7da0a7d23bbec30e48cf52cad7b2b0ad8014df3bc29d0d(sha512)  (https://mirror.2degrees.nz/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml).
CentOS Stream 9 - BaseOS                                                               584  B/s | 3.9 kB     00:06    
Errors during downloading metadata for repository 'baseos':
  - Downloading successful, but checksum doesn't match. Calculated: dd496ae5f01df5b27da7fa98cf7509c03f8db34b982e79d320cd9ecbacd4504a80e1fe5cc893447d2d5b66fe624a4d1a06b2eeeb39cddb5be9a7c1e824e8fd4b(sha512)  Expected: 8cbc4c85dca2f889c3df2d8efe73a1f287618afed2ce9b18c560d6bd9ca53a231f07a73ac70100578e7da0a7d23bbec30e48cf52cad7b2b0ad8014df3bc29d0d(sha512) 
Cleaning up.

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/dnf/repo.py", line 574, in load
    ret = self._repo.load()
  File "/usr/lib64/python3.9/site-packages/libdnf/repo.py", line 331, in load
    return _repo.Repo_load(self)
libdnf._error.Error: Failed to download metadata for repo 'baseos': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/dnf/cli/main.py", line 67, in main
    return _main(base, args, cli_class, option_parser_class)
  File "/usr/lib/python3.9/site-packages/dnf/cli/main.py", line 106, in _main
    return cli_run(cli, base)
  File "/usr/lib/python3.9/site-packages/dnf/cli/main.py", line 122, in cli_run
    cli.run()
  File "/usr/lib/python3.9/site-packages/dnf/cli/cli.py", line 1055, in run
    return self.command.run()
  File "/usr/lib/python3.9/site-packages/dnf/cli/commands/makecache.py", line 50, in run
    return self.base.update_cache(timer)
  File "/usr/lib/python3.9/site-packages/dnf/base.py", line 370, in update_cache
    self.fill_sack(load_system_repo=False, load_available_repos=True)  # performs the md sync
  File "/usr/lib/python3.9/site-packages/dnf/base.py", line 399, in fill_sack
    self._add_repo_to_sack(r)
  File "/usr/lib/python3.9/site-packages/dnf/base.py", line 139, in _add_repo_to_sack
    repo.load()
  File "/usr/lib/python3.9/site-packages/dnf/repo.py", line 581, in load
    raise dnf.exceptions.RepoError(str(e))
dnf.exceptions.RepoError: Failed to download metadata for repo 'baseos': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
Error: Failed to download metadata for repo 'baseos': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

But

When I check the authoritative repomd.xml on mirror.stream.centos.org it's up2date :

curl --silent http://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/repodata/repomd.xml|sha512sum
dd496ae5f01df5b27da7fa98cf7509c03f8db34b982e79d320cd9ecbacd4504a80e1fe5cc893447d2d5b66fe624a4d1a06b2eeeb39cddb5be9a7c1e824e8fd4b  -

Also the file for repomd.xml is from oct 14th and I notice the issue still on oct 16th, so it looks like if mirrormanager crawler was stuck on 9-stream and probably with previous repomd.xml, meaning that it's invalidating all real up2date mirrors, and so no dnf transaction is possible


Should be fixed now. Changes should be live in one hour. This is the same as https://pagure.io/centos-infra/issue/482

if you have a potential fix we can implement it but all nodes behind mirror.stream.centos.org are up2date and repomd.xml from two days ago so how is that possible that mirrormanager couldn't see the newer repomd.xml earlier ? Also, is that possible to always return something that dnf users can use instead of just having empty list ? for mirrorlist.centos.org we had previous entry in cache but also because we aren't using metalink .
So let's assume that if crawler finds a new repomd.xml but then it would invalidate (don't know the crawler time) all mirrors until some have it (in the next crawl time) ?

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: dev, medium-gain, medium-trouble

a month ago

I think this has been solved. If there's anything further to do, feel free to reopen.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

a month ago

Login to comment on this ticket.

Metadata
Boards 1
dev Status: Backlog