RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1706321

Is there any validation/monitoring for Fedora mirrors?

Yes, every 12 hours we scan all mirrors.

Sometimes we get a broken metadata on one of the mirrors. For example
repo: downloading from remote: updates-modular
error: Curl error (52): Server returned nothing (no headers, no data) for https://mirror.yandex.ru/fedora/linux/updates/30/Modular/x86_64/repodata/9bb278c48d73a3fe356fd255177f55a7900c4d598e7aa96f1b192aaba2406ce8-filelists.xml.zck [Empty reply from server] (https://mirror.yandex.ru/fedora/linux/updates/30/Modular/x86_64/repodata/9bb278c48d73a3fe356fd255177f55a7900c4d598e7aa96f1b192aaba2406ce8-filelists.xml.zck).
error: Curl error (52): Server returned nothing (no headers, no data) for https://mirror.yandex.ru/fedora/linux/updates/30/Modular/x86_64/repodata/f850ebe08c1f80a08866f62242b0e6a7091e500ef2a205bd3dc1c16640f06f1c-primary.xml.zck [Empty reply from server] (https://mirror.yandex.ru/fedora/linux/updates/30/Modular/x86_64/repodata/f850ebe08c1f80a08866f62242b0e6a7091e500ef2a205bd3dc1c16640f06f1c-primary.xml.zck).
error: Zchunk header checksum didn't match expected checksum (http://mirror.yandex.ru/fedora/linux/updates/30/Modular/x86_64/repodata/f850ebe08c1f80a08866f62242b0e6a7091e500ef2a205bd3dc1c16640f06f1c-primary.xml.zck).
error: Zchunk header checksum didn't match expected checksum (http://mirror.yandex.ru/fedora/linux/updates/30/Modular/x86_64/repodata/9bb278c48d73a3fe356fd255177f55a7900c4d598e7aa96f1b192aaba2406ce8-filelists.xml.zck).
Fedora Modular 30 - x86_64 - Updates 1.4 MB/s | 877 kB 00:00
Cannot download 'https://mirrors.fedoraproject.org/metalink?repo=updates-released-modular-f30&arch=x86_64': Checksum error /var/cache/dnf/updates-modular-783da5de2e38c644/repodata/f850ebe08c1f80a08866f62242b0e6a7091e500ef2a205bd3dc1c16640f06f1c-primary.xml.zck: Unable to read zchunk lead.
Failed to synchronize cache for repo 'updates-modular'

This is a new one

Can we add a periodic validation for mirrors in the fedora mirror list, so that we temporarily exclude broken mirrors and don't send them to the enduser?

As mentioned above we already do this. But as it takes up to 12 hours to validate all mirrors and as mirrors can change their state during these 12 hours it is complicated. Therefore yum/dnf usually skips broken mirrors. We tell the clients the checksum of valid repomd.xml files and dnf/yum only uses mirrors which have these repomd.xml file. Even if the mirror is on the list of valid mirrors, yum/dnf goes to the next mirror.

Not knowing anything about zchunk, it seems that zchunk downloads are not as robust to missing files as repomd.xml downloads.

From my current point of view, we cannot scan mirrors faster and mirrors can fail any time, therefore zchunk downloading needs to be more robust against wrong content or missing content and use the next mirror from the mirrorlist.

This is a typical problem for Yandex mirrors. Lots of people in Russian Fedora Telegram group complains about this.

Another major issue with Yandex is a repo inconsistent condition (metadata updated, packages - don't or vice versa).

I think that Yandex mirrors need to be completely disabled for all Fedora users due to their internal instability.

So, I think we can close this:

• updated librepo will work around the zchunk related issues with yandex
• we do validate mirrors, it just takes a bit.

If there's further action items here I missed, please do reopen or let us know.

Thanks!

:minibus: