Hey @adrian / @abompard, I’m seeing issues with download redirects for EPEL mirrors, where some are redirecting to Cloudfront instead of the intended target. Michal suggested it could be a mirror or mirror manager issue. Any pointers on how to debug this?

This is super strange. There seems to be no URL in the database which matches mirrors.edge.kernel.org:

mirrormanager2=> select * from host_category_url where url like '%kernel%';
  id   | host_category_id |                      url                       | private 
-------+------------------+------------------------------------------------+---------
 11644 |             8740 | rsync://primary.mirrors.kernel.org/fedora      | f
 11645 |             8741 | rsync://primary.mirrors.kernel.org/fedora-epel | f
 11853 |             8825 | https://na.edge.kernel.org/fedora              | f
 11854 |             8826 | https://na.edge.kernel.org/fedora-epel         | f
 11855 |             8827 | https://eu.edge.kernel.org/fedora              | f
 11856 |             8828 | https://eu.edge.kernel.org/fedora-epel         | f
 11880 |             8836 | https://ap.edge.kernel.org/fedora              | f
 11881 |             8837 | https://ap.edge.kernel.org/fedora-epel         | f
(8 rows)

So, not sure. All kernel.org EPEL mirrors have the correct URL /fedora-epel/. So this is very strange. I need more information how this is happening. What command triggers this. From which IP address.

Hello @adrian thanks for responding. I am seeing this regularly in OpenStack CI infrastructure. For example in this job: https://zuul.opendev.org/t/openstack/build/ec58cd8b52db4eeaa6fb9510c5df89cd

It was running in an OVH virtual machine in region BHS1 (Montréal, Canada). The source IP should be 158.69.70.181.

Specific details that could help troubleshooting:

/etc/yum.repos.d/epel.repo: https://b77df92de44f30f36c94-8e50817bcbd575f2341914da48a13cbe.ssl.cf2.rackcdn.com/openstack/ec58cd8b52db4eeaa6fb9510c5df89cd/primary/system_logs/yum.repos.d/epel.repo

dnf logs: https://b77df92de44f30f36c94-8e50817bcbd575f2341914da48a13cbe.ssl.cf2.rackcdn.com/openstack/ec58cd8b52db4eeaa6fb9510c5df89cd/primary/system_logs/dnf.librepo.txt

Job output (VM details at the beginning): https://b77df92de44f30f36c94-8e50817bcbd575f2341914da48a13cbe.ssl.cf2.rackcdn.com/openstack/ec58cd8b52db4eeaa6fb9510c5df89cd/job-output.txt

And what is triggering it is the Ansible ansible.builtin.package module installing the gdisk package, which is only in EPEL for EL10.

Thanks for the details. That makes it easier.

My first question is, however, why aren't you using metalinks? You are using baseurl in your repository file and pointing to an URL which directs you to a single host. If that fails there is no fallback. With metalinks this would not happen. The main question is why did you change the epel.repo file.

To let you know what happens in the background. download.fedoraproject.org is a redirector, which does the following in the background for your example:

$ curl 'https://mirrors.fedoraproject.org/mirrorlist?path=pub/epel/10/Everything/x86_64/Packages/g/gdisk-1.0.10-2.el10_0.x86_64.rpm&ip=158.69.70.181&redirect=1' -v
[...]
location: http://mirror.layeronline.com/epel/10/Everything/x86_64/Packages/g/gdisk-1.0.10-2.el10_0.x86_64.rpm
[...]

So you are being redirected to http://mirror.layeronline.com/epel/10/Everything/x86_64/Packages/g/gdisk-1.0.10-2.el10_0.x86_64.rpm.

For some strange reason this mirror just does a redirect:

$ curl  http://mirror.layeronline.com/epel/10/Everything/x86_64/Packages/g/gdisk-1.0.10-2.el10_0.x86_64.rpm -v
[...]
Location: https://mirrors.edge.kernel.org/epel/10/Everything/x86_64/Packages/g/gdisk-1.0.10-2.el10_0.x86_64.rpm

So, this mirror server redirects you to a non-existing URL. This is not something we can control. That is some strange setting in that mirror. I have disabled this mirror for now and in about 60 minutes this error should be gone.

You should really change your repo file to use metalinks to avoid this error scenario.

Thank you for explaining and fixing the issue so quickly!

I agree our repository configuration is not ideal. We normally template the repo file to use a local mirror in the OpenStack CI infrastructure, but we don't have EPEL 10 mirrored yet, only EPEL 9. That's how we end up using the default download server instead.

If we remain without a local mirror I will change to use metalinks as suggested.