I just checked all our proxies, and all of them are serving mirrorlists correctly, and none of them have had an outage for a significant period of time.
That being said, the error dnf logged is of a temporary nature, that unfortunately will happen from time to time, and while we are doing are utmost best to remove any occurrences of this, that is a significant project.

It will most likely help significantly if you restore the DNF "retries" value back to its documented (http://dnf.readthedocs.io/en/latest/conf_ref.html) value of 10, from the seemingly current value of 1 ("07:16:09 check_transfer_statuses: No more retries (tried: 1)" from your log), since the retries value is explicitly to make dnf recover in the face of temporary service failures.

Also, note that you are already doing this for other distros (e.g. Tumbleweed) by setting retries=20: https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/copr/backend/files/provision/files/mock/opensuse-tumbleweed-x86_64.cfg#n18

Also note that IP is by design, and hence so is the internet, meant to be an unreliable protocol hence the reason dnf defaults to a sane level of retries to mitigate these sorts of issues.

retries=20 is defined for all the chroots by default mock configs. You can see the chroot mock config that have been used for the build here: https://copr-be.cloud.fedoraproject.org/results/wanzenbug/firefox-nightly/fedora-28-x86_64/00756465-firefox-nightly/configs/fedora-28-x86_64.cfg

That would be worth of investigating if the value is being properly respected. @msuchy , could you please, take a look at it with dnf team?

Complete mock config set:

So, looking at the DNF code, it will not actually retry in this condition, which feels like a DNF error to me.

https://github.com/rpm-software-management/librepo/blob/master/librepo/downloader.c#L1327 - marks this situation as a transfer_err, not fatal_error and not serious_error.

With that combination, the only chance to get retry = TRUE is if can_try_more_mirrors() returns true, per https://github.com/rpm-software-management/librepo/blob/master/librepo/downloader.c#L1765.

In the case of Fedora metalink, it could actually be said that there are more mirrors available of the metalink: all of the (8 - 15) "mirrors" (proxies) that are returned by DNS to be serving this metalink, although I doubt that librepo takes that into account, meaning there aren't any other "mirrors" to try, and it won't retry further, from what I can read.

In other words, I think that this is a DNF bug/RFE to make it retry against the same, or (ideally), against the other DNS results for mirrors.fedoraproject.org.

So, can someone file that dnf bug?

It might also be good to know exactly when it's hitting this to see if we can tie it to any event on our side or network event, but really dnf should retry.

Sorry for delay, issue filed here:

https://bugzilla.redhat.com/show_bug.cgi?id=1591225