Since the beginning of December 2020, I'm facing an extremely low download rate.
It is only a few kbit/s from dl.fedoraproject.org and from koji.fedoraproject.org.
Downloads via dnf are fast as lightning, and "normal" downloads are generally too.
I'm living in Germany, I don't have special network settings - no proxy - no VPN.
Additional information :
The same behavior is valid for mirror.openshift.com - downloads are taking "ages".
Also, the websites che.openshift.io and rol.redhat.com need some minutes to load.
Possible root cause (?) :
There seems to exist a peering problem between Amazon (CDN hoster) and Deutsche Telekom (ISP).
Amazon (and maybe other CDN hosters as well ?) doesn't seem to direct traffic to the optimal routes.
As a workaround I have set up a VPN and once I turn it on, download speed increases tremendously.
Proof for suspicion :
(unfortunately in DE)
The network links to our main datacentre are full and there isn't much we can do about this. We are trying to make sure that mirrors get most of the traffic so that they can offload this but they will not be able to help with koji as that needs 100+ TB of disk space to mirror.
[ssmoogen@localhost pkistore (master)]$ host dl.fedoraproject.org
dl.fedoraproject.org has address 18.104.22.168
dl.fedoraproject.org has address 22.214.171.124
dl.fedoraproject.org has address 126.96.36.199
[ssmoogen@localhost pkistore (master)]$ host koji.fedoraproject.org
koji.fedoraproject.org has address 188.8.131.52
koji.fedoraproject.org has address 184.108.40.206
These are not at Amazon so I am not sure the problem is 1:1 the same. [The mirroring you are getting for dnf may be hitting our Amazon CDN which may give the impression we are in Amazon.] I do not know why using a VPN would give you better downloads for koji.fedoraproject.org as it should still have to deal with the same full pipe.
@smooge Is it possible there's a problem between Amazon CDN and Deutsche Telekom?
I have also seen this. Same ISP. It takes forever to download build.log from koji for example. Happy to hear that this is not just me.
@smooge Thank you, Stephen ! What I didn't mention is that I did some research beforehand.
Other users experience it too, what the link to the Telekom discussion I've provided confirms.
I'm not a peering/routing expert - but I asked two friends in Germany to test the same for me.
And both reported back that downloads from dl and koji were as fast as from other websites.
By the way, downloads from access.redhat.com (CDN hoster Akamai) work normally as usual.
@mattdm There could be an issue with Amazon and Deutsche Telekom, but it should not be the problem affecting this since this system isn't in Amazon. It is more likely some other peering issue with DT and networks to the US. This happens regularly with EU2US networks getting saturated or depeered and downloads go to crap. There isn't anything we can do about that :cry: and a 10+ year plan to have a master mirror in EU has never gotten off the ground. :sob:
@clnetbox and @adrian could one of you provide a traceroute or 'mtr -r' from your system to dl.fedoraproject.org It might help pinpoint where the problem is but I think it will end up being partially our full pipe from datacentre to the internet and probably peering from DT to someone else.
$ traceroute dl.fedoraproject.org
traceroute to dl.fedoraproject.org (220.127.116.11), 30 hops max, 60 byte packets
1 fritz.box (XXX.XXX.XXX.X) 5.817 ms 6.549 ms 6.514 ms
2 p3e9bf53e.dip0.t-ipconnect.de (18.104.22.168) 19.749 ms 20.026 ms 20.007 ms
3 22.214.171.124 (126.96.36.199) 24.873 ms 24.855 ms 24.836 ms
4 188.8.131.52 (184.108.40.206) 24.819 ms 25.769 ms 25.749 ms
5 220.127.116.11 (18.104.22.168) 25.731 ms 25.712 ms 28.902 ms
6 be3187.ccr42.fra03.atlas.cogentco.com (22.214.171.124) 28.884 ms be3186.ccr41.fra03.atlas.cogentco.com (126.96.36.199) 18.916 ms be3187.ccr42.fra03.atlas.cogentco.com (188.8.131.52) 16.189 ms
7 be2799.ccr41.par01.atlas.cogentco.com (184.108.40.206) 63.705 ms 27.866 ms 64.472 ms
8 be2318.ccr32.bio02.atlas.cogentco.com (220.127.116.11) 46.237 ms 42.151 ms be2315.ccr31.bio02.atlas.cogentco.com (18.104.22.168) 44.988 ms
9 be2332.ccr42.dca01.atlas.cogentco.com (22.214.171.124) 111.998 ms be2331.ccr41.dca01.atlas.cogentco.com (126.96.36.199) 120.116 ms 120.090 ms
10 be3084.ccr41.iad02.atlas.cogentco.com (188.8.131.52) 115.672 ms be3083.ccr41.iad02.atlas.cogentco.com (184.108.40.206) 141.789 ms be3084.ccr41.iad02.atlas.cogentco.com (220.127.116.11) 141.767 ms
11 18.104.22.168 (22.214.171.124) 122.071 ms 121.308 ms 127.575 ms
12 vrrp2 (126.96.36.199) 148.797 ms gateway (188.8.131.52) 127.892 ms 127.868 ms
30 * * *
$ traceroute koji.fedoraproject.org
traceroute to koji.fedoraproject.org (184.108.40.206), 30 hops max, 60 byte packets
1 fritz.box (XXX.XXX.XXX.X) 3.508 ms 5.389 ms 5.666 ms
2 p3e9bf53e.dip0.t-ipconnect.de (220.127.116.11) 17.122 ms 18.807 ms 19.085 ms
3 18.104.22.168 (22.214.171.124) 22.795 ms 24.334 ms 26.311 ms
4 126.96.36.199 (188.8.131.52) 26.292 ms 32.890 ms 33.239 ms
5 184.108.40.206 (220.127.116.11) 34.433 ms 34.415 ms 34.395 ms
6 be3186.ccr41.fra03.atlas.cogentco.com (18.104.22.168) 34.376 ms be3187.ccr42.fra03.atlas.cogentco.com (22.214.171.124) 20.903 ms 20.196 ms
7 be2799.ccr41.par01.atlas.cogentco.com (126.96.36.199) 29.650 ms be2800.ccr42.par01.atlas.cogentco.com (188.8.131.52) 35.735 ms 36.306 ms
8 be2318.ccr32.bio02.atlas.cogentco.com (184.108.40.206) 47.495 ms be2315.ccr31.bio02.atlas.cogentco.com (220.127.116.11) 42.553 ms be2318.ccr32.bio02.atlas.cogentco.com (18.104.22.168) 45.032 ms
9 be2332.ccr42.dca01.atlas.cogentco.com (22.214.171.124) 112.545 ms * be2331.ccr41.dca01.atlas.cogentco.com (126.96.36.199) 119.326 ms
10 be3083.ccr41.iad02.atlas.cogentco.com (188.8.131.52) 122.657 ms 125.123 ms be3084.ccr41.iad02.atlas.cogentco.com (184.108.40.206) 130.056 ms
11 220.127.116.11 (18.104.22.168) 130.759 ms 131.491 ms 131.473 ms
12 vrrp2 (22.214.171.124) 132.809 ms * 141.247 ms
30 * * *
Thanks. As far as I can tell the issue is going to be our 2 GB pipe to the internet from the data centre. The cogentco link is consistently full and do not know when we will get an upgrade. My guess is that your vpn link may be hitting the secondary link which is less full from a different provider.
@smooge Thank you, Stephen ! This could explain the issue on the Fedora project side at least.
@mattdm Possibly you, as the FPL, can put some pressure on getting this infra upgrade soon ?
The other part can be solved by Red Hat (as CDN hoster customer) contacting Amazon/DTAG ?
Metadata Update from @smooge:
- Issue priority set to: Waiting on External (was: Needs Review)
- Issue tagged with: downloads, high-gain, high-trouble
Hi Matt and Stephen ! Just wanna let you know that everything is back to normal since a few days.
Thanks a lot @mattdm and @smooge for your assistance and interactions - seems that helped. In
addition I had asked a Red Hat TAM to contact the relevant team. Let's hope that it's persistent ...
Metadata Update from @smooge:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
to comment on this ticket.