I've been in contact with DNF team regarding improving DNF time when refreshing repos. Yesterday they've achieved a large speedup, and now they seem mostly limited by Fedora's MirrorManager response time. This ticket is a request to look into the matter and improve Fedora's MM response time if possible.
If I try to download the metalink from Fedora's MM, it usually takes about 1 second (from the Brno office):
$ time wget -4 -O /dev/null 'https://mirrors.fedoraproject.org/metalink?repo=fedora-29&arch=x86_64' --2019-02-21 11:05:30-- https://mirrors.fedoraproject.org/metalink?repo=fedora-29&arch=x86_64 Resolving mirrors.fedoraproject.org (mirrors.fedoraproject.org)... 140.211.169.196, 67.219.144.68, 8.43.85.67, ... Connecting to mirrors.fedoraproject.org (mirrors.fedoraproject.org)|140.211.169.196|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 28367 (28K) [application/metalink+xml] Saving to: ‘/dev/null’ /dev/null 100%[===============================================>] 27.70K 149KB/s in 0.2s 2019-02-21 11:05:31 (149 KB/s) - ‘/dev/null’ saved [28367/28367] real 0m1.081s user 0m0.090s sys 0m0.016s
However, when I try to same approach with rpmfusion MM, it only takes about 200 ms:
$ time wget -4 -O /dev/null 'https://mirrors.rpmfusion.org/metalink?repo=free-fedora-29&arch=x86_64' --2019-02-21 11:09:14-- https://mirrors.rpmfusion.org/metalink?repo=free-fedora-29&arch=x86_64 Resolving mirrors.rpmfusion.org (mirrors.rpmfusion.org)... 78.47.19.71, 51.15.112.174, 158.69.195.211, ... Connecting to mirrors.rpmfusion.org (mirrors.rpmfusion.org)|78.47.19.71|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 10429 (10K) [application/metalink+xml] Saving to: ‘/dev/null’ /dev/null 100%[===============================================>] 10.18K --.-KB/s in 0s 2019-02-21 11:09:15 (74.7 MB/s) - ‘/dev/null’ saved [10429/10429] real 0m0.208s user 0m0.089s sys 0m0.014s
Why is Fedora's MM so much slower? Can something be done to improve the response time? That would make DNF operations considerably faster (it often checks the metalink for updates, and with 4-6 fedora repos enabled in the system, it makes the delay quite noticeable).
Thank you!
no particular date, a general improvement
always applies
sub-optimal user experience with DNF/PackageKit operations
CC @dmach
Fedora MM you have connected to is located in western USA, in OSUOSL to be precise. There are known networking issues between RH Brno offices and OSUOSL that are under investigation (private ticket #6641). But rpmfusion MM instance you used seems to be located in Germany, much closer to Brno.
Could somebody from North America post their response times to Fedora and rpmfusion MM (see commands above)? Thanks.
@kparal Try adding the following line to /etc/hosts to see if it makes things better for you:
185.141.165.254 mirrors.fedoraproject.org
This is for testing only and I don't recommend using this setting in production.
That is certainly a major improvement:
$ time wget -4 -O /dev/null 'https://mirrors.fedoraproject.org/metalink?repo=fedora-29&arch=x86_64' --2019-02-21 12:20:24-- https://mirrors.fedoraproject.org/metalink?repo=fedora-29&arch=x86_64 Resolving mirrors.fedoraproject.org (mirrors.fedoraproject.org)... 185.141.165.254 Connecting to mirrors.fedoraproject.org (mirrors.fedoraproject.org)|185.141.165.254|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 28367 (28K) [application/metalink+xml] Saving to: ‘/dev/null’ /dev/null 100%[===============================================>] 27.70K --.-KB/s in 0.01s 2019-02-21 12:20:24 (2.21 MB/s) - ‘/dev/null’ saved [28367/28367] real 0m0.163s user 0m0.058s sys 0m0.014s
So it seems MM is not slow, we're just affected by some routing slowdown.
My conclusion would be: Most of rpmfusion infrastructure is located in Europe, so you have a high chance of hitting a fast, European MM, while most of Fedora infrastructure is located in North America, so you have low chances of hitting European MM.
@mizdebsk @kparal yes that is the usual case. Out of the current 12 active proxies, we have very few in the EU (and have actually lost 2 in the last couple of years). This means that you are most likely going to hit a long route to get updates.
Getting adequate resources in Europe has been a Sisyphean task, and not one I am able to give any sort of timeline on. OK to close this as something?
Is 1s response time expected when going Europe <-> North America, or is that going to improve (possibly relevant to the private ticket linked)? I'd hope for a faster roundtrip.
But the MM manager itself doesn't seem to be slow, and if the network delay is getting solved in a different ticket, yes, we can close this one. Thanks everyone for responses.
One second time is expected. Establishing new connection requires several packet round-trips - first TCP handshake, then TLS handshake.
This is a generic problem and doesn't affect only MM, but to a somewhat lesser extent most of Fedora apps and websites too. Even for requests that need to be handled by backend servers in US, using a close proxy is faster because the time required to establish connection is reduced greatly - proxy servers communicate with backends over already-established VPN connection. And proxies may even have already-established HTTP session with backend server. Moreover, some requests are handled directly at proxy level - that includes MM requests, or cached static data.
The problem won't be solved by fixing routing problems, which is the topic of the private ticket mentioned above.
Note that mirrors.fedoraproject.org is NOT a single host at osuosl... it maps to all our proxy servers in the "region".
We have dns views for 'EU' and 'NA' and 'DEFAULT' (default gets used for ipv6)
ie,
~ host mirrors.fedoraproject.org mirrors.fedoraproject.org is an alias for wildcard.fedoraproject.org. wildcard.fedoraproject.org has address 8.43.85.73 wildcard.fedoraproject.org has address 67.203.2.67 wildcard.fedoraproject.org has address 152.19.134.198 wildcard.fedoraproject.org has address 67.219.144.68 wildcard.fedoraproject.org has address 140.211.169.206 wildcard.fedoraproject.org has address 140.211.169.196 wildcard.fedoraproject.org has address 152.19.134.142 wildcard.fedoraproject.org has address 8.43.85.67 wildcard.fedoraproject.org has address 209.132.181.16 wildcard.fedoraproject.org has address 209.132.190.2 wildcard.fedoraproject.org has address 209.132.181.15 wildcard.fedoraproject.org has IPv6 address 2610:28:3090:3001:dead:beef:cafe:fed3 wildcard.fedoraproject.org has IPv6 address 2604:1580:fe00:0:dead:beef:cafe:fed1 wildcard.fedoraproject.org has IPv6 address 2605:bc80:3010:600:dead:beef:cafe:feda wildcard.fedoraproject.org has IPv6 address 2605:bc80:3010:600:dead:beef:cafe:fed9
So, @kparal can you check your dns for a list, then test each one and see if you can see what proxies are slow for you? It might be they are all the same and there's not much we can do now, but it might be there is a slow one you are hitting we can look at? If your list there is all NA servers, then are you using aws? There seems to be some issue with us misdetecting aws as NA.
@kparal
1 second is appropriate for the 10,000+ km trip (it is 8880 if the wire was direct but you have 20+ retransmits in between and non of the wires are straight). You basically got the farthest away proxy from Brno in EU. (assuming you did this from Brno) Add in tcp packet overhead, any VPNs, etc and that most transits are at 80% capacity and that is normal.
Each of the mirrors that @nirik points out above will have different times and most should be less than 1s
I'm not on the corporate network today, working from home (still Brno). Here's my list:
$ host mirrors.fedoraproject.org mirrors.fedoraproject.org is an alias for wildcard.fedoraproject.org. wildcard.fedoraproject.org has address 140.211.169.196 wildcard.fedoraproject.org has address 209.132.181.15 wildcard.fedoraproject.org has address 152.19.134.198 wildcard.fedoraproject.org has address 209.132.181.16 wildcard.fedoraproject.org has address 67.219.144.68 wildcard.fedoraproject.org has address 152.19.134.142 wildcard.fedoraproject.org has address 67.203.2.67 wildcard.fedoraproject.org has address 140.211.169.206 wildcard.fedoraproject.org has address 8.43.85.73 wildcard.fedoraproject.org has address 8.43.85.67 wildcard.fedoraproject.org has address 209.132.190.2 wildcard.fedoraproject.org has IPv6 address 2604:1580:fe00:0:dead:beef:cafe:fed1 wildcard.fedoraproject.org has IPv6 address 2605:bc80:3010:600:dead:beef:cafe:feda wildcard.fedoraproject.org has IPv6 address 2605:bc80:3010:600:dead:beef:cafe:fed9 wildcard.fedoraproject.org has IPv6 address 2610:28:3090:3001:dead:beef:cafe:fed3
All the proxies (I only tested IPv4 ones) are equally slow, about 1s response time (1.0s - 1.2s).
I don't see any of European proxies on the list. Are you perhaps using DNS resolution service or corporate DNS resolvers over VPN? If yes, then you may be given American proxies only. Try with recursive DNS resolver, or use local DNS resolver.
OK our ability to give out specific DNS zones relies on a couple of 'hacks' which can be fooled by various things varying from VPNs to specific DNS to outdated geoip data. I need to know what your ip address is for this and what your DNS resolver is.
However for true testing let us go with another hack in your /etc/hosts 85.236.55.6 wildcard.fedoraproject.org mirrors.fedoraproject.org 185.141.165.254 wildcard.fedoraproject.org mirrors.fedoraproject.org
From this you will get 2 mirrors in Germany and you can get better testing. If you are still seeing 1 sec then we have something going on.. because it should be anywhere from 0.288 to 0.6 seconds per request in the US. Our static pages serve at a similar time so this is just the minimum I can see us delivering.
Yeah, I'm sorry, I was on company VPN (Brno's VPN), I haven't realized it can affect this. Without being on VPN, I see this list:
$ host mirrors.fedoraproject.org mirrors.fedoraproject.org is an alias for wildcard.fedoraproject.org. wildcard.fedoraproject.org has address 209.132.190.2 wildcard.fedoraproject.org has address 185.141.165.254 wildcard.fedoraproject.org has address 152.19.134.142 wildcard.fedoraproject.org has address 209.132.181.15 wildcard.fedoraproject.org has address 8.43.85.67 wildcard.fedoraproject.org has address 209.132.181.16 wildcard.fedoraproject.org has address 152.19.134.198 wildcard.fedoraproject.org has address 140.211.169.206 wildcard.fedoraproject.org has address 67.219.144.68 wildcard.fedoraproject.org has address 85.236.55.6 wildcard.fedoraproject.org has IPv6 address 2001:4178:2:1269::fed2 wildcard.fedoraproject.org has IPv6 address 2605:bc80:3010:600:dead:beef:cafe:fed9 wildcard.fedoraproject.org has IPv6 address 2610:28:3090:3001:dead:beef:cafe:fed3 wildcard.fedoraproject.org has IPv6 address 2604:1580:fe00:0:dead:beef:cafe:fed1
and these times:
209.132.190.2 0.7s 185.141.165.254 0.2s 152.19.134.142 0.7s 209.132.181.15 0.7s 8.43.85.67 0.6s 209.132.181.16 0.8s 152.19.134.198 0.7s 140.211.169.206 1.1s 67.219.144.68 0.7s 85.236.55.6 0.2s
85.236.55.6 wildcard.fedoraproject.org mirrors.fedoraproject.org 185.141.165.254 wildcard.fedoraproject.org mirrors.fedoraproject.org
That's 0.2s.
I also asked a colleague currently sitting in the Brno office to send me his output of DNS, here it is:
$ host mirrors.fedoraproject.org mirrors.fedoraproject.org is an alias for wildcard.fedoraproject.org. wildcard.fedoraproject.org has address 209.132.181.16 wildcard.fedoraproject.org has address 8.43.85.73 wildcard.fedoraproject.org has address 67.203.2.67 wildcard.fedoraproject.org has address 67.219.144.68 wildcard.fedoraproject.org has address 140.211.169.196 wildcard.fedoraproject.org has address 152.19.134.142 wildcard.fedoraproject.org has address 209.132.181.15 wildcard.fedoraproject.org has address 209.132.190.2 wildcard.fedoraproject.org has address 8.43.85.67 wildcard.fedoraproject.org has address 140.211.169.206 wildcard.fedoraproject.org has address 152.19.134.198 wildcard.fedoraproject.org has IPv6 address 2610:28:3090:3001:dead:beef:cafe:fed3 wildcard.fedoraproject.org has IPv6 address 2605:bc80:3010:600:dead:beef:cafe:feda wildcard.fedoraproject.org has IPv6 address 2604:1580:fe00:0:dead:beef:cafe:fed1 wildcard.fedoraproject.org has IPv6 address 2605:bc80:3010:600:dead:beef:cafe:fed9
It's the same output as I received when connected over VPN. Does this mean that when we're in the Brno office, we're only getting USA proxies, and that's why we consistently see slow responses? Can something be done for this case?
Does this mean that when we're in the Brno office, we're only getting USA proxies, and that's why we consistently see slow responses?
Partially yes. The other part is what I said before - there are too few EU proxies, so you will most often get NA proxy anyway.
Can something be done for this case?
I'm afraid that not on Fedora infra side. You can talk to your corporate IT to consider using a EU DNS resolver for internal 10.x.x.x IP ranges that are known to be in EU.
Donating enough proxies in EU could solve the other part of this issue.
OK, thanks. I think we can close this ticket.
Metadata Update from @mizdebsk: - Issue close_status updated to: Will Not/Can Not fix - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.