#7587 Improve MirrorManager response time
Closed: Will Not/Can Not fix 5 years ago by mizdebsk. Opened 5 years ago by kparal.

Describe what you need us to do:

I've been in contact with DNF team regarding improving DNF time when refreshing repos. Yesterday they've achieved a large speedup, and now they seem mostly limited by Fedora's MirrorManager response time. This ticket is a request to look into the matter and improve Fedora's MM response time if possible.

If I try to download the metalink from Fedora's MM, it usually takes about 1 second (from the Brno office):

$ time wget -4 -O /dev/null 'https://mirrors.fedoraproject.org/metalink?repo=fedora-29&arch=x86_64'
--2019-02-21 11:05:30--  https://mirrors.fedoraproject.org/metalink?repo=fedora-29&arch=x86_64
Resolving mirrors.fedoraproject.org (mirrors.fedoraproject.org)... 140.211.169.196, 67.219.144.68, 8.43.85.67, ...
Connecting to mirrors.fedoraproject.org (mirrors.fedoraproject.org)|140.211.169.196|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 28367 (28K) [application/metalink+xml]
Saving to: ‘/dev/null’

/dev/null                     100%[===============================================>]  27.70K   149KB/s    in 0.2s    

2019-02-21 11:05:31 (149 KB/s) - ‘/dev/null’ saved [28367/28367]


real    0m1.081s
user    0m0.090s
sys 0m0.016s

However, when I try to same approach with rpmfusion MM, it only takes about 200 ms:

$ time wget -4 -O /dev/null 'https://mirrors.rpmfusion.org/metalink?repo=free-fedora-29&arch=x86_64'
--2019-02-21 11:09:14--  https://mirrors.rpmfusion.org/metalink?repo=free-fedora-29&arch=x86_64
Resolving mirrors.rpmfusion.org (mirrors.rpmfusion.org)... 78.47.19.71, 51.15.112.174, 158.69.195.211, ...
Connecting to mirrors.rpmfusion.org (mirrors.rpmfusion.org)|78.47.19.71|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10429 (10K) [application/metalink+xml]
Saving to: ‘/dev/null’

/dev/null                     100%[===============================================>]  10.18K  --.-KB/s    in 0s      

2019-02-21 11:09:15 (74.7 MB/s) - ‘/dev/null’ saved [10429/10429]


real    0m0.208s
user    0m0.089s
sys 0m0.014s

Why is Fedora's MM so much slower? Can something be done to improve the response time? That would make DNF operations considerably faster (it often checks the metalink for updates, and with 4-6 fedora repos enabled in the system, it makes the delay quite noticeable).

Thank you!

When do you need this? (YYYY/MM/DD)

no particular date, a general improvement

When is this no longer needed or useful? (YYYY/MM/DD)

always applies

If we cannot complete your request, what is the impact?

sub-optimal user experience with DNF/PackageKit operations

CC @dmach


Fedora MM you have connected to is located in western USA, in OSUOSL to be precise. There are known networking issues between RH Brno offices and OSUOSL that are under investigation (private ticket #6641). But rpmfusion MM instance you used seems to be located in Germany, much closer to Brno.

Could somebody from North America post their response times to Fedora and rpmfusion MM (see commands above)? Thanks.

@kparal Try adding the following line to /etc/hosts to see if it makes things better for you:

185.141.165.254 mirrors.fedoraproject.org

This is for testing only and I don't recommend using this setting in production.

That is certainly a major improvement:

$ time wget -4 -O /dev/null 'https://mirrors.fedoraproject.org/metalink?repo=fedora-29&arch=x86_64'
--2019-02-21 12:20:24--  https://mirrors.fedoraproject.org/metalink?repo=fedora-29&arch=x86_64
Resolving mirrors.fedoraproject.org (mirrors.fedoraproject.org)... 185.141.165.254
Connecting to mirrors.fedoraproject.org (mirrors.fedoraproject.org)|185.141.165.254|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 28367 (28K) [application/metalink+xml]
Saving to: ‘/dev/null’

/dev/null                     100%[===============================================>]  27.70K  --.-KB/s    in 0.01s   

2019-02-21 12:20:24 (2.21 MB/s) - ‘/dev/null’ saved [28367/28367]


real    0m0.163s
user    0m0.058s
sys 0m0.014s

So it seems MM is not slow, we're just affected by some routing slowdown.

My conclusion would be: Most of rpmfusion infrastructure is located in Europe, so you have a high chance of hitting a fast, European MM, while most of Fedora infrastructure is located in North America, so you have low chances of hitting European MM.

@mizdebsk @kparal yes that is the usual case. Out of the current 12 active proxies, we have very few in the EU (and have actually lost 2 in the last couple of years). This means that you are most likely going to hit a long route to get updates.

Getting adequate resources in Europe has been a Sisyphean task, and not one I am able to give any sort of timeline on. OK to close this as something?

Is 1s response time expected when going Europe <-> North America, or is that going to improve (possibly relevant to the private ticket linked)? I'd hope for a faster roundtrip.

But the MM manager itself doesn't seem to be slow, and if the network delay is getting solved in a different ticket, yes, we can close this one. Thanks everyone for responses.

One second time is expected. Establishing new connection requires several packet round-trips - first TCP handshake, then TLS handshake.

This is a generic problem and doesn't affect only MM, but to a somewhat lesser extent most of Fedora apps and websites too. Even for requests that need to be handled by backend servers in US, using a close proxy is faster because the time required to establish connection is reduced greatly - proxy servers communicate with backends over already-established VPN connection. And proxies may even have already-established HTTP session with backend server. Moreover, some requests are handled directly at proxy level - that includes MM requests, or cached static data.

The problem won't be solved by fixing routing problems, which is the topic of the private ticket mentioned above.

Note that mirrors.fedoraproject.org is NOT a single host at osuosl... it maps to all our proxy servers in the "region".

We have dns views for 'EU' and 'NA' and 'DEFAULT' (default gets used for ipv6)

ie,

~ host mirrors.fedoraproject.org 
mirrors.fedoraproject.org is an alias for wildcard.fedoraproject.org.
wildcard.fedoraproject.org has address 8.43.85.73
wildcard.fedoraproject.org has address 67.203.2.67
wildcard.fedoraproject.org has address 152.19.134.198
wildcard.fedoraproject.org has address 67.219.144.68
wildcard.fedoraproject.org has address 140.211.169.206
wildcard.fedoraproject.org has address 140.211.169.196
wildcard.fedoraproject.org has address 152.19.134.142
wildcard.fedoraproject.org has address 8.43.85.67
wildcard.fedoraproject.org has address 209.132.181.16
wildcard.fedoraproject.org has address 209.132.190.2
wildcard.fedoraproject.org has address 209.132.181.15
wildcard.fedoraproject.org has IPv6 address 2610:28:3090:3001:dead:beef:cafe:fed3
wildcard.fedoraproject.org has IPv6 address 2604:1580:fe00:0:dead:beef:cafe:fed1
wildcard.fedoraproject.org has IPv6 address 2605:bc80:3010:600:dead:beef:cafe:feda
wildcard.fedoraproject.org has IPv6 address 2605:bc80:3010:600:dead:beef:cafe:fed9

So, @kparal can you check your dns for a list, then test each one and see if you can see what proxies are slow for you? It might be they are all the same and there's not much we can do now, but it might be there is a slow one you are hitting we can look at? If your list there is all NA servers, then are you using aws? There seems to be some issue with us misdetecting aws as NA.

@kparal

1 second is appropriate for the 10,000+ km trip (it is 8880 if the wire was direct but you have 20+ retransmits in between and non of the wires are straight). You basically got the farthest away proxy from Brno in EU. (assuming you did this from Brno) Add in tcp packet overhead, any VPNs, etc and that most transits are at 80% capacity and that is normal.

Each of the mirrors that @nirik points out above will have different times and most should be less than 1s

I'm not on the corporate network today, working from home (still Brno). Here's my list:

$ host mirrors.fedoraproject.org
mirrors.fedoraproject.org is an alias for wildcard.fedoraproject.org.
wildcard.fedoraproject.org has address 140.211.169.196
wildcard.fedoraproject.org has address 209.132.181.15
wildcard.fedoraproject.org has address 152.19.134.198
wildcard.fedoraproject.org has address 209.132.181.16
wildcard.fedoraproject.org has address 67.219.144.68
wildcard.fedoraproject.org has address 152.19.134.142
wildcard.fedoraproject.org has address 67.203.2.67
wildcard.fedoraproject.org has address 140.211.169.206
wildcard.fedoraproject.org has address 8.43.85.73
wildcard.fedoraproject.org has address 8.43.85.67
wildcard.fedoraproject.org has address 209.132.190.2
wildcard.fedoraproject.org has IPv6 address 2604:1580:fe00:0:dead:beef:cafe:fed1
wildcard.fedoraproject.org has IPv6 address 2605:bc80:3010:600:dead:beef:cafe:feda
wildcard.fedoraproject.org has IPv6 address 2605:bc80:3010:600:dead:beef:cafe:fed9
wildcard.fedoraproject.org has IPv6 address 2610:28:3090:3001:dead:beef:cafe:fed3

All the proxies (I only tested IPv4 ones) are equally slow, about 1s response time (1.0s - 1.2s).

I don't see any of European proxies on the list. Are you perhaps using DNS resolution service or corporate DNS resolvers over VPN? If yes, then you may be given American proxies only. Try with recursive DNS resolver, or use local DNS resolver.

OK our ability to give out specific DNS zones relies on a couple of 'hacks' which can be fooled by various things varying from VPNs to specific DNS to outdated geoip data. I need to know what your ip address is for this and what your DNS resolver is.

However for true testing let us go with another hack in your /etc/hosts
85.236.55.6 wildcard.fedoraproject.org mirrors.fedoraproject.org
185.141.165.254 wildcard.fedoraproject.org mirrors.fedoraproject.org

From this you will get 2 mirrors in Germany and you can get better testing. If you are still seeing 1 sec then we have something going on.. because it should be anywhere from 0.288 to 0.6 seconds per request in the US. Our static pages serve at a similar time so this is just the minimum I can see us delivering.

Yeah, I'm sorry, I was on company VPN (Brno's VPN), I haven't realized it can affect this. Without being on VPN, I see this list:

$ host mirrors.fedoraproject.org
mirrors.fedoraproject.org is an alias for wildcard.fedoraproject.org.
wildcard.fedoraproject.org has address 209.132.190.2
wildcard.fedoraproject.org has address 185.141.165.254
wildcard.fedoraproject.org has address 152.19.134.142
wildcard.fedoraproject.org has address 209.132.181.15
wildcard.fedoraproject.org has address 8.43.85.67
wildcard.fedoraproject.org has address 209.132.181.16
wildcard.fedoraproject.org has address 152.19.134.198
wildcard.fedoraproject.org has address 140.211.169.206
wildcard.fedoraproject.org has address 67.219.144.68
wildcard.fedoraproject.org has address 85.236.55.6
wildcard.fedoraproject.org has IPv6 address 2001:4178:2:1269::fed2
wildcard.fedoraproject.org has IPv6 address 2605:bc80:3010:600:dead:beef:cafe:fed9
wildcard.fedoraproject.org has IPv6 address 2610:28:3090:3001:dead:beef:cafe:fed3
wildcard.fedoraproject.org has IPv6 address 2604:1580:fe00:0:dead:beef:cafe:fed1

and these times:

209.132.190.2  0.7s
185.141.165.254  0.2s
152.19.134.142  0.7s
209.132.181.15  0.7s
8.43.85.67  0.6s
209.132.181.16  0.8s
152.19.134.198  0.7s
140.211.169.206  1.1s
67.219.144.68  0.7s
85.236.55.6  0.2s

85.236.55.6 wildcard.fedoraproject.org mirrors.fedoraproject.org
185.141.165.254 wildcard.fedoraproject.org mirrors.fedoraproject.org

That's 0.2s.

I also asked a colleague currently sitting in the Brno office to send me his output of DNS, here it is:

$ host mirrors.fedoraproject.org
mirrors.fedoraproject.org is an alias for wildcard.fedoraproject.org.
wildcard.fedoraproject.org has address 209.132.181.16
wildcard.fedoraproject.org has address 8.43.85.73
wildcard.fedoraproject.org has address 67.203.2.67
wildcard.fedoraproject.org has address 67.219.144.68
wildcard.fedoraproject.org has address 140.211.169.196
wildcard.fedoraproject.org has address 152.19.134.142
wildcard.fedoraproject.org has address 209.132.181.15
wildcard.fedoraproject.org has address 209.132.190.2
wildcard.fedoraproject.org has address 8.43.85.67
wildcard.fedoraproject.org has address 140.211.169.206
wildcard.fedoraproject.org has address 152.19.134.198
wildcard.fedoraproject.org has IPv6 address 2610:28:3090:3001:dead:beef:cafe:fed3
wildcard.fedoraproject.org has IPv6 address 2605:bc80:3010:600:dead:beef:cafe:feda
wildcard.fedoraproject.org has IPv6 address 2604:1580:fe00:0:dead:beef:cafe:fed1
wildcard.fedoraproject.org has IPv6 address 2605:bc80:3010:600:dead:beef:cafe:fed9

It's the same output as I received when connected over VPN. Does this mean that when we're in the Brno office, we're only getting USA proxies, and that's why we consistently see slow responses? Can something be done for this case?

Does this mean that when we're in the Brno office, we're only getting USA proxies, and that's why we consistently see slow responses?

Partially yes. The other part is what I said before - there are too few EU proxies, so you will most often get NA proxy anyway.

Can something be done for this case?

I'm afraid that not on Fedora infra side. You can talk to your corporate IT to consider using a EU DNS resolver for internal 10.x.x.x IP ranges that are known to be in EU.

Donating enough proxies in EU could solve the other part of this issue.

OK, thanks. I think we can close this ticket.

Metadata Update from @mizdebsk:
- Issue close_status updated to: Will Not/Can Not fix
- Issue status updated to: Closed (was: Open)

5 years ago

Login to comment on this ticket.

Metadata