#10612 Fedora GEOIP broken ( was Mirrormanager2 IP geolocation returning APAC mirrors instead of GB/WEUR)
Closed: Fixed a year ago by kevin. Opened 2 years ago by bcfse1.

I switched to a new ISP about 5 months ago and have been noticing slow/throttled downloads from Fedora mirrors. Speed is always extremely fast when pointing to local mirrors (e.g. mirror.ox.ac.uk, UK mirror service, anorien.warwick.ac.uk etc).
It finally got annoying enough for me to check the actual repository source and it appears I am being redirected to APAC mirrors.

metalink repo mirror list...

$ curl -s 'https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64'
<?xml version="1.0" encoding="utf-8"?>
<metalink version="3.0" xmlns="http://www.metalinker.org/" type="dynamic" pubdate="Mon, 28 Mar 2022 13:28:27 GMT" generator="mirrormanager" xmlns:mm0="http://fedorahosted.org/mirrormanager">
 <files>
  <file name="repomd.xml">
   <mm0:timestamp>1635226287</mm0:timestamp>
   <size>6285</size>
   <verification>
    <hash type="md5">8dcd06f0e4ca8f94b603d52022eee65f</hash>
    <hash type="sha1">3ca888dfb52dfb111ae8104e44d1f7bb0afe2c63</hash>
    <hash type="sha256">2b8e8f4786f4c8a6746f532735dc96ea8ef8c36cb4cde77f2aedb2af6905b136</hash>
    <hash type="sha512">bd3fcad0b5bd3e693b0eba28305e0652c687d76a2a3ca29501400d69b9b9888c46087e405a6f1d7f08d2beaf7865aee41eac18869bc1c905d12505a4800ec8d1</hash>
   </verification>
   <resources maxconnections="1">
    <url protocol="http" type="http" location="IN" preference="100">http://fedora.mirror.net.in/fedora/linux/releases/35/Everything/x86_64/os/repodata/repomd.xml</url>
    <url protocol="http" type="http" location="JP" preference="99">http://ftp.jaist.ac.jp/pub/Linux/Fedora/releases/35/Everything/x86_64/os/repodata/repomd.xml</url>
    <url protocol="rsync" type="rsync" location="JP" preference="99">rsync://ftp.jaist.ac.jp/pub/Linux/Fedora/releases/35/Everything/x86_64/os/repodata/repomd.xml</url>
    <url protocol="https" type="https" location="CN" preference="98">https://mirrors.tuna.tsinghua.edu.cn/fedora/releases/35/Everything/x86_64/os/repodata/repomd.xml</url>
    <url protocol="http" type="http" location="CN" preference="98">http://mirrors.tuna.tsinghua.edu.cn/fedora/releases/35/Everything/x86_64/os/repodata/repomd.xml</url>
    <url protocol="rsync" type="rsync" location="CN" preference="98">rsync://mirrors.tuna.tsinghua.edu.cn/fedora/releases/35/Everything/x86_64/os/repodata/repomd.xml</url>
    <url protocol="rsync" type="rsync" location="ID" preference="97">rsync://fedora.mirror.angkasa.id/fedora-enchilada/linux/releases/35/Everything/x86_64/os/repodata/repomd.xml</url>
...

geolocation data returning correct in maxmind, ip2location

[bgp.tools]
AS      | IP               | BGP Prefix          | CC | Registry | Allocated  | AS Name
208189  | 31.22.13.104      | 31.22.12.0/22       | GB | RIPE     | 2019-02-25 | Swish Fibre Ltd

Prefix has been announced by AS208189 since 2020... https://stat.ripe.net/widget/routing-history#w.resource=31.22.12.0%2F22&w.starttime=2018-01-08T00%3A00%3A00&w.endtime=2022-03-28T00%3A00%3A00

Simulated the data that appears to be used by mirrormanager2 which contains the correct prefix-AS mapping...
https://github.com/fedora-infra/mirrormanager2/blob/master/utility/mm2_get_global_netblocks

$ wget http://ftp.routeviews.org/dnszones/rib.bz2
$ bzcat rib.bz2 | perl zebra-dump-parser/zebra-dump-parser.pl | uniq >> list.out
$ grep '^31\.22\.12\.' list.out
31.22.12.0/22 208189

Please could you confirm the source of geolocation data used for mirror detection and confirm that mirrormanager2 is using up-to-date information?
Happy to submit a data correction request to the relevant data source if required.


Metadata Update from @mohanboddu:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: low-gain, low-trouble, ops

2 years ago

I have the same issue of wrong IP location. I am in France. All IP location databases report my IP 212.114.28.25 as FR (see https://www.iplocation.net/ip-lookup).

But my IP is detected by fedora mirror manager in Russia and I am redirected to the server http://mirror.linux-ia64.org/.

Hmmm the GeoIP tool is giving the following:

$ geoiplookup 212.114.28.25
GeoIP Country Edition: RU, Russian Federation

Interesting. My IP isn't found from the geoiplookup tool...

$ geoiplookup 31.22.13.104
GeoIP Country Edition: IP Address not found

$ geoiplookup -v 31.22.13.104
GeoIP Country Edition: GEO-106FREE 20180327 Build 1 Copyright (c) 2018 MaxMind Inc All Rights Reserved

Does that mean it's using a 4 year old database?

This tool is definitively giving wrong answers :(. Its database is totally outdated even in Fedora 35. It depends of GeoIP-GeoLite-data-2018.06-8.fc35.noarch package. It explains why my IP location (modified in 2020 or 2021) and bcfse1 one (2019) were wrong.

See the location of the up-to-date RIPE record: https://apps.db.ripe.net/db-web-ui/query?bflag=false&dflag=false&rflag=true&searchtext=212.114.28.25&source=RIPE

It could be that we are using an out of date database. Not sure it was ever updated since we switched to geoip2.

It seems an account is required to get updated databases. Not sure if there if GeoLite2 is free of charge or not.

Does Fedora already have an account to use GeoLite2 databases? For MirrorManager the country edition would be good enough. There is also the anaconda geolocation service which uses a GeoLite2 database.

GeoLite2 free databases were removed from usage due to GDPR and some lawsuits. I don't think we currently have an account with GeoLite2.

OK I can download the 'Lite' versions with my old account, but I do not have time to automate this. This is going to need to be copied to a lot of different places to make this work.

Metadata Update from @smooge:
- Issue untagged with: low-gain, low-trouble
- Issue tagged with: high-trouble, medium-gain

2 years ago

I am changing the priority of this for the following reasons:
1. Multiple applications are broken:
* GNOME first boot placement for users
* Mirrorlist
* Fedora DNS zones
2. It is not going to be an easy fix.
* A role needs to be written to autodownload the updates (either paid or unpaid)
* The role needs to copy this to a place on various servers
* The above parts all look in different places so may need code changes.

OK I have slapped a fix on for the GNOME and Mirrorlist tooling which should help clean up these errors for people. This was done by manually downloading new databases and putting them on sundries (for gnome geoip) and mm-backend01 (for mirrorlist).

We also need it on all proxies.

ah missed that. It is currently provided by the package geolite2-city-20191217-5.fc35.noarch and geolite2-city-20191217-5.fc35.noarch . Either we need to make an rpm which overwrites this or a different way to distribute the files.

Turning it over to someone else to do that patching.

The mirrorlist-server process has a parameter to use the GeoIP2 database from another location. So if you copy the file to another directory I can provide the change to adapt the mirrorlist-server service file.

Ok so there seems to be some sort of problem with the proxies. Some proxies have geolite2-country-20191217-5.fc35.noarch and geolite2-city-20191217-5.fc35.noarch installed but only due to older playbook issues. I have made https://pagure.io/fedora-infra/ansible/pull-request/1068 to try and 'fix' this issue for the newer proxies and such.

OK I have pushed this fix and run it on the proxies. I hope this will clear up bad location data in the next 2 to 24 hours.

Amazing, thank you all. I can confirm mirrormanager is now returning mirrors in WEUR... much better than downloading from CN!

Metadata Update from @smooge:
- Issue assigned to smooge

2 years ago

So, we have the temp fix for now here... do we want to keep this open to track regularly updating this/putting place automation to do so?

Or open a new one for that?

If it is fixed for now, I would say close it. A larger ticket needs to be made where a login/password and account set up are needed for getting this done regularly. I think that should be a different ticket.

Metadata Update from @smooge:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

Noticed this weekend that this has regressed, or load balancers in a pool have not been updated or a stale cache...

$ curl -s 'https://mirrors.fedoraproject.org/metalink?repo=fedora-36&arch=x86_64' | grep -o 'location="[A-Z]\{1,2\}"' | sort | uniq
location="CN"
location="ID"
location="JP"
location="SG"
location="TH"
$ curl -s 'https://mirrors.fedoraproject.org/metalink?repo=fedora-36&arch=x86_64' | grep -o 'location="[A-Z]\{1,2\}"' | sort | uniq
location="CN"
location="ID"
location="JP"
location="SG"
location="TH"
$ curl -s 'https://mirrors.fedoraproject.org/metalink?repo=fedora-36&arch=x86_64' | grep -o 'location="[A-Z]\{1,2\}"' | sort | uniq
location="AT"
location="BE"
location="BG"
location="BY"
location="CH"
location="CY"
location="CZ"
location="DE"
location="DK"
location="FI"
location="FR"
location="GB"
location="GR"
location="HU"
location="IS"
location="LT"
location="LU"
location="MD"
location="NL"
location="PT"
location="RS"
location="RU"
location="SE"
location="SK"
location="TR"
location="UA"
$ curl -s 'https://mirrors.fedoraproject.org/metalink?repo=fedora-36&arch=x86_64' | grep -o 'location="[A-Z]\{1,2\}"' | sort | uniq
location="AT"
location="BE"
location="BG"
location="BY"
location="CH"
location="CY"
location="CZ"
location="DE"
location="DK"
location="FI"
location="FR"
location="GB"
location="GR"
location="HU"
location="IS"
location="LT"
location="LU"
location="MD"
location="NL"
location="PT"
location="RS"
location="RU"
location="SE"
location="SK"
location="TR"
location="UA"
$ curl -s 'https://mirrors.fedoraproject.org/metalink?repo=fedora-36&arch=x86_64' | grep -o 'location="[A-Z]\{1,2\}"' | sort | uniq
location="CN"
location="ID"
location="JP"
location="SG"
location="TH"

The AS and geo-mapping of the IP in question hasn't changed in the last 3 months...

AS      | IP               | BGP Prefix          | CC | Registry | Allocated  | AS Name
208189  | 31.22.13.104     | 31.22.12.0/22       | GB | RIPE     | 2019-02-25 | Swish Fibre Ltd

Metadata Update from @bcfse1:
- Issue status updated to: Open (was: Closed)

2 years ago

Metadata Update from @smooge:
- Assignee reset

a year ago

I missed that this was re-opened.

Are you still seeing the issue? We have not changed these files in a while...

Please file a new issue if you are still seeing this?

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

a year ago

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog