Issue #8609: Name server not working properly in some areas - fedora-infrastructure

fedora-infrastructure

#8609 Name server not working properly in some areas

Closed: Fixed 4 years ago by smooge. Opened 4 years ago by zsun.

I'm based in China. Local Fedora users told me that it seems name server of Fedora is not working properly. To be clear, some name server return SERVFAIL.

I confirmed that this happens to me as well by checking pagure.io

$ nslookup
> pagure.io
Server:         192.168.1.1
Address:        192.168.1.1#53

** server can't find pagure.io: SERVFAIL

Note, 192.168.1.1 is the IP of my gateway.

So I have to use my office network to file this ticket.

This probably not only applies to China, because it seems the issue can be detected elsewhere by on-line services like
https://www.whatsmydns.net/#CNAME/mirrors.fedoraproject.org

And following are some info that might be useful from one of the mirror admin in China. Note I don't fully understand the following info provided by the mirror admin, but I can pass information to them if needed.

root@antidns /h/shanker# dig NS fedoraproject.org @a0.org.afilias-nst.info.

; <<>> DiG 9.12.0 <<>> NS fedoraproject.org @a0.org.afilias-nst.info.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49334
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 3, ADDITIONAL: 6
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: b556d3f9d1643418debb5a4b5e390b2fabbbe241cf4bacb0 (good)
;; QUESTION SECTION:
;fedoraproject.org.     IN  NS

;; AUTHORITY SECTION:
fedoraproject.org.  86400   IN  NS  ns02.fedoraproject.org.
fedoraproject.org.  86400   IN  NS  ns05.fedoraproject.org.
fedoraproject.org.  86400   IN  NS  ns04.fedoraproject.org.

;; ADDITIONAL SECTION:
ns02.fedoraproject.org. 86400   IN  AAAA    2610:28:3090:3001:dead:beef:cafe:fed5
ns05.fedoraproject.org. 86400   IN  AAAA    2001:4178:2:1269:dead:beef:cafe:fed5
ns02.fedoraproject.org. 86400   IN  A   152.19.134.139
ns04.fedoraproject.org. 86400   IN  A   209.132.181.17
ns05.fedoraproject.org. 86400   IN  A   85.236.55.10

;; Query time: 182 msec
;; SERVER: 2001:500:e::1#53(2001:500:e::1)
;; WHEN: Tue Feb 04 14:12:00 CST 2020
;; MSG SIZE  rcvd: 235

root@antidns /h/shanker# dig +norecurse SOA fedoraproject.org @85.236.55.10

; <<>> DiG 9.12.0 <<>> +norecurse SOA fedoraproject.org @85.236.55.10
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 47282
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 564b83b8d4b8a2affbd57a535e390b466fa98300038e749d (good)
;; QUESTION SECTION:
;fedoraproject.org.     IN  SOA

;; Query time: 300 msec
;; SERVER: 85.236.55.10#53(85.236.55.10)
;; WHEN: Tue Feb 04 14:12:22 CST 2020
;; MSG SIZE  rcvd: 74

root@antidns /h/shanker# dig +norecurse SOA fedoraproject.org @209.132.181.17

; <<>> DiG 9.12.0 <<>> +norecurse SOA fedoraproject.org @209.132.181.17
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 62840
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 6533ab9904a9d76c4f441e3c5e390b4c3a3a38181fcd0507 (good)
;; QUESTION SECTION:
;fedoraproject.org.     IN  SOA

;; Query time: 158 msec
;; SERVER: 209.132.181.17#53(209.132.181.17)
;; WHEN: Tue Feb 04 14:12:28 CST 2020
;; MSG SIZE  rcvd: 74

root@antidns /h/shanker# dig +norecurse SOA fedoraproject.org @152.19.134.139

; <<>> DiG 9.12.0 <<>> +norecurse SOA fedoraproject.org @152.19.134.139
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 36643
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: da7344230afc25cc1f428c5b5e390b523d86ea6d59cf077f (good)
;; QUESTION SECTION:
;fedoraproject.org.     IN  SOA

;; Query time: 229 msec
;; SERVER: 152.19.134.139#53(152.19.134.139)
;; WHEN: Tue Feb 04 14:12:34 CST 2020
;; MSG SIZE  rcvd: 74

root@antidns /h/shanker# dig +norecurse SOA fedoraproject.org @2001:4178:2:1269:dead:beef:cafe:fed5

; <<>> DiG 9.12.0 <<>> +norecurse SOA fedoraproject.org @2001:4178:2:1269:dead:beef:cafe:fed5
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 502
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: f8ba04c3b8e70ff3441e2eeb5e390b5953d813ebfbb2db85 (good)
;; QUESTION SECTION:
;fedoraproject.org.     IN  SOA

;; Query time: 297 msec
;; SERVER: 2001:4178:2:1269:dead:beef:cafe:fed5#53(2001:4178:2:1269:dead:beef:cafe:fed5)
;; WHEN: Tue Feb 04 14:12:42 CST 2020
;; MSG SIZE  rcvd: 74

root@antidns /h/shanker# dig +norecurse SOA fedoraproject.org @2610:28:3090:3001:dead:beef:cafe:fed5

; <<>> DiG 9.12.0 <<>> +norecurse SOA fedoraproject.org @2610:28:3090:3001:dead:beef:cafe:fed5
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 42922
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 47ecd3d443f088f6cb4040645e390b669051c954aae54e72 (good)
;; QUESTION SECTION:
;fedoraproject.org.     IN  SOA

;; Query time: 219 msec
;; SERVER: 2610:28:3090:3001:dead:beef:cafe:fed5#53(2610:28:3090:3001:dead:beef:cafe:fed5)
;; WHEN: Tue Feb 04 14:12:54 CST 2020
;; MSG SIZE  rcvd: 74

pingou commented 4 years ago

We've just had a report of someone in Australia facing this issue as well, which matches the current output of: https://www.whatsmydns.net/#A/pagure.io

pingou commented 4 years ago

Some more info from @crcinau who is in Australia and also facing the issue:

$ dig ns fedoraproject.org @8.8.8.8
...
;; ANSWER SECTION:
fedoraproject.org.      299     IN      NS      ns04.fedoraproject.org.
fedoraproject.org.      299     IN      NS      ns02.fedoraproject.org.
fedoraproject.org.      299     IN      NS      ns05.fedoraproject.org.

$ host ns04.fedoraproject.org 8.8.8.8
Using domain server:
Name: 8.8.8.8
Address: 8.8.8.8#53
Aliases:  

ns04.fedoraproject.org has address 209.132.181.17

$ host ns02.fedoraproject.org 8.8.8.8
ns02.fedoraproject.org has address 152.19.134.139
ns02.fedoraproject.org has IPv6 address 2610:28:3090:3001:dead:beef:cafe:fed5

$ host ns05.fedoraproject.org 8.8.8.8
ns05.fedoraproject.org has address 85.236.55.10
ns05.fedoraproject.org has IPv6 address 2001:4178:2:1269:dead:beef:cafe:fed5

$ for i in 209.132.181.17 152.19.134.139 85.236.55.10; do host fedoraproject.org $i; done
Using domain server:
Name: 209.132.181.17
Address: 209.132.181.17#53
Aliases:  

Host fedoraproject.org not found: 2(SERVFAIL)
Using domain server:
Name: 152.19.134.139
Address: 152.19.134.139#53
Aliases:  

Host fedoraproject.org not found: 2(SERVFAIL)
Using domain server:
Name: 85.236.55.10
Address: 85.236.55.10#53
Aliases:  

Host fedoraproject.org not found: 2(SERVFAIL)

Edited 4 years ago by pingou

pingou commented 4 years ago

Some ping stats from @crcinau

        for i in 209.132.181.17 152.19.134.139 85.236.55.10; do ping -c2 $i; done
        PING 209.132.181.17 (209.132.181.17) 56(84) bytes of data.
        64 bytes from 209.132.181.17: icmp_seq=1 ttl=43 time=199 ms
        64 bytes from 209.132.181.17: icmp_seq=2 ttl=43 time=196 ms

        --- 209.132.181.17 ping statistics ---
        2 packets transmitted, 2 received, 0% packet loss, time 1001ms
        rtt min/avg/max/mdev = 196.419/197.816/199.213/1.397 ms
        PING 152.19.134.139 (152.19.134.139) 56(84) bytes of data.
        64 bytes from 152.19.134.139: icmp_seq=1 ttl=43 time=252 ms
        64 bytes from 152.19.134.139: icmp_seq=2 ttl=43 time=253 ms

        --- 152.19.134.139 ping statistics ---
        2 packets transmitted, 2 received, 0% packet loss, time 1000ms
        rtt min/avg/max/mdev = 252.277/252.434/252.591/0.157 ms
        PING 85.236.55.10 (85.236.55.10) 56(84) bytes of data.
        64 bytes from 85.236.55.10: icmp_seq=1 ttl=52 time=329 ms
        64 bytes from 85.236.55.10: icmp_seq=2 ttl=52 time=339 ms

        --- 85.236.55.10 ping statistics ---
        2 packets transmitted, 2 received, 0% packet loss, time 1002ms
        rtt min/avg/max/mdev = 328.757/333.803/338.850/5.046 ms

mtasaka commented 4 years ago

BTW while I don't know this is related, but I have not received any mails from fedora mailing list or pagure.io since about 2020/02/04 11:00 JST (JST=UTC+9).

crcinau commented 4 years ago

Good news:

$ for i in 209.132.181.17 152.19.134.139 85.236.55.10; do host fedoraproject.org $i; done
Using domain server:
Name: 209.132.181.17
Address: 209.132.181.17#53
Aliases:

fedoraproject.org has address 13.250.126.156
fedoraproject.org has address 209.132.181.15
fedoraproject.org has address 209.132.181.16
fedoraproject.org has address 8.43.85.67
fedoraproject.org mail is handled by 10 mx1.redhat.com.
fedoraproject.org mail is handled by 20 mx2.redhat.com.
Using domain server:
Name: 152.19.134.139
Address: 152.19.134.139#53
Aliases:

fedoraproject.org has address 209.132.181.16
fedoraproject.org has address 8.43.85.67
fedoraproject.org has address 13.250.126.156
fedoraproject.org has address 209.132.181.15
fedoraproject.org mail is handled by 10 mx1.redhat.com.
fedoraproject.org mail is handled by 20 mx2.redhat.com.
Using domain server:
Name: 85.236.55.10
Address: 85.236.55.10#53
Aliases:

fedoraproject.org has address 209.132.181.15
fedoraproject.org has address 13.250.126.156
fedoraproject.org has address 209.132.181.16
fedoraproject.org has address 8.43.85.67
fedoraproject.org mail is handled by 20 mx2.redhat.com.
fedoraproject.org mail is handled by 10 mx1.redhat.com.

mtasaka commented 4 years ago

Well, now it seems network is also working for me, and now I began to receive mails from fedora mailing list (I live in Japan)

smooge commented 4 years ago

Root Cause Analysis:
Last week a proxy was set up in Asia to help work with load balancing. Changes were made to the ansible named config for a new BIND view for countries in APAC. However the changes were not completed in a seperate set of trees which contain all the DNS files.

When a master ansible run was done yesterday, the dns was updated on the named servers and they began to fail for Asia because no APAC zone existed. The diagnosis took a while but when someone said it worked in NZ but not in AU, I looked for files which contained those strings. The asia zone did not contain NZ which meant hosts from there defaulted to the default zone which worked. All the other countries in the APAC zone failed because they were getting SERVFAIL due to non-existant files.

Adding the zones to the dns repositories files and forcing a build fixed the issue.

smooge commented 4 years ago

OK it looks like it is working for all people I could contact. Closing.

Metadata Update from @smooge:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

4 years ago

Metadata

Assignee

None

Tags

None

Blocking

None

Depending on

None

Priority

Needs Review

fedora-infrastructure

Source Code

#8609 Name server not working properly in some areas Closed: Fixed 4 years ago by smooge. Opened 4 years ago by zsun.

Metadata

#8609 Name server not working properly in some areas

Closed: Fixed 4 years ago by smooge. Opened 4 years ago by zsun.