#356 CentOS CI: DNS issue resolving time-c-g.nist.gov
Closed: Invalid 2 years ago by arrfab. Opened 2 years ago by dustymabe.

One of the tests we run involves NTP configuration/validation. We chose to use time-c-g.nist.gov which is currently failing to resolve in CentOS CI for some reason. Can I get someone to check it out and see if they see the same behavior?

sh-4.4$ getent hosts time-c-g.nist.gov
sh-4.4$ echo $?
2
sh-4.4$ getent hosts registry.fedoraproject.org
38.145.60.21    registry.fedoraproject.org
38.145.60.20    registry.fedoraproject.org

Metadata Update from @arrfab:
- Issue assigned to arrfab

2 years ago

Hey @dustymabe ... I quickly checked and I tested against internal resolvers and get same result :

for i in 252 253 12 ; do dig +short time-c-g.nist.gov @172.19.0.${i};done
129.6.15.30
129.6.15.30
129.6.15.30

worth knowing that .252 and .253 are the ones supposed to be the two "official" internal resolvers to use, but some previous nodes were also using .12, which is now just forwarder to .252 and .253 ...
Wondering if there was a transient routing issue between other internal resolvers and external ones but seems to work now.
Can you just confirm ?

Metadata Update from @arrfab:
- Issue tagged with: centos-ci-infra

2 years ago

Hmm. Not working for me. Can you try doing the resolution in our jenkins container?

https://console-openshift-console.apps.ocp.ci.centos.org/k8s/ns/fedora-coreos/pods/jenkins-17-wjwds/terminal

Here's what I see:

sh-4.4$ for i in 252 253 12 ; do dig +short time-c-g.nist.gov @172.19.0.${i};done
;; connection timed out; no servers could be reached
sh-4.4$ cat /etc/resolv.conf 
search fedora-coreos.svc.cluster.local svc.cluster.local cluster.local ci.centos.org
nameserver 172.30.0.10
options ndots:5

hmm, it's using an internal openshift resolver, but I guess/hope that openshift forward to the real internal resolvers in that vlan (which are the ones above and working).
@dkirwan would you mind having a look ?

Metadata Update from @arrfab:
- Assignee reset

2 years ago

Metadata Update from @arrfab:
- Issue assigned to dkirwan

2 years ago

update : in fact just checking that specific time-c-g.nist.gov fqdn, but it seems it vanished from all resolvers,so probably not a openshift/ci resolver issue:

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 time-c-g.nist.gov
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 6249
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;time-c-g.nist.gov.             IN      A

;; Query time: 8 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Jun 14 05:59:24 UTC 2021
;; MSG SIZE  rcvd: 46

Wondering if that's instead a DNSSEC issue for that domain, as they at least advertise it :

dig DS nist.gov +short
52515 7 2 89558E424B73D58C2F0CE91B0380B23016CADEE8F5FA3C9554814C40 6F954F1D
52515 7 1 41FB8043F5695CD3FB4B19CB4FF7083524ADEC59

but it also fails validation

dig @8.8.8.8 time-c-g.nist.gov +dnssec +multi

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 time-c-g.nist.gov +dnssec +multi
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 36199
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;time-c-g.nist.gov. IN A

;; Query time: 8 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Jun 14 06:16:23 UTC 2021
;; MSG SIZE  rcvd: 46

Metadata Update from @arrfab:
- Issue assigned to arrfab (was: dkirwan)

2 years ago

Metadata Update from @arrfab:
- Issue priority set to: Waiting on Reporter (was: Needs Review)
- Issue tagged with: low-gain, low-trouble, need-more-info

2 years ago

yep, tested and report from https://dnssec-analyzer.verisignlabs.com/time-c-g.nist.gov gives (last lines when I tested it) :

Query to bea2.nist.gov for nist.gov/DNSKEY
    Unknown host bea2.nist.gov
    Query to bea.nist.gov for nist.gov/DNSKEY
    Unknown host bea.nist.gov
    Query to gea.nist.gov for nist.gov/DNSKEY
    Unknown host gea.nist.gov
    Query to gea2.nist.gov for nist.gov/DNSKEY
    Unknown host gea2.nist.gov
    Failed to get DNSKEY RR set for zone nist.gov
    Query to bea.nist.gov for time-c-g.nist.gov/A
    Unknown host bea.nist.gov
    Query to gea.nist.gov for time-c-g.nist.gov/A
    Unknown host gea.nist.gov
    Query to gea2.nist.gov for time-c-g.nist.gov/A
    Unknown host gea2.nist.gov
    Query to bea2.nist.gov for time-c-g.nist.gov/A
    Unknown host bea2.nist.gov
    No response from nist.gov nameservers

Got ya. Yeah I figured it was specific to CentOS CI because my local system could resolve fine. I didn't know it was a DNS issue with nist.

As it's not related to CentOS CI infra, let me so close it :)

Metadata Update from @arrfab:
- Issue close_status updated to: Invalid
- Issue status updated to: Closed (was: Open)

2 years ago

👍 - sorry for the noise

Login to comment on this ticket.

Metadata
Boards 1
CentOS CI Infra Status: Backlog