fedora-infrastructure

Clone
Source Code
GIT

#9422 fedoraproject.org dnssec validation failure on Fedora 33
Closed: Fixed 2 years ago by kevin. Opened 3 years ago by pspacek.

Closed: Fixed
Reopen Issue

Describe what you would like us to do:

Upgrade DNSSEC signatures on fedoraproject.org to avoid obsolete algorithms which involve SHA1 hash.

When do you need this to be done by? (YYYY/MM/DD)

Up to you. Clients running Fedora 33 might have trouble accessing fedoraproject.org domain until migration is done.

Background:

Domain fedoraproject.org fails DNSSEC validation on Fedora 33 because it uses SHA1 algorithm, which is now forbidden in Fedora 33 as part of https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2 .

This affects DNS resolvers which follow this system-wide policy, e.g. https://bugzilla.redhat.com/show_bug.cgi?id=1892704 .

Workarounds for users are either changing the local policy, or disabling DNSSEC validation altogether. Both approaches lower end user security so I would recommend moving fedoraproject.org (and possibly other affected domains) to modern algorithms, preferably ECDSAP256SHA256.

Beware that migration to different algorithm is a sensitive step and might break domain completely if done sloppily. I'm happy to help with that process if you want, just drop me an e-mail.

Thanks I am going to roll in a different ticket on this. I will need help and advice to do this correctly.

Metadata Update from @smooge:
- Issue assigned to smooge
3 years ago

Metadata Update from @smooge:
- Issue tagged with: dns, medium-gain, medium-trouble, ops
3 years ago

I am adding a security flag so I can get a review from our security officer @puiterwijk what algorithms and versions are needed to make sure other parts of the infrastructure he depends on keep working.

Metadata Update from @smooge:
- Issue tagged with: security
3 years ago

Metadata Update from @smooge:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
3 years ago

I have added a new set of keys for +014+ to most of the Fedoraproject zones. The longest TTL for fedoraproject.org is 4Weeks (the expire) so I will remove the old keys in 4 weeks from various zones.

Current status. @puiterwijk is looking at what is needed to make this work. My change would have fixed some things but not all.

So, it's been a few months... where are we at here?

fedoraproject.org DNSKEY still includes both. DS references still only RSASHA1 key. Not ready yet.

$ dig +multi -t DNSKEY fedoraproject.org

; <<>> DiG 9.16.11-RedHat-9.16.11-2.fc32 <<>> +multi -t DNSKEY fedoraproject.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34923
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;fedoraproject.org. IN DNSKEY

;; ANSWER SECTION:
fedoraproject.org.  218 IN DNSKEY 257 3 5 (
                AwEAAdTXJc0joiKGfTvLXi+LXxGpKvPvOoJEst9PR8TC
                CvXGVp7h3BY3uXLkjckuT0aopCp2KF8zHgNgpMK03p1f
                d94pn9JZSuxfqvKsiYH2KvNOa/655oPj06jRhqAP5grX
                01Iz4BH411ZhGxIQ1BzZtOr1wAazojMJzLUgChRJs8GV
                t3LU0e6T8z1RQF33Dt9UMHIR5EAsFAqfZ/tsbfJDYktG
                oZi3nFlW7A745+ObM1LNXOWq3FcYPVzhH08Q7/7Wpxmz
                M6/ET8VeqWIsvh8EnZNDNMfJyPbY9B1BOIrFCpE03ALg
                FMejaBZwmeQaX+D4Duup5xGOmdtCO4GSpM1YH6c=
                ) ; KSK; alg = RSASHA1 ; key id = 16207
fedoraproject.org.  218 IN DNSKEY 256 3 14 (
                04ZsDOgyzs3kJsJ4jEY3MYufkCOWm1OI8N4M+dlBOBmw
                eln0TSaKfafHzNCkaPiVG4bdgdnrzwxmjpK5GQgsiB47
                np+I8850Ea3EJG5ORDl3f//lrr92HiYh5DxCNhkG
                ) ; ZSK; alg = ECDSAP384SHA384 ; key id = 60624
fedoraproject.org.  218 IN DNSKEY 257 3 14 (
                7ttmhus8JD56ybsvMVZVsXa3U2R+2+WmOPIP7BU6t2Li
                cosMZ2Ju3pfvijsa5LvBvVCB4xVtLSqEdLSvW4vJPLSA
                B2uyJwHPJMezh0SzGmVCImLU6qDxsxjHqtZ76/Sf
                ) ; KSK; alg = ECDSAP384SHA384 ; key id = 58125
fedoraproject.org.  218 IN DNSKEY 256 3 5 (
                AwEAAcCWNQWl5pCI3iOOP2r8nStL60Zjb/2JQLQytamV
                ap0L44z0YWftu7pu0hx3cnIM1ejQOsEwbg2/10IyC+38
                cYqJDXbSdFg1zGztOS5xNz7r9hzSRK5N2jkycdJ/BoBy
                J4Y+XGpDqfG4I97++8sIzSrw60TmGAKTvM9viL3ByeCN
                ) ; ZSK; alg = RSASHA1 ; key id = 7725

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Út úno 16 17:36:24 CET 2021
;; MSG SIZE  rcvd: 694

$ dig +multi -t DS fedoraproject.org

; <<>> DiG 9.16.11-RedHat-9.16.11-2.fc32 <<>> +multi -t DS fedoraproject.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27260
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;fedoraproject.org. IN DS

;; ANSWER SECTION:
fedoraproject.org.  59179 IN DS 16207 5 2 (
                A7C9BF5AFE374C9650ED678F3D36931A7DE9256B86A7
                BC34D6DEED7D4E492E5E )
fedoraproject.org.  59179 IN DS 16207 5 1 (
                8DD099791A2A110851FDE5D14F6C62ADC3DD7C18 )

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Út úno 16 17:39:28 CET 2021
;; MSG SIZE  rcvd: 130

So, what do we need to do here to move this forward?

I think @puiterwijk said we were still not signing things correctly with the new key?
It would be nice to move this forward.

I think this should be finished before https://pagure.io/fedora-infrastructure/issue/9852 starts. Finishing the rotation will affect rations of UDP:TCP traffic etc. and that might affect planning infrastructure changes.

Metadata Update from @smooge:
- Assignee reset
3 years ago

Dropping myself from this. It needs puiterwijk to look at scripts and fix them

All authoritative servers seem to offer DNSSEC signatures also with a new key, both for zone apex and records. It think DS update can be made now to include both KSK keys in DS record,
enabling a new key to be used for validation too.

for NS in $(dig +short ns fedoraproject.org); do dig +norec +multi +dnssec +nocrypto fedoraproject.org dnskey @$NS; done
(echo "managed-keys {"; dig +short fedoraproject.org dnskey | while \
   read FLAGS DIGEST ALG KEYDATA; 
   do [ "$FLAGS" = 257 -a "$ALG" = 14 ] && echo "fedoraproject.org initial-key $FLAGS $DIGEST $ALG \"$KEYDATA\";"; 
   done; echo "};") > fedoraproject.key

for HOST in fedoraproject.org apps.fedoraproject.org; do 
  for NS in $(dig +short -t ns fedoraproject.org); do
    delv @$NS -a fedoraproject.key +root=fedoraproject.org $HOST; 
  done;
done

@puiterwijk seemed to indicate we missed something and something was not signed by the new key correctly. ;(

But I agree it seems to look ok...

I am interested in looking at https://pagure.io/fedora-infrastructure/issue/9852, and it references this ticket (9422). Is this ticket ongoing, or has it been completed?

Unfortunately it is not yet finished. DS record at org. zone still uses SHA-1 only. No other digest is present. Is anything in fedoraproject.org zone not yet signed also by ECDSAP384SHA384 keys? Is any other step required but update of DS record at org. parent, @puiterwijk ? Can it be moved forward?

Any new progress? It seems almost finished, but is stuck before finish. Can be DS record at org parent updated to point to new KSK key also? alg = ECDSAP384SHA384 ; key id = 58125

Who does have access to request that? Is there reason, why fedorainfracloud.org is already switched, but fedoraproject.org is still on SHA-1?
Edited 2 years ago by pemensik

I can make that request, but I haven't because at one point @puiterwijk said there was a problem and we were not signing things correctly. Unfortunately, I haven't been able to find out what he was seeing...
I'll try and contact him again next week and if it still looks ok to everyone else I'll request the whois updates.

Are those scripts doing signing in public repository? Can we help extending them to multiple keys if not already supported?

Probably a noob question as I'm just now learning what's involved with zone signing, would either of these be options for fedoraproject.org?

  1. Remove the DNSKEY and RRSIG records that are SHA1 and contact our registrary about removing the DS record that would be SHA1?

  2. If #1 can't work, go through the hoops to re-sign fedoraproject.org with the better algorithm, which would include coordinating our registrar of removing all DS records and accepting newly created DS records from the new signing.

These ideas assume that fedoraproject.org can tolerate some time not being signed in the case of idea #2.

First step should be adding a new SHA-2 DS record for the new key, KSK; alg = ECDSAP384SHA384 ; key id = 58125. I would not request any removal until we have that added. I think removal of DS for RSASHA1 key 16207 can be done at the same time, both its SHA1 and SHA2 digests.

I don't see any reason, why should be removing all signatures necessary. It seems ready to me, except DS record.

# cat /tmp/f.o.ta
trust-anchors {
        fedoraproject.org. initial-ds 58125 14 2 "FCC70DB7608C9837F060D6D92DF9E53A22D1F830752B9E7038FC48EA411DFF46";
};

# delv +vtrace -a /tmp/f.o.ta +root=fedoraproject.org. www.fedoraproject.org
Edited 2 years ago by pemensik

ok. I met up with Patrick a few weeks ago now and we went over this. It does indeed seem ready to publish the new DS records.

I've asked out legal contact to update ds keys on fedora.us to make sure I understand the process and they do and the registrar is doing the right thing.

Once thats done and we validate it, we can schedule a time to do the rest.

I do note that we sign a lot more domains that we have setup DS records for. Is there any list of those for updating purposes... ?

fedoraproject.org should now finally have the updated DS record (along will still the old one).

@pemensik can you check and confirm it looks right to you? If so, we can then ask them to remove the old sha1 DS record.

They didn't update fedorapeople.org yet, but thats next. The others (getfedora.org, etc) should also already be done.

Sorry for delay, @kevin. It seems ready.

# cat fedora.conf
trust-anchors {
"fedoraproject.org" static-ds 58125 14 2 "FCC70DB7608C9837F060D6D92DF9E53A22D1F830752B9E7038FC48EA 411DFF46";
"fedora.us"  static-key 257 3 14 "nGXPiPk/4LHb2W8sICDMMX9VXo9cphZMEOzMC6hTrnHmkokqFNM6ZD8X EzbqhIV54D8BCxwL4WpxDiDu4HAZ3bsQ4ETvqyH2Ai2LjU5B5/0QjpUs TCL8C4EzO+CxS2b7";
};

Then tested with:

$ delv -a fedora.conf +vtrace +root=fedoraproject.org apps.fedoraproject.org
;; fetch: apps.fedoraproject.org/A
;; validating apps.fedoraproject.org/CNAME: starting
;; validating apps.fedoraproject.org/CNAME: attempting positive response validation
;; fetch: fedoraproject.org/DNSKEY
;; validating fedoraproject.org/DNSKEY: starting
;; validating fedoraproject.org/DNSKEY: attempting positive response validation
;; validating fedoraproject.org/DNSKEY: verify rdataset (keyid=58125): success
;; validating fedoraproject.org/DNSKEY: marking as secure (DS)
;; validating apps.fedoraproject.org/CNAME: in fetch_callback_dnskey
;; validating apps.fedoraproject.org/CNAME: keyset with trust secure
;; validating apps.fedoraproject.org/CNAME: resuming validate
;; validating apps.fedoraproject.org/CNAME: verify rdataset (keyid=7725): success
;; validating apps.fedoraproject.org/CNAME: marking as secure, noqname proof not needed
;; fetch: wildcard.fedoraproject.org/A
;; validating wildcard.fedoraproject.org/A: starting
;; validating wildcard.fedoraproject.org/A: attempting positive response validation
;; validating wildcard.fedoraproject.org/A: keyset with trust secure
;; validating wildcard.fedoraproject.org/A: verify rdataset (keyid=7725): success
;; validating wildcard.fedoraproject.org/A: marking as secure, noqname proof not needed
; fully validated
apps.fedoraproject.org. 30  IN  CNAME   wildcard.fedoraproject.org.
...

dnsviz.net also seems okay. When I try delv +vtrace fedoraproject.org, it still uses SHA-1, but I think that would change as soon as it is removed.

Yeah, it looked ok to me too. :)

I will wait until after freeze and then ask them to remove the old entries.

Metadata Update from @kevin:
- Issue tagged with: unfreeze
2 years ago

I have asked them to remove the old entries on fedoraproject.org and fedorapeople.org

fedorapeople.org. DS 301 5 1 79EB668E735B6E8662AEED0781D5E31EFF95A4DA
fedorapeople.org. DS 301 5 2 17D5A4A61B88AF2F6F7BA9B6F952354B4C4E1487040168949D64D3F7 5DE59968
fedorapeople.org. DS 25506 10 1 6DEF733BE467FE73387C8E2FDE6F9443EBB6E09F
fedorapeople.org. DS 25506 10 2 97F7D388CC952C57E2DC41685C79B0AE9FDB7AB3A368238DA478DA8C BD4A571A

fedoraproject.org:

fedoraproject.org. DS 16207 5 1 8DD099791A2A110851FDE5D14F6C62ADC3DD7C18
fedoraproject.org. DS 16207 5 2 A7C9BF5AFE374C9650ED678F3D36931A7DE9256B86A7BC34D6DEED7D 4E492E5E

I'm going to close this now, let us know if there's anything more to do here.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
2 years ago

Did the removal actually happen? I can still see old values:

fedoraproject.org.  86400   IN  DS  16207 5 1 8DD099791A2A110851FDE5D14F6C62ADC3DD7C18
fedoraproject.org.  86400   IN  DS  16207 5 2 A7C9BF5AFE374C9650ED678F3D36931A7DE9256B86A7BC34D6DEED7D 4E492E5E
fedoraproject.org.  86400   IN  DS  58125 14 2 FCC70DB7608C9837F060D6D92DF9E53A22D1F830752B9E7038FC48EA 411DFF46
fedoraproject.org.  86400   IN  RRSIG   DS 8 2 86400 20211122152726 20211101142726 63966 org. AAHDeMFvIptdAPtr3nsBlrh58cbnzYD022c9zloZqI7XGptehnbtHlrd N3JuRQtZ36XjpuSOSPdE6EiXVlaVs1BBKJ4hc0cNWobPuYcDkKtFBuzt 8izaBuoMbxwzLr7jFSnjX09361cVd8mzpUjsk5BXNC0TTje5uWJcPuoU Ewo=
;; Received 341 bytes from 199.19.56.1#53(a0.org.afilias-nst.info) in 163 ms

No, it has not yet, I just requested it. Hopefully they will do it here soon...

Login to comment on this ticket.

Metadata
None

dns ops medium-gain medium-trouble security unfreeze

None
None
Waiting on Assignee
Boards 1
ops Status: In Progress
Powered by Pagure 5.13.3
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.