#11588 FUTURE Cryptographic policy blocks rpm-ostree update
Closed: Will Not/Can Not fix 19 days ago by zlopez. Opened 6 months ago by sherghan.

TL&DR

It seems that, in Fedora IoT 38, setting Cryptographic policy to FUTURE breaks ability to update system using rpm-ostree update.

Description

rpm-ostree upgrade returns "No valid mirrors were found in mirrorlist" after setting Cryptographic policy to FUTURE in my Fedora IoT.
It tried to dig in more and the mirror exists and it's fine, it is just that the End-Entity Certificate is not secure enough to be trusted.

Shell dump...

$ sudo rpm-ostree upgrade --check
error: While pulling fedora/stable/aarch64/iot: No valid mirrors were found in mirrorlist 'https://ostree.fedoraproject.org/iot/mirrorlist'

$ curl https://ostree.fedoraproject.org/iot/mirrorlist
https://d2ju0wfl996cmc.cloudfront.net/

$ curl https://d2ju0wfl996cmc.cloudfront.net/fedora/stable/aarch64/iot
curl: (60) SSL certificate problem: EE certificate key too weak
More details here: https://curl.se/docs/sslcerts.html

$ sudo update-crypto-policies --show
FUTURE

$ sudo rpm-ostree status
State: idle
Deployments:
● fedora-iot:fedora/stable/aarch64/iot
                  Version: 38.20230822.0 (2023-08-22T13:07:58Z)
               BaseCommit: 7d32093ca824aaf907c44f5f0677e3ea8b0df656230ff7127a8a56bd51caaeb1
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464
          LayeredPackages: crypto-policies-scripts htop mc moreutils mosh nano podman-compose rsync wireguard-tools zsh

Describe what you would like us to do:

It seems that certificate update could solve this problem.


When do you need this to be done by? (YYYY/MM/DD)

New Fedora release (2023/10/31) is approaching, it would be nice to have it by then. No other pressing deadlines.


This is the amazon cloudfront cert, which we have 0 control over. ;(

We could use a custom domain/cert there, but thats a bunch of hassle for something that really affects almost no users ;)

Perhaps we could setup a way to bypass the cloudfront cdn, but that might be very slow and/or cause us a lot of traffic.

cc: @pbrobinson @pwhalen

Can we engage with AWS via @davidduncan to possibly come up with a solution? They may have a timeline for the replacement of the certificate. It looks like it might only be valid for a year so maybe it might be replaced in ~6 weeks. I'm sure it'll not be long before others see similar issues on the CDN so with luck they may already have a schedule.

Looking at the certificate for https://d2ju0wfl996cmc.cloudfront.net/ in Firefox it seems it's the generic *.cloufront.net cert and it expires on Dec 7th. @sherghan can you confirm that's the one you're referring to and the explicit issues that FUTURE has with the certificate.

Metadata Update from @phsmoura:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: low-gain, low-trouble, ops

6 months ago

One additional consideration making this problem more important is that in the UI administration interface (cockpit-project) system is permanently marked as up-to-date while this problem blocks fetching of the updates.
User / Administrator will not get informed about the updates for this system and thus might never discover the problem exists.
...will create a ticket elsewhere for this. ...Cockpit project ticket: System up-to-date showed when update fails to fetch #19535

I've also checked and changing the Cryptographic policy to DEFAULT temporarily solves the problem, so it can be used as a workaround.
(More details in the shell dump below)


@sherghan can you confirm that's the one you're referring to and the explicit issues that FUTURE has with the certificate.

Yes, I believe we are talking about the same certificate.

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            02:fa:ba:ec:d4:32:d1:b3:94:20:20:7c:59:d7:93:e4
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Amazon, CN = Amazon RSA 2048 M01
        Validity
            Not Before: Dec  8 00:00:00 2022 GMT
            Not After : Dec  7 23:59:59 2023 GMT
        Subject: CN = *.cloudfront.net

After deeper analysis of the certificate chain I think the problem is deeper. The certificates higher in the chain also seem to be too week.
Though I'm in no way expert on certificates. We need to seek expertise of someone smarter than me on this one...
(more details in shell dump below)

depth=0 CN = *.cloudfront.net
verify error:num=66:EE certificate key too weak
verify return:1
depth=1 C = US, O = Amazon, CN = Amazon RSA 2048 M01
verify error:num=67:CA certificate key too weak
verify return:1
depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
verify error:num=67:CA certificate key too weak
verify return:1
depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
verify return:1
depth=1 C = US, O = Amazon, CN = Amazon RSA 2048 M01
verify return:1
depth=0 CN = *.cloudfront.net
verify return:1

Some documentation

More details including specification of certificate requirements can be found here:
- https://fedoraproject.org/wiki/Changes/StrongCryptoSettings3Forewarning1

Shell dump

$ sudo rpm-ostree update --check
error: While pulling fedora/stable/aarch64/iot: No valid mirrors were found in mirrorlist 'https://ostree.fedoraproject.org/iot/mirrorlist'

$ sudo update-crypto-policies --set DEFAULT
Setting system policy to DEFAULT
Note: System-wide crypto policies are applied on application start-up.
It is recommended to restart the system for the change of policies
to fully take place.

$ sudo rpm-ostree update --check
error: While pulling fedora/stable/aarch64/iot: No valid mirrors were found in mirrorlist 'https://ostree.fedoraproject.org/iot/mirrorlist'

# Reboot really seems required for the changes to apply
$ sudo reboot
Connection to link closed by remote host.

# ===== After reboot and reconnect ======
$ sudo rpm-ostree update --check
⠠ Writing objects: 1...
2 metadata, 0 content objects fetched; 21 KiB transferred in 6 seconds; 0 bytes content written
Writing objects: 1... done
Enabled rpm-md repositories: fedora-cisco-openh264 fedora updates
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2023-03-14T10:56:46Z solvables: 4
rpm-md repo 'fedora' (cached); generated: 2023-04-13T20:36:48Z solvables: 59720
rpm-md repo 'updates' (cached); generated: 2023-10-24T01:16:44Z solvables: 22995
Note: --check and --preview may be unreliable.  See https://github.com/coreos/rpm-ostree/issues/1579
AvailableUpdate:
        Version: 38.20231024.0 (2023-10-24T13:19:20Z)
         Commit: d10f8bc7c0f7e5e10267191d4a21b1a616536a5b6ab0d19796eb264413e28952
   GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464
  SecAdvisories: 2 low, 2 moderate, 4 important
           Diff: 97 upgraded, 1 added

# ===== Earlier with crypto-policy still on FUTURE ======
$ openssl x509 -text -noout -in <(openssl s_client -connect d2ju0wfl996cmc.cloudfront.net:443)
depth=0 CN = *.cloudfront.net
verify error:num=66:EE certificate key too weak
verify return:1
depth=1 C = US, O = Amazon, CN = Amazon RSA 2048 M01
verify error:num=67:CA certificate key too weak
verify return:1
depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
verify error:num=67:CA certificate key too weak
verify return:1
depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
verify return:1
depth=1 C = US, O = Amazon, CN = Amazon RSA 2048 M01
verify return:1
depth=0 CN = *.cloudfront.net
verify return:1
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            02:fa:ba:ec:d4:32:d1:b3:94:20:20:7c:59:d7:93:e4
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Amazon, CN = Amazon RSA 2048 M01
        Validity
            Not Before: Dec  8 00:00:00 2022 GMT
            Not After : Dec  7 23:59:59 2023 GMT
        Subject: CN = *.cloudfront.net
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:d7:2c:39:6f:98:45:76:12:8d:2e:33:90:6b:d0:
                    8b:27:5b:e0:a1:f9:23:1d:73:6c:a6:92:a4:c9:8c:
                    f2:48:36:d6:f9:3a:4e:58:f5:75:a9:42:3c:9c:0e:
                    f2:63:10:bf:9a:c6:d2:a9:9d:6e:8d:a8:68:6f:57:
                    bc:e2:c5:35:8b:37:64:93:17:5d:ee:e4:67:17:b6:
                    74:15:b2:d2:bf:2a:21:f3:f1:c4:4e:14:8e:6f:51:
                    1c:e4:eb:a9:98:95:9e:e8:9a:15:80:fe:e0:00:c6:
                    20:22:c8:6e:c9:22:81:d9:99:51:d4:9b:c3:aa:a2:
                    3b:62:54:63:45:e9:ef:bb:13:dd:af:8b:9f:0d:94:
                    57:14:ef:9f:21:e6:b9:ce:48:31:04:d6:83:1d:53:
                    d4:8a:ec:b6:c0:c1:4f:0d:3a:38:b7:82:a3:43:d0:
                    18:12:70:22:71:76:80:cf:4f:c0:24:13:e0:13:63:
                    75:86:70:38:e2:da:3d:a9:b2:66:ad:b2:8d:31:51:
                    a7:7a:37:47:02:1c:7d:37:53:e5:58:43:2f:18:fe:
                    28:1d:da:f5:3f:93:64:31:52:46:81:8d:28:3c:37:
                    c7:22:d0:c8:6f:9a:4a:3b:3a:43:d7:d2:11:e0:3f:
                    69:f1:32:32:7b:90:ff:a7:33:e2:d3:40:5e:13:db:
                    1f:5f
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Authority Key Identifier: 
                81:B8:0E:63:8A:89:12:18:E5:FA:3B:3B:50:95:9F:E6:E5:90:13:85
            X509v3 Subject Key Identifier: 
                A0:3F:34:72:60:BA:01:E1:CF:6B:8D:60:2C:DC:C1:64:5B:14:DB:CD
            X509v3 Subject Alternative Name: 
                DNS:cloudfront.net, DNS:*.cloudfront.net
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 CRL Distribution Points: 
                Full Name:
                  URI:http://crl.r2m01.amazontrust.com/r2m01.crl
            X509v3 Certificate Policies: 
                Policy: 2.23.140.1.2.1
            Authority Information Access: 
                OCSP - URI:http://ocsp.r2m01.amazontrust.com
                CA Issuers - URI:http://crt.r2m01.amazontrust.com/r2m01.cer
            X509v3 Basic Constraints: critical
                CA:FALSE
            CT Precertificate SCTs: 
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : AD:F7:BE:FA:7C:FF:10:C8:8B:9D:3D:9C:1E:3E:18:6A:
                                B4:67:29:5D:CF:B1:0C:24:CA:85:86:34:EB:DC:82:8A
                    Timestamp : Dec  8 20:36:31.529 2022 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:44:02:20:76:9E:A6:3E:6F:D9:B1:48:62:58:C8:5F:
                                F4:45:7A:D0:35:B4:64:C6:45:3B:AE:77:23:0E:65:C3:
                                C4:B4:70:0F:02:20:25:1B:98:7C:19:75:EF:26:40:15:
                                57:16:F0:09:F4:B4:D9:78:CC:83:E7:79:17:44:2C:AB:
                                ED:8E:96:C8:C4:7C
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : B3:73:77:07:E1:84:50:F8:63:86:D6:05:A9:DC:11:09:
                                4A:79:2D:B1:67:0C:0B:87:DC:F0:03:0E:79:36:A5:9A
                    Timestamp : Dec  8 20:36:31.563 2022 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:21:00:FB:0D:90:80:AB:63:3F:08:7C:A3:EB:
                                90:46:07:F6:41:4B:48:92:B7:21:61:8C:73:F0:74:65:
                                D5:C7:8D:CE:47:02:20:3E:B0:94:3D:37:88:C7:70:0D:
                                E2:FE:AE:EC:33:32:9A:8A:56:E6:5C:64:4D:86:92:41:
                                9E:86:8F:B4:7E:5B:2F
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : B7:3E:FB:24:DF:9C:4D:BA:75:F2:39:C5:BA:58:F4:6C:
                                5D:FC:42:CF:7A:9F:35:C4:9E:1D:09:81:25:ED:B4:99
                    Timestamp : Dec  8 20:36:31.483 2022 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:21:00:8B:5C:64:5B:08:5B:AC:50:14:8E:DA:
                                2A:58:AF:F8:81:5A:98:C8:C4:37:A0:86:80:01:40:0E:
                                9A:A4:47:F1:14:02:20:4B:BC:87:4C:DA:08:89:D9:03:
                                3B:1D:12:F5:E1:57:D9:05:76:14:0C:31:16:B6:C5:CE:
                                48:F5:59:EC:FA:57:70
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        3c:d7:99:d9:56:39:3e:73:c8:42:83:5d:c9:63:a3:bd:af:3d:
        17:26:c6:d1:84:2d:4b:58:52:99:bd:d2:9f:a0:13:c0:78:8d:
        0c:67:34:9b:b9:d5:7a:24:08:02:67:d0:ee:10:31:29:b0:a1:
        14:f2:8f:07:19:b4:4f:f9:27:2b:84:1d:6e:d0:37:52:e2:d2:
        a5:15:48:f7:1c:21:53:4f:2a:7f:7b:e7:0e:4f:54:df:8a:8a:
        70:2f:70:3e:d0:29:c5:82:14:f7:05:ce:c5:57:0a:28:1c:23:
        4b:17:e8:09:a6:02:f2:16:7f:80:12:97:6c:02:77:ee:06:c9:
        b0:52:b1:17:51:99:d9:3d:0e:5a:0b:fd:e5:33:d5:5f:86:ea:
        d1:0a:16:89:a8:fc:06:ed:b5:20:e7:af:2c:60:1b:06:ce:85:
        db:90:8c:9d:ca:f6:0d:c7:2c:e6:98:cb:92:e3:fc:be:fd:91:
        1a:e7:82:37:80:35:e9:d0:3c:d0:e3:40:41:02:cf:88:5b:93:
        09:da:71:ca:57:87:97:6e:f2:e1:3d:de:db:e0:c8:61:b7:b8:
        7b:50:a4:73:8e:3f:59:44:06:2c:13:e8:66:21:0e:8f:33:df:
        f1:9a:0b:dc:fa:cb:5e:30:33:75:b3:26:64:da:17:12:39:07:
        fa:0d:fb:bf

One additional consideration making this problem more important is that in the UI administration interface (cockpit-project) system is permanently marked as up-to-date while this problem blocks fetching of the updates.
User / Administrator will not get informed about the updates for this system and thus might never discover the problem exists.

That's a bug in cockpit or what ever interface it uses to check for updates, it should be bubbling those errors up to the user, please don't complicate this with other bugs. Please just report that to cockpit and let them deal with it.

[backlog_refinement]
Closing this as not fix as it's something that we don't want to support yet.

Metadata Update from @zlopez:
- Issue close_status updated to: Will Not/Can Not fix
- Issue status updated to: Closed (was: Open)

19 days ago

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog