It seems that, in Fedora IoT 38, setting Cryptographic policy to FUTURE breaks ability to update system using rpm-ostree update.
rpm-ostree update
rpm-ostree upgrade returns "No valid mirrors were found in mirrorlist" after setting Cryptographic policy to FUTURE in my Fedora IoT. It tried to dig in more and the mirror exists and it's fine, it is just that the End-Entity Certificate is not secure enough to be trusted.
rpm-ostree upgrade
Shell dump...
$ sudo rpm-ostree upgrade --check error: While pulling fedora/stable/aarch64/iot: No valid mirrors were found in mirrorlist 'https://ostree.fedoraproject.org/iot/mirrorlist' $ curl https://ostree.fedoraproject.org/iot/mirrorlist https://d2ju0wfl996cmc.cloudfront.net/ $ curl https://d2ju0wfl996cmc.cloudfront.net/fedora/stable/aarch64/iot curl: (60) SSL certificate problem: EE certificate key too weak More details here: https://curl.se/docs/sslcerts.html $ sudo update-crypto-policies --show FUTURE $ sudo rpm-ostree status State: idle Deployments: ● fedora-iot:fedora/stable/aarch64/iot Version: 38.20230822.0 (2023-08-22T13:07:58Z) BaseCommit: 7d32093ca824aaf907c44f5f0677e3ea8b0df656230ff7127a8a56bd51caaeb1 GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464 LayeredPackages: crypto-policies-scripts htop mc moreutils mosh nano podman-compose rsync wireguard-tools zsh
It seems that certificate update could solve this problem.
New Fedora release (2023/10/31) is approaching, it would be nice to have it by then. No other pressing deadlines.
This is the amazon cloudfront cert, which we have 0 control over. ;(
We could use a custom domain/cert there, but thats a bunch of hassle for something that really affects almost no users ;)
Perhaps we could setup a way to bypass the cloudfront cdn, but that might be very slow and/or cause us a lot of traffic.
cc: @pbrobinson @pwhalen
Can we engage with AWS via @davidduncan to possibly come up with a solution? They may have a timeline for the replacement of the certificate. It looks like it might only be valid for a year so maybe it might be replaced in ~6 weeks. I'm sure it'll not be long before others see similar issues on the CDN so with luck they may already have a schedule.
Looking at the certificate for https://d2ju0wfl996cmc.cloudfront.net/ in Firefox it seems it's the generic *.cloufront.net cert and it expires on Dec 7th. @sherghan can you confirm that's the one you're referring to and the explicit issues that FUTURE has with the certificate.
Metadata Update from @phsmoura: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: low-gain, low-trouble, ops
One additional consideration making this problem more important is that in the UI administration interface (cockpit-project) system is permanently marked as up-to-date while this problem blocks fetching of the updates. User / Administrator will not get informed about the updates for this system and thus might never discover the problem exists. ...will create a ticket elsewhere for this. ...Cockpit project ticket: System up-to-date showed when update fails to fetch #19535
I've also checked and changing the Cryptographic policy to DEFAULT temporarily solves the problem, so it can be used as a workaround. (More details in the shell dump below)
@sherghan can you confirm that's the one you're referring to and the explicit issues that FUTURE has with the certificate.
Yes, I believe we are talking about the same certificate.
Certificate: Data: Version: 3 (0x2) Serial Number: 02:fa:ba:ec:d4:32:d1:b3:94:20:20:7c:59:d7:93:e4 Signature Algorithm: sha256WithRSAEncryption Issuer: C = US, O = Amazon, CN = Amazon RSA 2048 M01 Validity Not Before: Dec 8 00:00:00 2022 GMT Not After : Dec 7 23:59:59 2023 GMT Subject: CN = *.cloudfront.net
After deeper analysis of the certificate chain I think the problem is deeper. The certificates higher in the chain also seem to be too week. Though I'm in no way expert on certificates. We need to seek expertise of someone smarter than me on this one... (more details in shell dump below)
depth=0 CN = *.cloudfront.net verify error:num=66:EE certificate key too weak verify return:1 depth=1 C = US, O = Amazon, CN = Amazon RSA 2048 M01 verify error:num=67:CA certificate key too weak verify return:1 depth=2 C = US, O = Amazon, CN = Amazon Root CA 1 verify error:num=67:CA certificate key too weak verify return:1 depth=2 C = US, O = Amazon, CN = Amazon Root CA 1 verify return:1 depth=1 C = US, O = Amazon, CN = Amazon RSA 2048 M01 verify return:1 depth=0 CN = *.cloudfront.net verify return:1
More details including specification of certificate requirements can be found here: - https://fedoraproject.org/wiki/Changes/StrongCryptoSettings3Forewarning1
$ sudo rpm-ostree update --check error: While pulling fedora/stable/aarch64/iot: No valid mirrors were found in mirrorlist 'https://ostree.fedoraproject.org/iot/mirrorlist' $ sudo update-crypto-policies --set DEFAULT Setting system policy to DEFAULT Note: System-wide crypto policies are applied on application start-up. It is recommended to restart the system for the change of policies to fully take place. $ sudo rpm-ostree update --check error: While pulling fedora/stable/aarch64/iot: No valid mirrors were found in mirrorlist 'https://ostree.fedoraproject.org/iot/mirrorlist' # Reboot really seems required for the changes to apply $ sudo reboot Connection to link closed by remote host. # ===== After reboot and reconnect ====== $ sudo rpm-ostree update --check ⠠ Writing objects: 1... 2 metadata, 0 content objects fetched; 21 KiB transferred in 6 seconds; 0 bytes content written Writing objects: 1... done Enabled rpm-md repositories: fedora-cisco-openh264 fedora updates Importing rpm-md... done rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2023-03-14T10:56:46Z solvables: 4 rpm-md repo 'fedora' (cached); generated: 2023-04-13T20:36:48Z solvables: 59720 rpm-md repo 'updates' (cached); generated: 2023-10-24T01:16:44Z solvables: 22995 Note: --check and --preview may be unreliable. See https://github.com/coreos/rpm-ostree/issues/1579 AvailableUpdate: Version: 38.20231024.0 (2023-10-24T13:19:20Z) Commit: d10f8bc7c0f7e5e10267191d4a21b1a616536a5b6ab0d19796eb264413e28952 GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464 SecAdvisories: 2 low, 2 moderate, 4 important Diff: 97 upgraded, 1 added # ===== Earlier with crypto-policy still on FUTURE ====== $ openssl x509 -text -noout -in <(openssl s_client -connect d2ju0wfl996cmc.cloudfront.net:443) depth=0 CN = *.cloudfront.net verify error:num=66:EE certificate key too weak verify return:1 depth=1 C = US, O = Amazon, CN = Amazon RSA 2048 M01 verify error:num=67:CA certificate key too weak verify return:1 depth=2 C = US, O = Amazon, CN = Amazon Root CA 1 verify error:num=67:CA certificate key too weak verify return:1 depth=2 C = US, O = Amazon, CN = Amazon Root CA 1 verify return:1 depth=1 C = US, O = Amazon, CN = Amazon RSA 2048 M01 verify return:1 depth=0 CN = *.cloudfront.net verify return:1 Certificate: Data: Version: 3 (0x2) Serial Number: 02:fa:ba:ec:d4:32:d1:b3:94:20:20:7c:59:d7:93:e4 Signature Algorithm: sha256WithRSAEncryption Issuer: C = US, O = Amazon, CN = Amazon RSA 2048 M01 Validity Not Before: Dec 8 00:00:00 2022 GMT Not After : Dec 7 23:59:59 2023 GMT Subject: CN = *.cloudfront.net Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:d7:2c:39:6f:98:45:76:12:8d:2e:33:90:6b:d0: 8b:27:5b:e0:a1:f9:23:1d:73:6c:a6:92:a4:c9:8c: f2:48:36:d6:f9:3a:4e:58:f5:75:a9:42:3c:9c:0e: f2:63:10:bf:9a:c6:d2:a9:9d:6e:8d:a8:68:6f:57: bc:e2:c5:35:8b:37:64:93:17:5d:ee:e4:67:17:b6: 74:15:b2:d2:bf:2a:21:f3:f1:c4:4e:14:8e:6f:51: 1c:e4:eb:a9:98:95:9e:e8:9a:15:80:fe:e0:00:c6: 20:22:c8:6e:c9:22:81:d9:99:51:d4:9b:c3:aa:a2: 3b:62:54:63:45:e9:ef:bb:13:dd:af:8b:9f:0d:94: 57:14:ef:9f:21:e6:b9:ce:48:31:04:d6:83:1d:53: d4:8a:ec:b6:c0:c1:4f:0d:3a:38:b7:82:a3:43:d0: 18:12:70:22:71:76:80:cf:4f:c0:24:13:e0:13:63: 75:86:70:38:e2:da:3d:a9:b2:66:ad:b2:8d:31:51: a7:7a:37:47:02:1c:7d:37:53:e5:58:43:2f:18:fe: 28:1d:da:f5:3f:93:64:31:52:46:81:8d:28:3c:37: c7:22:d0:c8:6f:9a:4a:3b:3a:43:d7:d2:11:e0:3f: 69:f1:32:32:7b:90:ff:a7:33:e2:d3:40:5e:13:db: 1f:5f Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Authority Key Identifier: 81:B8:0E:63:8A:89:12:18:E5:FA:3B:3B:50:95:9F:E6:E5:90:13:85 X509v3 Subject Key Identifier: A0:3F:34:72:60:BA:01:E1:CF:6B:8D:60:2C:DC:C1:64:5B:14:DB:CD X509v3 Subject Alternative Name: DNS:cloudfront.net, DNS:*.cloudfront.net X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication X509v3 CRL Distribution Points: Full Name: URI:http://crl.r2m01.amazontrust.com/r2m01.crl X509v3 Certificate Policies: Policy: 2.23.140.1.2.1 Authority Information Access: OCSP - URI:http://ocsp.r2m01.amazontrust.com CA Issuers - URI:http://crt.r2m01.amazontrust.com/r2m01.cer X509v3 Basic Constraints: critical CA:FALSE CT Precertificate SCTs: Signed Certificate Timestamp: Version : v1 (0x0) Log ID : AD:F7:BE:FA:7C:FF:10:C8:8B:9D:3D:9C:1E:3E:18:6A: B4:67:29:5D:CF:B1:0C:24:CA:85:86:34:EB:DC:82:8A Timestamp : Dec 8 20:36:31.529 2022 GMT Extensions: none Signature : ecdsa-with-SHA256 30:44:02:20:76:9E:A6:3E:6F:D9:B1:48:62:58:C8:5F: F4:45:7A:D0:35:B4:64:C6:45:3B:AE:77:23:0E:65:C3: C4:B4:70:0F:02:20:25:1B:98:7C:19:75:EF:26:40:15: 57:16:F0:09:F4:B4:D9:78:CC:83:E7:79:17:44:2C:AB: ED:8E:96:C8:C4:7C Signed Certificate Timestamp: Version : v1 (0x0) Log ID : B3:73:77:07:E1:84:50:F8:63:86:D6:05:A9:DC:11:09: 4A:79:2D:B1:67:0C:0B:87:DC:F0:03:0E:79:36:A5:9A Timestamp : Dec 8 20:36:31.563 2022 GMT Extensions: none Signature : ecdsa-with-SHA256 30:45:02:21:00:FB:0D:90:80:AB:63:3F:08:7C:A3:EB: 90:46:07:F6:41:4B:48:92:B7:21:61:8C:73:F0:74:65: D5:C7:8D:CE:47:02:20:3E:B0:94:3D:37:88:C7:70:0D: E2:FE:AE:EC:33:32:9A:8A:56:E6:5C:64:4D:86:92:41: 9E:86:8F:B4:7E:5B:2F Signed Certificate Timestamp: Version : v1 (0x0) Log ID : B7:3E:FB:24:DF:9C:4D:BA:75:F2:39:C5:BA:58:F4:6C: 5D:FC:42:CF:7A:9F:35:C4:9E:1D:09:81:25:ED:B4:99 Timestamp : Dec 8 20:36:31.483 2022 GMT Extensions: none Signature : ecdsa-with-SHA256 30:45:02:21:00:8B:5C:64:5B:08:5B:AC:50:14:8E:DA: 2A:58:AF:F8:81:5A:98:C8:C4:37:A0:86:80:01:40:0E: 9A:A4:47:F1:14:02:20:4B:BC:87:4C:DA:08:89:D9:03: 3B:1D:12:F5:E1:57:D9:05:76:14:0C:31:16:B6:C5:CE: 48:F5:59:EC:FA:57:70 Signature Algorithm: sha256WithRSAEncryption Signature Value: 3c:d7:99:d9:56:39:3e:73:c8:42:83:5d:c9:63:a3:bd:af:3d: 17:26:c6:d1:84:2d:4b:58:52:99:bd:d2:9f:a0:13:c0:78:8d: 0c:67:34:9b:b9:d5:7a:24:08:02:67:d0:ee:10:31:29:b0:a1: 14:f2:8f:07:19:b4:4f:f9:27:2b:84:1d:6e:d0:37:52:e2:d2: a5:15:48:f7:1c:21:53:4f:2a:7f:7b:e7:0e:4f:54:df:8a:8a: 70:2f:70:3e:d0:29:c5:82:14:f7:05:ce:c5:57:0a:28:1c:23: 4b:17:e8:09:a6:02:f2:16:7f:80:12:97:6c:02:77:ee:06:c9: b0:52:b1:17:51:99:d9:3d:0e:5a:0b:fd:e5:33:d5:5f:86:ea: d1:0a:16:89:a8:fc:06:ed:b5:20:e7:af:2c:60:1b:06:ce:85: db:90:8c:9d:ca:f6:0d:c7:2c:e6:98:cb:92:e3:fc:be:fd:91: 1a:e7:82:37:80:35:e9:d0:3c:d0:e3:40:41:02:cf:88:5b:93: 09:da:71:ca:57:87:97:6e:f2:e1:3d:de:db:e0:c8:61:b7:b8: 7b:50:a4:73:8e:3f:59:44:06:2c:13:e8:66:21:0e:8f:33:df: f1:9a:0b:dc:fa:cb:5e:30:33:75:b3:26:64:da:17:12:39:07: fa:0d:fb:bf
One additional consideration making this problem more important is that in the UI administration interface (cockpit-project) system is permanently marked as up-to-date while this problem blocks fetching of the updates. User / Administrator will not get informed about the updates for this system and thus might never discover the problem exists.
That's a bug in cockpit or what ever interface it uses to check for updates, it should be bubbling those errors up to the user, please don't complicate this with other bugs. Please just report that to cockpit and let them deal with it.
[backlog_refinement] Closing this as not fix as it's something that we don't want to support yet.
Metadata Update from @zlopez: - Issue close_status updated to: Will Not/Can Not fix - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.