#7536 [F28] SubCA failing, keys are orphan
Closed: fixed 5 years ago Opened 5 years ago by cheimes.

Issue

On F28 and FreeIPA master, SubCA key and cert replication is broken.

Steps to Reproduce

  1. Install master and replica
  2. ipa ca-add sub1
  3. check output of certutil
  4. ipa caacl-add-ca hosts_services_caIPAserviceCert --cas=sub1 --cas=ipa
  5. ipa-getcert request -w -X "sub1" -k /etc/pki/tls/private/test.key -f /etc/pki/tls/certs/test.pem

Actual behavior

Master
# certutil -d /etc/pki/pki-tomcat/alias/ -f /etc/pki/pki-tomcat/alias/pwdfile.txt -K
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
< 0> rsa      8fbac55a27bfe02a5109390cb9a984cf39e1e42f   NSS Certificate DB:Server-Cert cert-pki-ca
< 1> rsa      85df34dd3c9d3d5e382c8ae553ff100204a19b18   caSigningCert cert-pki-ca
< 2> rsa      1d0501637950643ecf1203d6632d70541e520951   ocspSigningCert cert-pki-ca
< 3> rsa      e8e211d9fafb222190dd87619831527e0ec3d93a   subsystemCert cert-pki-ca
< 4> rsa      39c8c1f66126e46e205d6951098c8d43619e32c0   auditSigningCert cert-pki-ca
< 5> rsa      f7ea2971f8a591ff924c771e415c5504d82b9cf5   transportCert cert-pki-kra
< 6> rsa      7a8d8f5cce9adf4ef87062805c11a9e046d47df1   storageCert cert-pki-kra
< 7> rsa      938a2c590c69f8159f87d2be93b08b223107041f   auditSigningCert cert-pki-kra
< 8> rsa      de25f46f61f81622a191519b97376a486c066a81   caSigningCert cert-pki-ca b852ddc5-d432-458c-b9a6-964af849ad7a
< 9> rsa      1c73be300732ab73a72e377c453c5e0a21e89377   caSigningCert cert-pki-ca 95cdf6e8-b269-49ae-9b56-badde37e1193
# certutil -d /etc/pki/pki-tomcat/alias/ -f /etc/pki/pki-tomcat/alias/pwdfile.txt -L

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

caSigningCert cert-pki-ca                                    CTu,Cu,Cu
ocspSigningCert cert-pki-ca                                  u,u,u
subsystemCert cert-pki-ca                                    u,u,u
auditSigningCert cert-pki-ca                                 u,u,Pu
Server-Cert cert-pki-ca                                      u,u,u
transportCert cert-pki-kra                                   u,u,u
storageCert cert-pki-kra                                     u,u,u
auditSigningCert cert-pki-kra                                u,u,Pu
caSigningCert cert-pki-ca b852ddc5-d432-458c-b9a6-964af849ad7a u,u,u
caSigningCert cert-pki-ca 95cdf6e8-b269-49ae-9b56-badde37e1193 u,u,u
Replica
# certutil -d /etc/pki/pki-tomcat/alias/ -f /etc/pki/pki-tomcat/alias/pwdfile.txt -K
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
< 0> rsa      85df34dd3c9d3d5e382c8ae553ff100204a19b18   caSigningCert cert-pki-ca
< 1> rsa      1d0501637950643ecf1203d6632d70541e520951   ocspSigningCert cert-pki-ca
< 2> rsa      39c8c1f66126e46e205d6951098c8d43619e32c0   auditSigningCert cert-pki-ca
< 3> rsa      e8e211d9fafb222190dd87619831527e0ec3d93a   subsystemCert cert-pki-ca
< 4> rsa      22bfa439d17fad88accf38d1260215ce2eda67f3   NSS Certificate DB:Server-Cert cert-pki-ca
< 5> rsa      938a2c590c69f8159f87d2be93b08b223107041f   auditSigningCert cert-pki-kra
< 6> rsa      7a8d8f5cce9adf4ef87062805c11a9e046d47df1   storageCert cert-pki-kra
< 7> rsa      f7ea2971f8a591ff924c771e415c5504d82b9cf5   transportCert cert-pki-kra
< 8> rsa      8e60e31832e07ca79c2482832de785dc7c21e693   (orphan)
< 9> rsa      df4d56f539df53694ee7fef1cd6d058f476981d7   (orphan)
# certutil -d /etc/pki/pki-tomcat/alias/ -f /etc/pki/pki-tomcat/alias/pwdfile.txt -L

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

caSigningCert cert-pki-ca                                    CT,C,C
ocspSigningCert cert-pki-ca                                  ,,   
auditSigningCert cert-pki-ca                                 ,,P  
subsystemCert cert-pki-ca                                    ,,   
Server-Cert cert-pki-ca                                      u,u,u
auditSigningCert cert-pki-kra                                ,,P  
storageCert cert-pki-kra                                     ,,   
transportCert cert-pki-kra                                   ,,   
sub1 - IPA.EXAMPLE                                             c,c, 
sub2 - IPA.EXAMPLE                                             c,c, 
CA debug log on replica
2018-05-07 14:51:54 [KeyRetrieverRunner-95cdf6e8-b269-49ae-9b56-badde37e1193] FINE: Running ExternalProcessKeyRetriever
2018-05-07 14:51:54 [KeyRetrieverRunner-95cdf6e8-b269-49ae-9b56-badde37e1193] FINE: About to execute command: [/usr/libexec/ipa/ipa-pki-retrieve-key, caSigningCert cert-pki-ca 95cdf6e8-b269-49ae-9b56-badde37e1193, master.ipa.example]
2018-05-07 14:51:57 [KeyRetrieverRunner-95cdf6e8-b269-49ae-9b56-badde37e1193] FINE: Importing key and cert
2018-05-07 14:51:57 [KeyRetrieverRunner-95cdf6e8-b269-49ae-9b56-badde37e1193] FINE: Reinitialising SigningUnit
2018-05-07 14:51:57 [KeyRetrieverRunner-95cdf6e8-b269-49ae-9b56-badde37e1193] FINE: CertificateAuthority:initSigUnit: ca cert found
2018-05-07 14:51:57 [KeyRetrieverRunner-95cdf6e8-b269-49ae-9b56-badde37e1193] FINE: CertificateAuthority: initSigUnit 1- setting mIssuerObj and mSubjectObj
2018-05-07 14:51:57 [KeyRetrieverRunner-95cdf6e8-b269-49ae-9b56-badde37e1193] FINE: CA SigningUnit.init(ca, ca.signing, caSigningCert cert-pki-ca 95cdf6e8-b269-49ae-9b56-badde37e1193)
2018-05-07 14:51:57 [KeyRetrieverRunner-95cdf6e8-b269-49ae-9b56-badde37e1193] FINE: Setting ca.signing.newNickname=caSigningCert cert-pki-ca 95cdf6e8-b269-49ae-9b56-badde37e1193
2018-05-07 14:51:57 [KeyRetrieverRunner-95cdf6e8-b269-49ae-9b56-badde37e1193] FINE: SigningUnit: Logging into token Internal Key Storage Token
2018-05-07 14:51:57 [KeyRetrieverRunner-95cdf6e8-b269-49ae-9b56-badde37e1193] FINE: SigningUnit: Loading certificate caSigningCert cert-pki-ca 95cdf6e8-b269-49ae-9b56-badde37e1193
2018-05-07 14:51:57 [KeyRetrieverRunner-95cdf6e8-b269-49ae-9b56-badde37e1193] FINE: SigningUnit: Unable to find certificate caSigningCert cert-pki-ca 95cdf6e8-b269-49ae-9b56-badde37e1193
2018-05-07 14:51:57 [KeyRetrieverRunner-95cdf6e8-b269-49ae-9b56-badde37e1193] FINE: CA signing key and cert not (yet) present in NSSDB
2018-05-07 14:51:57 [KeyRetrieverRunner-95cdf6e8-b269-49ae-9b56-badde37e1193] FINE: Failed to re-init SigningUnit
2018-05-07 14:51:57 [KeyRetrieverRunner-95cdf6e8-b269-49ae-9b56-badde37e1193] FINE: Retrying in 576 seconds

Expected behavior

Sub CA works

Version/Release/Distribution

freeipa-server-4.6.90.pre1.dev201804291746+git73c3495db-0.fc28.x86_64
freeipa-client-4.6.90.pre1.dev201804291746+git73c3495db-0.fc28.x86_64
package ipa-server is not installed
package ipa-client is not installed
389-ds-base-1.4.0.8-1.fc28.x86_64
pki-ca-10.6.0-1.fc28.noarch
krb5-server-1.16-24.fc28.x86_64

Additional info:


Metadata Update from @cheimes:
- Issue assigned to ftweedal
- Issue priority set to: important
- Issue set to the milestone: FreeIPA 4.7

5 years ago

The ipa-getcert request -w -X "sub1" -k /etc/pki/tls/private/test.key -f /etc/pki/tls/certs/test.pem call on the replica fails with HTTPRequestError on the replica. certmonger (?) then retries the call on the master and succeeds.

Oh goody, this one looks like a hoot! Thanks for the report @cheimes .

I compared the output of certutil with a F27 / 4.6 installation. in FreeIPA, the certs on the replica have the same naming scheme (caSigningCert cert-pki-ca uuid) as on the master. The name
sub1 - IPA.EXAMPLE in the replica's NSSDB point to an issue with Custodia, NSS, and/or Dogtag's PKCS11 importer.

Yeah, I suspect it's related to the nickname-stomping behaviour in
the SQLite NSSDB backend. Still haven't gotten to investigating
this yet but it's close to the front of my queue.

Cheers,
Fraser

Finally onto investigating this issue. What I have discovered so far:

  • nickname issues aside, the certificate import process is not properly associating the certificate with the imported private key. This is even the case when using certutil -A to add the certificate for the orphan key imported via jss.

  • looking at what's actually stored in the SQL key DB, I have noticed one significant difference between an orphan key generated in the NSS versus the key wrapped into the database via jss. The a165 field is 0x01 in the keygen case and 0x00 in the jss unwrap case.

  • SQL columns seeminly correspond to PKCS #11 attribute codes. 0x165 is CKA_ALWAYS_SENSITIVE.

The investigatiln will now turn to working out why this attribute is not set for the jss unwrap, whether it should be set, and if so, implementing that change.

Key imported from PKIArchiveOptions gets imported with incorrect CKA_ID (attribute 0x102 / column a102). This seems critical. Not yet sure if the CKA_ALWAYS_SENSITIVE discrepancy discussed above is critical.

The CKA_ID is computed as a digest of the public key material. So I need to work out where that's going wrong.

I have cloned the ticket to Fedora and marked it as urgent, https://bugzilla.redhat.com/show_bug.cgi?id=1583140

@cheimes the JSS fix (https://github.com/dogtagpki/jss/pull/1) should also resolve the LWCA replication issue and I think we should try and land it independently of NSS fixes. All the more because a timeframe for a fix in NSS is unclear at this stage.

JSS fix merged to master (0a13321d8b59c8edbce98763cb335c4ccaed0603). Once we have
an f28 build we can bump the dep in spec file to close this out.

master:

  • fb16bc9 Require JSS 4.4.4 with fix for sub CA replication

Metadata Update from @cheimes:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

5 years ago

Login to comment on this ticket.

Metadata