#93 Certmonger cannot obtain a SCEP certificate from a Dogtag sub-CA
Closed: fixed 6 years ago Opened 6 years ago by tvaughan.

I have a Dogtag setup that has a root CA with a subordinate CA.

Setup

  • The root CA does not have the KRA service enabled.
  • The sub CA does have the KRA service enabled with the following configuration items set:
    • auths.instance.flatFileAuth.deferOnFailure=false
    • ca.scep.allowedEncryptionAlgorithms=DES,DES3
    • ca.scep.allowedHashAlgorithms=MD5,SHA1,SHA256,SHA384,SHA512
    • ca.scep.enable=true
    • ca.scep.encryptionAlgorithm=DES
    • ca.scep.hashAlgorithm=MD5

Note: The algorithms are set as such due to #89

Test

Certmonger is tested using the following steps:

  • Update the CA flatfile.txt to include an appropriate UID/PWD pair
  • getcert add-scep-ca -c Test -u https://$HOSTNAME:8443/ca/cgi-bin/pkiclient.exe -R /etc/pki/site-pki.pem
  • getcert request -c Test -k /etc/pki/my_cert.pem -f /etc/pki/my_cert.pub -I Host_Cert -R -w -L password

Debugging

I've run certmonger from the command line to attempt to obtain debug information using /usr/libexec/certmonger/certmonger-session -n -F -d 100 -S 2>&1.

The output from the test is as follows:

2018-02-02 14:55:45 [10281] Redirecting stdin and stderr to /dev/null, leaving stdout open for child "/usr/libexec/certmonger/scep-submit".                                                                        
2018-02-02 14:55:45 [10105] Request1('Host_Cert') moved to state 'SUBMITTING'                            
2018-02-02 14:55:45 [10105] Will revisit Request1('Host_Cert') on traffic from 22.                       
2018-02-02 14:55:46 [10105] Certificate submission still ongoing.                                        
2018-02-02 14:55:46 [10105] Will revisit Request1('Host_Cert') on traffic from 22.                       
2018-02-02 14:55:46 [10105] Certificate submission attempt complete.                                     
2018-02-02 14:55:46 [10105] Child status = 3.       
2018-02-02 14:55:46 [10105] Child output:           
"Error: failed to verify signature on server response.                                                    
"                                                   
2018-02-02 14:55:46 [10105] Certificate not (yet?) issued.                                               
2018-02-02 14:55:46 [10105] Request1('Host_Cert') moved to state 'CA_UNREACHABLE'                        
2018-02-02 14:55:46 [10105] Will revisit Request1('Host_Cert') in 604800 seconds.

I subsequently added printf statements to the code to determine exactly where the failure was happening and it resulted in isolation at https://pagure.io/certmonger/blob/master/f/src/pkcs7.c#_1065 (I don't know why the debug message there didn't print).

I can patch the certmonger code to try different things but have been unable to determine how to successfully print the server response.

Additional Information

I have determined that the Dogtag installation is correct by using the sscep command to obtain certificates from the same CA.

The sscep.conf file is as follows:

CACertFile=ca.crt
CertReqFile=local.csr
EncAlgorithm=des
LocalCertFile=local.crt
PrivateKeyFile=local.key
SigAlgorithm=md5
URL=http://${HOSTNAME}:8080/ca/cgi-bin/pkiclient.exe

Using that config file:

sscep getca -f sscep.conf
mkrequest -dns $HOSTNAME password
sscep enroll -f sscep.conf

This results in the following output (and a valid, signed certificate):

sscep: Found private key local.key as file. If the engine can handle it, loading the file
sscep: sending certificate request
sscep: valid response from server
sscep: reply transaction id: AA0FE652952D0FA691CF6CAAF656A520
sscep: pkistatus: SUCCESS

Thank you for the terrific reproduction steps. I've made a bit of progress in that I was able to get IPA to issue SCEP request. I duplicated the issue without having to create the subCA.

What I think is happening is certs is empty when calling PKCS7_verify(). The CA cert is only in the store variable which isn't searched for the signing cert (only for the chain to verify said cert in my reading of the man page).

I hacked pkcs7.c to insert a copy of roots into the certs variable and was able to successfully issue a cert over SCEP from certmonger. I'm not yet convinced this is the right solution. I need to trace the code further up to see what the difference is between certs and othercerts to certmonger. Only othercerts would be added to the certs variable in the call to PKCS7_verify() and in my case othercerts is empty. I suspect something is reversed further up the chain.

So basically the reason it couldn't verify the signer is it had no certs to verify with.

@rcritten Thanks for the update.

I noticed that, in the CA configuration saved by certmonger, there is a ca_encryption_issuer_cert and a ca_encryption_cert_pool variable pair that appear to have the certificates from my CAs in them.

I'm not sure if this would make a difference in the root vs. sub-CA setup and your debugging.

I assume you are using the subCA with the -R command? Can you create another certmonger CA ( add-scep-ca) but use the main IPA CA instead?

@rcritten Will that work without an IPA server on the other end? I'm running pure Dogtag.

Sure I should have just said try the root CA to see if that helps.

I'd have to re-roll the setup. I don't add a KRA to my root CA since it's not supposed to be used for automatic key enrollment. Might not be able to get to it for a few days.

I mean just the root signing cert. You shouldn't need any additional services.

Tried the following with no difference: getcert add-scep-ca -c Test -u https://$HOSTNAME:8443/ca/cgi-bin/pkiclient.exe -R /etc/pki/site-pki.pem -r /etc/pki/site-pki.pem

  • /etc/pki/site-pki.pem is the root CA certificate

I think I've tried all combinations of -r, -R, and -I with no luck. This includes using the entire certificate chain for -I and different combinations of the root and sub-CA certificates.

Ok. In trying to get more information for you I discovered that what I thought were individual successful SCEP requests were in fact just retrieving the original cert I was issued (even if I revoked it). Continuing to look.

A bit of debugging advice.

Ensure you SCEP request is cleared out of certmonger. Remove the requesting key and cert (though there probably isn't one) and stop the existing tracking.

stop certmonger

strace -fF -s 4096 -o /tmp/out certmonger -d 9 -n

Submit your cert tracking request

In /tmp/out you'll be able to see the stdout messages that scep-submit is printing. This includes the PKCS#7 blog that is returned. Look for response_code =

If it is 200 (and it probably is) then you'll see other status messages and below that a results = blog which is the PKCS#7 message (just without the header/footer).

scep-submit writes its log messages to stderr.

So, this is what I'm seeing:

  1. No response_code entries were found
  2. I did see quite a few instances of the server returning 500 but the server seems to think that it did everything that it was supposed to from the CA debug log

I haven't really duplicated your environment. I'm testing against a self-signed CA in an IPA environment.

Passing the IPA CA cert using -I is working for me.

Since you have already looked at instrumenting the code, this should do it to get the raw PKCS#7 signed data out:

diff --git a/src/pkcs7.c b/src/pkcs7.c
index 991ef91d..864d37f9 100644
--- a/src/pkcs7.c
+++ b/src/pkcs7.c
@@ -911,6 +911,7 @@ cm_pkcs7_verify_signed(unsigned char *data, size_t l
ength,
        char *s, buf[LINE_MAX], *p, *q;
        int ret = -1, i;
        long error;
+       FILE *fp;

        if (digest != NULL) {
                *digest = NULL;
@@ -946,6 +947,9 @@ cm_pkcs7_verify_signed(unsigned char *data, size_t l
ength,
                *payload_length = 0;
        }
        u = data;
+       fp = creat("/tmp/pkcs7.der", 0644);
+       write(fp, data, length);
+       close(fp);
        p7 = d2i_PKCS7(NULL, &u, length);
        if ((p7 == NULL) || (u != data + length)) {
                cm_log(1, "Error parsing what should be PKCS#7 signed-data.\n");

You can see who the signer is with:

# openssl asn1parse -inform der -in /tmp/pkcs7.der

@rcritten I can give you steps to spin up a full reproducer environment if you have somewhere you can run RVM and Vagrant. Let me know if you want me to post the steps.

Part of the key here is that I am not using IPA, just straight up Dogtag.

@rcritten Applied the patch that you supplied above and the code was never executed (the file was not created).

@rcritten I moved the block into https://pagure.io/certmonger/blob/master/f/src/pkcs7.c#_1065 and dumped the offending PKCS7 data.

Reading this in via the openssl command that you provided does not show any issues whatsoever.

It is a decodable PKCS#7 file but it should show you the signer of the blob which I hope will bring a eureka moment.

@rcritten The signer was just fine and as I expected.

What was the signer? The root CA or the subordinate CA? Can you show what the current dogtag_submit CA looks like?

If you could provide reproduction steps that would help a lot. I should mention that while I'm using an IPA-provisioned CA using SCEP talks directly to the CA and avoids all the IPA stuff. The real difference is the intermediate I think.

The signer was the root but it doesn't seem to matter.

The passwords are auto-generated garbage so don't let that worry you.

Root CA Config

[DEFAULT]
pki_instance_name=simp-pki-root

pki_security_domain_name=SIMP


pki_ds_bind_dn=cn=SIMP Directory Manager
pki_ds_database=simp-pki-root-ca
pki_ds_password=nEQ4pMEXY3TiQyRFzB46gYrUSTW148gOeXquLd4z20tXDBS7VSDIGcvEEXOqne3j

[CA]
pki_ds_hostname=127.0.0.1

pki_client_database_purge=False
pki_client_database_password=mM8mpY0XLuf7VPPIkN1iV8nVULpoWQCYSOkY3enO08K5ciC2FcHEP3VNAokNzaGI
pki_client_pkcs12_password=mM8mpY0XLuf7VPPIkN1iV8nVULpoWQCYSOkY3enO08K5ciC2FcHEP3VNAokNzaGI


pki_http_port=4508
pki_https_port=4509

pki_admin_email=caadmin@ca.int.localdomain
pki_admin_name=caadmin
pki_admin_nickname=caadmin
pki_admin_password=mM8mpY0XLuf7VPPIkN1iV8nVULpoWQCYSOkY3enO08K5ciC2FcHEP3VNAokNzaGI
pki_admin_uid=caadmin

Subordinate CA Config

[DEFAULT]
pki_instance_name=simp-site-pki

pki_security_domain_name=SIMP

pki_security_domain_hostname=ca.int.localdomain
pki_security_domain_https_port=4509
pki_security_domain_user=caadmin
pki_security_domain_password=mM8mpY0XLuf7VPPIkN1iV8nVULpoWQCYSOkY3enO08K5ciC2FcHEP3VNAokNzaGI

pki_ds_bind_dn=cn=SIMP Directory Manager
pki_ds_database=simp-site-pki-ca
pki_ds_password=nEQ4pMEXY3TiQyRFzB46gYrUSTW148gOeXquLd4z20tXDBS7VSDIGcvEEXOqne3j

[CA]
pki_ds_hostname=127.0.0.1

pki_client_database_purge=False
pki_client_database_password=lroWsjRudFF2NAO5PIpyWizT6xjaFlbsJISqUnhdtKKdSXpVazRYdinp30yIL8IW
pki_client_pkcs12_password=lroWsjRudFF2NAO5PIpyWizT6xjaFlbsJISqUnhdtKKdSXpVazRYdinp30yIL8IW

pki_subordinate=True
pki_issuing_ca_hostname=ca.int.localdomain
pki_issuing_ca_https_port=4509

pki_http_port=8080
pki_https_port=8443

pki_admin_email=caadmin@ca.int.localdomain
pki_admin_name=caadmin
pki_admin_nickname=caadmin
pki_admin_password=lroWsjRudFF2NAO5PIpyWizT6xjaFlbsJISqUnhdtKKdSXpVazRYdinp30yIL8IW
pki_admin_uid=caadmin

[Tomcat]
pki_ajp_port=8440
pki_tomcat_server_port=8441

Reproduction steps:

  1. Install VirtualBox and Vagrant (from the Vagrant website, not YUM)
  2. Install RVM per the RVM website
  3. rvm install 2.4
  4. rvm use 2.4 --default
  5. git clone https://github.com/simp/simp_pki_service
  6. cd simp_pki_service
  7. bundle update
  8. BEAKER_destroy=no rake beaker:suites
  9. wait until that's done.....
  10. cd .vagrant/beaker_vagrant_files/default.yml
  11. vagrant ssh ca
  12. sudo su -

You can then do what you need to do.

Run vagrant destroy -f in the directory with the Vagrantfile to destroy the system.

Thanks for the steps. I'll try to reproduce.

I can't get it to install. I get a slew of ruby/gem errors and it fails to even try starting the vagrant images:

=== Suite 'default' Starting ===

/usr/local/rvm/rubies/ruby-2.4.1/bin/ruby -I/usr/local/rvm/gems/ruby-2.4.1/gems/rspec-core-3.7.1/lib:/usr/local/rvm/gems/ruby-2.4.1/gems/rspec-support-3.7.1/lib /usr/local/rvm/gems/ruby-2.4.1/gems/rspec-core-3.7.1/exe/rspec spec/acceptance/suites/default --color
/usr/local/rvm/gems/ruby-2.4.1/gems/beaker-rspec-6.2.3/lib/beaker-rspec/helpers/serverspec.rb:43: warning: already initialized constant Module::VALID_OPTIONS_KEYS
/usr/local/rvm/gems/ruby-2.4.1/gems/specinfra-2.67.3/lib/specinfra/configuration.rb:4: warning: previous definition of VALID_OPTIONS_KEYS was here
Hypervisor for ca is vagrant
Beaker::Hypervisor, found some vagrant boxes to create
created Vagrantfile for VagrantHost ca

An error occurred while loading ./spec/acceptance/suites/default/00_default_spec.rb.
Failure/Error: require 'beaker-rspec'
RuntimeError:
Failed to exec 'vagrant up'. Error was Ignoring ffi-1.9.18 because its extensions are not built. Try: gem pristine ffi --version 1.9.18
Ignoring json-2.1.0 because its extensions are not built. Try: gem pristine json --version 2.1.0
Ignoring unf_ext-0.0.7.4 because its extensions are not built. Try: gem pristine unf_ext --version 0.0.7.4
/usr/local/rvm/rubies/ruby-2.4.1/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in require': cannot load such file -- bundler/setup (LoadError) from /usr/local/rvm/rubies/ruby-2.4.1/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:inrequire'

and a ton more...

Note that the first go-around I did a gem install of each of these packages and no errors were reported and apparently they were installed.

@rcritten My guess is that you are using the EPEL vagrant, not the one from vagrantup.com. Try removing all Vagrants and then installing from vagrantup.com.

Also, try running as BEAKER_destroy=no bundle exec rake beaker:suites

Ok, vagrant-2.0.2 is required. I was doing this on F-27 with 2.0.1. I have the image up.

One more question. Your initial report refers to /etc/pki/site-pki.pem. I don't have that on my system. I can certainly fetch the CA certificates but I'd like to reproduce as closely as possible to what you are seeing. Which certificate is here: the root, intermediate or both?

I realize you said you'd tried all combinations but again, just trying to start from the point you did and got from there.

@rcritten The VM is a base test harness that you can use the script that I posted before to test.

You'll have to extract the server PKI keys following the steps outlined here: https://github.com/simp/pupmod-simp-simp_pki_service#client-setup

Then you can use it the usual way.

I think I've got the setup done. I'm at least reproducing a similar looking failure.

Can you look at your pem files and see if there are headers like:

Bag Attributes
2.16.840.1.113730.5.1.1.1: CT,C,C
localKeyID: 00 9B 92 61 B3 05 7F EE 42 9B 6A AF DE 5B 08 FC D2 E7 35 09 15
friendlyName: CA Signing Certificate - SIMP
subject=/O=SIMP/OU=simp-pki-root/CN=CA Signing Certificate
issuer=/O=SIMP/OU=simp-pki-root/CN=CA Signing Certificate

Please try stripping that out. This worked for me:

# getcert add-scep-ca -c Test -u https://hostname:8443/ca/cgi-bin/pkiclient.exe -I /etc/pki/site-sub.pem -v -R /etc/pki/site-pki.pem
# getcert request -c Test -k /etc/pki/tls/private/test.key -f /etc/pki/tls/certs/test.pem -w -L 1234

Number of certificates and requests being tracked: 1.
Request ID '20180301180935':
        status: MONITORING
        stuck: no
        key pair storage: type=FILE,location='/etc/pki/tls/private/test.key'
        certificate: type=FILE,location='/etc/pki/tls/certs/test.pem'
        signing request thumbprint (MD5): BC7AE9AD 6359D78A 31A6BB24 99D4727A
        signing request thumbprint (SHA1): 14A4E587 666E1626 52C570E8 C6E8C6D4 6287B675
        CA: Test
        issuer: CN=CA Signing Certificate,OU=simp-site-pki,O=SIMP
        subject: CN=ca.example.com
        expires: 2020-02-13 21:19:42 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment
        eku: id-kp-clientAuth,id-kp-emailProtection
        pre-save command: 
        post-save command: 
        track: yes
        auto-renew: yes

Yeah, I'll give it a shot

@rcritten That did indeed work!

I'm surprised that the sscep code worked and this didn't since they appear to both call the same openssl routines.

Anyway, if this could be updated in the documentation that would be really helpful!

Thanks for helping with the debugging!

Thanks for confirming it works for you too. I don't think this extraneous data should cause certmonger to fail so I'm going to try to figure out what is causing it grief but at least you have a path forward now.

sscep is a completely different codebase using Java so who knows what cert processing it uses.

@rcritten sscep is C so you can probably just snag the code https://github.com/certnanny/sscep

Ah so it is. I don't recall installing the package and thought it came as a helper from dogtag. I'll take a look.

Metadata Update from @rcritten:
- Issue assigned to rcritten

6 years ago

Metadata Update from @rcritten:
- Issue priority set to: major
- Issue set to the milestone: 0.79

6 years ago

Metadata Update from @rcritten:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

6 years ago

Login to comment on this ticket.

Metadata