We're running FreeIPA 3.0.0 on CentOS 6.6, with the following relevant version numbers:
jss-4.2.6-24.el6.x86_64 tomcatjss-2.1.0-3.el6.noarch tomcat6-6.0.24-80.el6.x86_64 pki-ca-9.0.3-38.el6_6.noarch ipa-server-3.0.0-42.el6.centos.x86_64
The IPA server is in general working great, except when trying to manage certificates, either via the CLI or the web GUI. For instance:
$ ipa cert-show 1 ipa: ERROR: Certificate operation cannot be completed: Unable to communicate with CMS (Not Found)
Upon investigation, /var/log/pki-ca/localhost.2015-01-09.log seemed to indicate that the "/ca" servlet wasn't starting properly, throwing a NoSuchMethodError. Subsequent calls to the CA, as with the "ipa cert-show 1" above, would then generate an exception saying that the "CS server is not ready to serve."
I'll attach the localhost.2015-09-09.log and catalina.out from /var/log/pki-ca.
Let me know if I can provide any other information!
attachment localhost.2015-01-09.log
attachment catalina.out
Oh, and I neglected to mention that I'm not sure exactly when this problem started happening. I haven't tried to do any certificate management probably since the box was first installed in 2013. It's seen some system upgrades since, and it seems likely that some upgrade along the line must have triggered this behavior. I'm only noticing the errors now because I'd been attempting to renew the main SSL certificate, which is due to expire next week.
I've bounced the IPA services a few times (via "ipactl restart"), restarted the pki-ca a few times ("service ipa-cad restart"), and even done a full reboot on the box, to no avail.
attachment debug.gz
As requested, uploading pki-ca's debug log, in case there's something useful in there.
And as a record on the ticket (we've had some off-ticket conversation), I did find some references to the following in httpd's error_log, which indicates that there might be an expired ticket hanging around somewhere, though I can't find it:
[Fri Jan 09 10:55:25 2015] [error] [client 10.80.31.61] Credentials for HTTP/nslp-dmzauth1.nsight.com@NSIGHT.COM have expired or will soon expire - now 1420822525 endtime 1420343305, referer: https://nslp-dmzauth1.nsight.com/ipa/ui/
[Mon Jan 12 09:25:52 2015] [error] [client 172.16.3.100] Credentials for HTTP/nslp-dmzauth1.nsight.com@NSIGHT.COM have expired or will soon expire - now 1421076352 endtime 1420917385, referer: https://nslp-dmzauth1.nsight.com/ipa/xml
Interestingly, the "endtime" appears to have changed since I first looked this morning (when the Jan 9 entry was the most recent), though in both cases it's a date in the past.
Given that I was seeing those dates, I was going to try resetting the date on the box to sometime in the past, to see if that would allow things to start working again, though I haven't been able to find any expired certs just by looking around.
The "HTTP/nslp-dmzauth1.nsight.com@NSIGHT.COM" cert itself that I can see via the GUI says that it doesn't expire until Friday, FWIW.
Okay, I didn't end up having any luck with setting the date back. I ran the following:
ipactl stop service ntpd stop date 01011156 ipactl start
(I did verify that the date change "took" before starting.)
After the first boot there was a similar error in httpd's error_log, but talking about a cert that's not valid yet, like so:
[Thu Jan 01 11:56:31 2015] [error] [client 172.16.3.100] Credentials for HTTP/nslp-dmzauth1.nsight.com@NSIGHT.COM are not yet valid. now 1420134991 starttime 1421076352, referer: https://nslp-dmzauth1.nsight.com/ipa/xml
(now: Thu, 01 Jan 2015 11:56:31 CST, starttime: Mon, 12 Jan 2015 09:25:52 CST)
I figured that was probably due to something left-over from the previous run, so I did an "ipactl restart" and the error_log was clean upon startup, but an "ipa cert-show 1" still failed:
ipa: ERROR: Certificate operation cannot be completed: Unable to communicate with CMS (Not Found)
The error_log entries for that command, btw, is:
[Thu Jan 01 11:59:11 2015] [error] ipa: ERROR: ipaserver.plugins.dogtag.ra.get_certificate(): Unable to communicate with CMS (Not Found)
So then I stopped IPA, synced the date back to "now," started things back up, and it looks like there was an old cert hanging around:
[Mon Jan 12 12:05:15 2015] [error] [client 172.16.3.100] Credentials for HTTP/nslp-dmzauth1.nsight.com@NSIGHT.COM have expired or will soon expire - now 1421085915 endtime 1420221391, referer: https://nslp-dmzauth1.nsight.com/ipa/xml
(now: Mon, 12 Jan 2015 12:05:15 CST, endtime: Fri, 02 Jan 2015 11:56:31 CST)
So I restarted with "ipactl restart" once again, and now the most recent startup is free of errors like that in httpd's error_log, but the "ipa cert-show 1" still produces the same error.
Based on the debug log - it looks like the audit signing cert is failing the self tests. That may mean that it is expired or that it does not have the right permissions.
The /var/log/pki-ca/selftest.log should have some addtional details.
Also, please attach the output of :
certutil -L -d /var/lib/pki-ca/alias certutil -L -d /var/lib/pki-ca/alias -n "nickanme of audit signing cert"
After discussion on IRC, moved to Dogtag 9.0.x milestone.
Okay, here's some of the output you requested... I'm just including what I assume are the relevant bits of the certs in question, though I could paste/upload them in full if you like. I'll upload selfest.log in a second as well:
# certutil -L -d /var/lib/pki-ca/alias Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI auditSigningCert cert-pki-ca u,u,u caSigningCert cert-pki-ca CTu,Cu,Cu auditSigningCert cert-pki-ca u,u,Pu ocspSigningCert cert-pki-ca u,u,Pu Server-Cert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,Pu
I do see that there are two "auditSigningCert cert-pki-ca" entries in there... I'm not sure how to specify which one to show. The ones I can list based on just the nickname are all still valid:
auditSigningCert:
# certutil -L -d /var/lib/pki-ca/alias -n "auditSigningCert cert-pki-ca" Certificate: Data: Version: 3 (0x2) Serial Number: 20 (0x14) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: "CN=Certificate Authority,O=NSIGHT.COM" Validity: Not Before: Tue Dec 09 03:48:22 2014 Not After : Mon Nov 28 03:48:22 2016 Subject: "CN=CA Audit,O=NSIGHT.COM"
caSigningCert:
# certutil -L -d /var/lib/pki-ca/alias -n "caSigningCert cert-pki-ca" Certificate: Data: Version: 3 (0x2) Serial Number: 1 (0x1) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: "CN=Certificate Authority,O=NSIGHT.COM" Validity: Not Before: Tue Jan 15 20:17:11 2013 Not After : Fri Jan 15 20:17:11 2021 Subject: "CN=Certificate Authority,O=NSIGHT.COM"
ocspSigningCert:
# certutil -L -d /var/lib/pki-ca/alias -n "ocspSigningCert cert-pki-ca" Certificate: Data: Version: 3 (0x2) Serial Number: 19 (0x13) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: "CN=Certificate Authority,O=NSIGHT.COM" Validity: Not Before: Tue Dec 09 03:47:22 2014 Not After : Mon Nov 28 03:47:22 2016 Subject: "CN=OCSP Subsystem,O=NSIGHT.COM"
Server-Cert:
# certutil -L -d /var/lib/pki-ca/alias -n "Server-Cert cert-pki-ca" Certificate: Data: Version: 3 (0x2) Serial Number: 21 (0x15) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: "CN=Certificate Authority,O=NSIGHT.COM" Validity: Not Before: Tue Dec 09 03:47:22 2014 Not After : Mon Nov 28 03:47:22 2016 Subject: "CN=nslp-dmzauth1.nsight.com,O=NSIGHT.COM"
subsystemCert:
# certutil -L -d /var/lib/pki-ca/alias -n "subsystemCert cert-pki-ca" Certificate: Data: Version: 3 (0x2) Serial Number: 23 (0x17) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: "CN=Certificate Authority,O=NSIGHT.COM" Validity: Not Before: Tue Dec 09 03:47:22 2014 Not After : Mon Nov 28 03:47:22 2016 Subject: "CN=CA Subsystem,O=NSIGHT.COM"
attachment selftests.log
The extra "auditSigningCert cert-pki-ca" might have been added by certmonger during cert renewal, but it's missing a "P" trust attribute. Could you try adding the missing attribute with the following command?
$ certutil -M -d /var/lib/pki-ca/alias -n "auditSigningCert cert-pki-ca" -t "u,u,Pu"
Then restart IPA (including Dogtag) and see if it fixes the problem.
Aha! Yep, that absolutely did the trick. I can run an "ipa cert-show 1" without errors, and I was able to run an "ipa-getcert resubmit -t FOO" to resubmit the HTTP cert for renewal, and that's looking quite good at the moment.
I'll keep an eye on the others to see if they auto-renew, but I'm sure I can just force a resubmit right now if they don't, so I think we're good to go here.
Thanks for all your help!
It looks like the problem was already fixed in RHEL 7: https://bugzilla.redhat.com/show_bug.cgi?id=918335
So I'd suggest upgrading to CentOS 7 to get the proper fix. Thanks!
Just one further note on here to help anyone else who might have run into this, btw - it looks like the current version of ipa-server in el6 also does contain this fix. I checked /usr/lib64/ipa/certmonger/renew_ca_cert and it contains the fix that was patched in that bugzilla, so el6 should be fine as far as this bug's concerned. My certs must have just auto-renewed at a time when it hadn't been patched yet.
Anyway, thanks again, I appreciate it.
Metadata Update from @apocalyptech: - Issue set to the milestone: 9.0
Dogtag PKI is moving from Pagure issues to GitHub issues. This means that existing or new issues will be reported and tracked through Dogtag PKI's GitHub Issue tracker.
This issue has been cloned to GitHub and is available here: https://github.com/dogtagpki/pki/issues/1799
If you want to receive further updates on the issue, please navigate to the GitHub issue and click on Subscribe button.
Subscribe
Thank you for understanding, and we apologize for any inconvenience.
Log in to comment on this ticket.