I have an IPA deployment that went from 3.3 to 4.2 upgrades over several years. The latest setup has no ca.sslserver.certreq parameter in CS.cfg. This breaks KRA deployment as ipa-kra-install attempts to fetch the certreq and fails, causing null pointer exception.
Set of upgrades:
# for i in 49 47 46 45 30 27 ; do dnf history info $i | grep -A1 pki-base ; done Upgraded pki-base-10.2.6-4.fc22.noarch @updates-testing Upgrade 10.2.7-0.2.fc22.noarch @vakwetu-dogtag_10.2.7_test_builds Upgraded pki-base-10.2.6-2.fc22.noarch @updates-testing Upgrade 10.2.6-4.fc22.noarch @updates-testing Upgraded pki-base-10.2.6-1.fc22.noarch @updates-testing Upgrade 10.2.6-2.fc22.noarch @updates-testing Upgraded pki-base-10.2.5-1.fc22.noarch (unknown) Upgrade 10.2.6-1.fc22.noarch @updates-testing Upgraded pki-base-10.0.5-1.fc19.noarch @updates-testing/19 Upgrade 10.0.6-1.fc19.noarch @updates-testing/19 Upgraded pki-base-10.0.4-2.fc19.noarch (unknown) Upgrade 10.0.5-1.fc19.noarch @updates-testing/19
There should be a copy of the cert request stored in the database, but currently there is no tool to find the exact cert request for a given cert. A manual search probably can be done using ldapsearch. A better way is to use the CLI proposed in ticket #1552.
What is the urgency of this ticket?
So I went ahead and generated certificate requests based on the existing certificates using 'openssl x509 -x509toreq'. I needed to create certreq entries for 'Server-Cert cert-pki-ca' and 'subsystemCert cert-pki-ca'.
This allowed me to proceed few steps more. Step [2/7] failed because pki was unable to create ipakra user:
2015-08-12T09:52:52Z DEBUG Starting external process 2015-08-12T09:52:52Z DEBUG args='/usr/bin/pki' '-d' '/tmp/tmp-AdxN38' '-c' 'XXXXXXX' '-n' 'ipa-ca-agent' 'kra-user-add' 'ipakra' '--fullName' 'IPA KRA User' 2015-08-12T09:52:54Z DEBUG Process finished, return code=255 2015-08-12T09:52:54Z DEBUG stdout= 2015-08-12T09:52:54Z DEBUG stderr=ProcessingException: Unable to invoke request 2015-08-12T09:52:54Z DEBUG Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 416, in start_creation run_step(full_msg, method) File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 406, in run_step method() File "/usr/lib/python2.7/site-packages/ipaserver/install/krainstance.py", line 299, in __add_ra_user_to_agent_group ipautil.run(args) File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 373, in run raise CalledProcessError(p.returncode, arg_string, stdout) CalledProcessError: Command ''/usr/bin/pki' '-d' '/tmp/tmp-AdxN38' '-c' 'XXXXXXX' '-n' 'ipa-ca-agent' 'kra-user-add' 'ipakra' '--fullName' 'IPA KRA User'' returned non-zero exit status 255 2015-08-12T09:52:54Z DEBUG [error] CalledProcessError: Command ''/usr/bin/pki' '-d' '/tmp/tmp-AdxN38' '-c' 'XXXXXXXX' '-n' 'ipa-ca-agent' 'kra-user-add' 'ipakra' '--fullName' 'IPA KRA User'' returned non-zero exit status 255 2015-08-12T09:52:54Z ERROR Your system may be partly configured. Run ipa-kra-install --uninstall to clean up.
I think upgradability of PKI has to be reworked. We need to ensure upgrade from IPA 3.3 to 4.2 via all intermediate versions is working.
Just a note, the cert requests can also be obtained as follows:
The installation now fails to set up a KRA agent at the "kra-user-add" step. Please follow this instruction to set up the KRA agent manually: http://pki.fedoraproject.org/wiki/IPA_Development#KRA_Agent_Setup
If the manual step also fails, please repeat the last command in verbose mode and post the output, for example:
$ pki -v -c Secret123 -n ipa-ca-agent kra-user-add ipakra --fullName "IPA KRA User"
manual creation fails now with following output:
Server URI: http://id.vda.li:8080 Client security database: /tmp/tmp-FFo4P9 Message format: null Command: kra-user-add ipakra --fullName "IPA KRA User" Initializing client security database Logging into security token Module: kra HTTP request: GET /kra/rest/account/login HTTP/1.1 Accept-Encoding: gzip, deflate Accept: application/xml Host: id.vda.li:8080 Connection: Keep-Alive User-Agent: Apache-HttpClient/4.4 (Java 1.5 minimum; Java/1.8.0_51) HTTP response: HTTP/1.1 302 Found Server: Apache-Coyote/1.1 Cache-Control: private Expires: Thu, 01 Jan 1970 00:00:00 GMT Location: https://id.vda.li:8443/kra/rest/account/login Content-Length: 0 Date: Thu, 13 Aug 2015 11:56:47 GMT HTTP redirect: https://id.vda.li:8443/kra/rest/account/login Client certificate: ipa-ca-agent HTTP request: GET /kra/rest/account/login HTTP/1.1 Accept-Encoding: gzip, deflate Accept: application/xml Host: id.vda.li:8443 Connection: Keep-Alive User-Agent: Apache-HttpClient/4.4 (Java 1.5 minimum; Java/1.8.0_51) Server certificate: CN=id.vda.li,O=VDA.LI javax.ws.rs.ProcessingException: Unable to invoke request at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:287) at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:407) at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:102) at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:62) at com.sun.proxy.$Proxy23.login(Unknown Source) at com.netscape.certsrv.account.AccountClient.login(AccountClient.java:45) at com.netscape.certsrv.client.SubsystemClient.login(SubsystemClient.java:49) at com.netscape.cmstools.cli.KRACLI.login(KRACLI.java:50) at com.netscape.cmstools.cli.SubsystemCLI.execute(SubsystemCLI.java:54) at com.netscape.cmstools.cli.CLI.execute(CLI.java:337) at com.netscape.cmstools.cli.MainCLI.execute(MainCLI.java:557) at com.netscape.cmstools.cli.MainCLI.main(MainCLI.java:569) Caused by: java.io.IOException: SocketException cannot write on socket at org.mozilla.jss.ssl.SSLSocket.write(SSLSocket.java:1099) at org.mozilla.jss.ssl.SSLOutputStream.write(SSLOutputStream.java:56) at org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:159) at org.apache.http.impl.io.AbstractSessionOutputBuffer.flush(AbstractSessionOutputBuffer.java:166) at org.apache.http.impl.AbstractHttpClientConnection.doFlush(AbstractHttpClientConnection.java:272) at org.apache.http.impl.AbstractHttpClientConnection.flush(AbstractHttpClientConnection.java:277) at org.apache.http.impl.conn.ManagedClientConnectionImpl.flush(ManagedClientConnectionImpl.java:181) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:240) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:122) at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:883) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:283) ... 11 more
Tomcat does respond on port 8443 and attempt to GET /kra/rest/account/login returns 401 (unauthorized), asking to login.
Could you verify that the CA admin certificate is still valid?
$ certutil -L -d ~/.dogtag/nssdb -n ipa-ca-agent
Per IRC discussion the CA admin certificate has actually expired. Currently certmonger does not support tracking certificate in PKCS #12 file. A manual renewal may be necessary.
I've put ca-agent.p12 to NSS database in /etc/pki/tls/private/ipa-ca with pk12util and then ran renewal via certmonger using agent CA interface:
getcert request -d /etc/pki/tls/private/ipa-ca -p /etc/pki/tls/private/ipa-ca/pwdfile.txt -n ipa-ca-agent -c 'dogtag-ipa-renew-agent'
However, ipa-kra-install still fails after I copied the renewed certificate back to ca-agent.p12 with pk12util.
2015-08-19T15:29:07Z DEBUG args='/usr/bin/pki' '-d' '/tmp/tmp-rkoP5Q' '-c' 'XXXXXXX' '-n' 'ipa-ca-agent' 'kra-user-add' 'ipakra' '--fullName' 'IPA KRA User' 2015-08-19T15:29:09Z DEBUG Process finished, return code=255 2015-08-19T15:29:09Z DEBUG stdout= 2015-08-19T15:29:09Z DEBUG stderr=PKIException: Unauthorized
KRA instance log says it cannot map the certificate to any user:
19/Aug/2015:15:29:04][http-bio-8443-exec-3]: SourceConfigStore: storing jss.ssl.sslserver.ectype: ECDHE [19/Aug/2015:15:29:09][http-bio-8443-exec-5]: PKIRealm.logDebug: Authenticating certificate chain: [19/Aug/2015:15:29:09][http-bio-8443-exec-5]: PKIRealm.getAuditUserfromCert: certUID=CN=ipa-ca-agent, O=VDA.LI [19/Aug/2015:15:29:09][http-bio-8443-exec-5]: PKIRealm.logDebug: CN=ipa-ca-agent, O=VDA.LI [19/Aug/2015:15:29:09][http-bio-8443-exec-5]: CertUserDBAuth: started [19/Aug/2015:15:29:09][http-bio-8443-exec-5]: CertUserDBAuth: Retrieving client certificate [19/Aug/2015:15:29:09][http-bio-8443-exec-5]: CertUserDBAuth: Got client certificate [19/Aug/2015:15:29:09][http-bio-8443-exec-5]: Authentication: client certificate found [19/Aug/2015:15:29:09][http-bio-8443-exec-5]: In LdapBoundConnFactory::getConn() [19/Aug/2015:15:29:09][http-bio-8443-exec-5]: masterConn is connected: true [19/Aug/2015:15:29:09][http-bio-8443-exec-5]: getConn: conn is connected true [19/Aug/2015:15:29:09][http-bio-8443-exec-5]: getConn: mNumConns now 2 [19/Aug/2015:15:29:09][http-bio-8443-exec-5]: returnConn: mNumConns now 3 [19/Aug/2015:15:29:09][http-bio-8443-exec-5]: CertUserDBAuthentication: cannot map certificate to any user [19/Aug/2015:15:29:09][http-bio-8443-exec-5]: SignedAuditEventFactory: create() message=[AuditEvent=AUTH_FAIL][SubjectID=CN=ipa-ca-agent, O=VDA.LI][Outcome=Failure][AuthMgr=certUserDBAuthMgr][AttemptedCred=CN=ipa-ca-agent, O=VDA.LI] authentication failure
Per IRC discussion, the issue can be fixed with the following actions:
For reference, the stacktrace containing the certreq exception is located here: http://pastebin.test.redhat.com/304244
Now, what it shows is that we throw a null pointer exception in the method updateConfiguration() in SystemConfigService.java when we try to store the certreq.
After going through the code, my suspicion is that we do not actually need this certreq. The certreq is used initially when generating the request objects when the cert was originally generated.
I believe it used to be needed to display the cert request when cloning. As we no longer support the install panels though, this is no longer a concern.
Basically, I think we can modify updateConfiguration() to store the certreq only if its available, and perhaps to allow the request entry to be removed when the install completes.
This would have to be tested though - especially with cloning.
IPA has been making some changes:
So the only remaining tasks are:
Removing cert data and requests from CS.cfg will require thorough tests to avoid regression, so it will be implemented in the future in tickets #1598 and #1552.
For now if the parameters are missing it should be restored using the newly added tool as described in the following page: http://pki.fedoraproject.org/wiki/Subsystem_Certificate_Cache
Fixed in master:
Metadata Update from @abbra: - Issue assigned to edewata - Issue set to the milestone: 10.2.6
Dogtag PKI is moving from Pagure issues to GitHub issues. This means that existing or new issues will be reported and tracked through Dogtag PKI's GitHub Issue tracker.
This issue has been cloned to GitHub and is available here: https://github.com/dogtagpki/pki/issues/2110
If you want to receive further updates on the issue, please navigate to the GitHub issue and click on Subscribe button.
Subscribe
Thank you for understanding, and we apologize for any inconvenience.
Login to comment on this ticket.