KRA can no longer be installed when the server with the first KRA has been decommissioned and is no longer available.
1) Install IPA on first master with CA and KRA 2) Install IPA on first replica with CA and KRA 3) Uninstall IPA from first master, remove it from replication (I used ipa-replica-manage del vm-058-091) 4) Install IPA on second replica with CA and KRA
ipa-replica-manage del vm-058-091
ipa-kra-install on second replica (vm-231) will fail.
2017-04-25T09:10:24Z DEBUG Contents of pkispawn configuration file (/tmp/tmpDpM59W): [KRA] pki_security_domain_https_port = 443 pki_security_domain_password = XXXXXXXX pki_security_domain_user = admin-vm-231.ipa.example pki_issuing_ca_uri = https://vm-231.ipa.example:443 pki_enable_proxy = True pki_restart_configured_instance = False pki_backup_keys = True pki_backup_password = XXXXXXXX pki_client_database_dir = /var/lib/ipa/tmp-C8Cd3l pki_client_database_password = 6It[lf%i(rW_eL_f;P}z?qdb7y.5yM6bn{o8SNrRI pki_client_database_purge = True pki_client_pkcs12_password = XXXXXXXX pki_admin_name = admin-vm-231.ipa.example pki_admin_uid = admin-vm-231.ipa.example pki_admin_email = root@localhost pki_admin_password = XXXXXXXX pki_admin_nickname = ipa-ca-agent pki_admin_subject_dn = cn=ipa-ca-agent,O=DOM-058-091.IPA.EXAMPLE pki_import_admin_cert = True pki_admin_cert_file = /root/.dogtag/pki-tomcat/ca_admin.cert pki_client_admin_cert_p12 = /root/ca-agent.p12 pki_ds_ldap_port = 389 pki_ds_password = XXXXXXXX pki_ds_base_dn = o=kra,o=ipaca pki_ds_database = ipaca pki_ds_create_new_db = False pki_ds_ldaps_port = 636 pki_ds_secure_connection = True pki_ds_secure_connection_ca_pem_file = /etc/ipa/ca.crt pki_subsystem_subject_dn = cn=CA Subsystem,O=DOM-058-091.IPA.EXAMPLE pki_ssl_server_subject_dn = cn=vm-231.ipa.example,O=DOM-058-091.IPA.EXAMPLE pki_audit_signing_subject_dn = cn=KRA Audit,O=DOM-058-091.IPA.EXAMPLE pki_transport_subject_dn = cn=KRA Transport Certificate,O=DOM-058-091.IPA.EXAMPLE pki_storage_subject_dn = cn=KRA Storage Certificate,O=DOM-058-091.IPA.EXAMPLE pki_subsystem_nickname = subsystemCert cert-pki-ca pki_ssl_server_nickname = Server-Cert cert-pki-ca pki_audit_signing_nickname = auditSigningCert cert-pki-kra pki_transport_nickname = transportCert cert-pki-kra pki_storage_nickname = storageCert cert-pki-kra pki_share_db = True pki_share_dbuser_dn = uid=pkidbuser,ou=people,o=ipaca pki_security_domain_hostname = vm-058-114.ipa.example pki_clone = True pki_clone_pkcs12_path = /tmp/tmpMUQaJo pki_clone_pkcs12_password = XXXXXXXX pki_clone_setup_replication = False pki_clone_uri = https://vm-058-114.ipa.example:443 2017-04-25T09:10:24Z DEBUG Starting external process 2017-04-25T09:10:24Z DEBUG args=/usr/sbin/pkispawn -s KRA -f /tmp/tmpDpM59W 2017-04-25T09:12:09Z DEBUG Process finished, return code=1 2017-04-25T09:12:09Z DEBUG stdout=Log file: /var/log/pki/pki-kra-spawn.20170425111025.log Loading deployment configuration from /tmp/tmpDpM59W. Installing KRA into /var/lib/pki/pki-tomcat. Storing deployment configuration into /etc/sysconfig/pki/tomcat/pki-tomcat/kra/deployment.cfg. Importing certificates from /tmp/tmpMUQaJo: --------------- 5 entries found --------------- Certificate ID: 2e28b3a038ceea63801019edcb351b0561c05c08 Serial Number: 0xc Nickname: storageCert cert-pki-kra Subject DN: CN=KRA Storage Certificate,O=DOM-058-091.IPA.EXAMPLE Issuer DN: CN=Certificate Authority,O=DOM-058-091.IPA.EXAMPLE Trust Flags: u,u,u Has Key: true Certificate ID: 25923acc2aa4c9d061212808a34762f2f37f4a5e Serial Number: 0x1 Nickname: caSigningCert cert-pki-ca Subject DN: CN=Certificate Authority,O=DOM-058-091.IPA.EXAMPLE Issuer DN: CN=Certificate Authority,O=DOM-058-091.IPA.EXAMPLE Trust Flags: CT,C,C Has Key: false Certificate ID: 6a0d8510ce34348fac69759f3751fa67a8423aac Serial Number: 0x4 Nickname: subsystemCert cert-pki-ca Subject DN: CN=CA Subsystem,O=DOM-058-091.IPA.EXAMPLE Issuer DN: CN=Certificate Authority,O=DOM-058-091.IPA.EXAMPLE Trust Flags: u,u,u Has Key: true Certificate ID: 81dcd405cb2270d614c818c57e493c71796b3be8 Serial Number: 0xd Nickname: auditSigningCert cert-pki-kra Subject DN: CN=KRA Audit,O=DOM-058-091.IPA.EXAMPLE Issuer DN: CN=Certificate Authority,O=DOM-058-091.IPA.EXAMPLE Trust Flags: u,u,u Has Key: true Certificate ID: 42a3130cbfac57ac2e5a6d3bb1ff1a0b3ddc9f74 Serial Number: 0xb Nickname: transportCert cert-pki-kra Subject DN: CN=KRA Transport Certificate,O=DOM-058-091.IPA.EXAMPLE Issuer DN: CN=Certificate Authority,O=DOM-058-091.IPA.EXAMPLE Trust Flags: u,u,u Has Key: true WARNING: cert caSigningCert cert-pki-ca already exists --------------- Import complete --------------- Imported certificates in /etc/pki/pki-tomcat/alias: Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI ocspSigningCert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u Server-Cert cert-pki-ca u,u,u storageCert cert-pki-kra u,u,u caSigningCert cert-pki-ca CTu,Cu,Cu auditSigningCert cert-pki-ca u,u,Pu Installation failed: Please check the KRA logs in /var/log/pki/pki-tomcat/kra. 2017-04-25T09:12:09Z DEBUG stderr= 2017-04-25T09:12:09Z CRITICAL Failed to configure KRA instance: Command '/usr/sbin/pkispawn -s KRA -f /tmp/tmpDpM59W' returned non-zero exit status 1 2017-04-25T09:12:09Z CRITICAL See the installation logs and the following files/directories for more information: 2017-04-25T09:12:09Z CRITICAL /var/log/pki/pki-tomcat 2017-04-25T09:12:09Z DEBUG Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 423, in start_creation run_step(full_msg, method) File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 413, in run_step method() File "/usr/lib/python2.7/site-packages/ipaserver/install/krainstance.py", line 286, in __spawn_instance nolog_list=(self.dm_password, self.admin_password, pki_pin) File "/usr/lib/python2.7/site-packages/ipaserver/install/dogtaginstance.py", line 148, in spawn_instance self.handle_setup_error(e) File "/usr/lib/python2.7/site-packages/ipaserver/install/dogtaginstance.py", line 395, in handle_setup_error raise RuntimeError("%s configuration failed." % self.subsystem) RuntimeError: KRA configuration failed. 2017-04-25T09:12:09Z DEBUG [error] RuntimeError: KRA configuration failed.
[25/Apr/2017:11:10:42][http-bio-8443-exec-3]: === Token Authentication === [25/Apr/2017:11:10:42][http-bio-8443-exec-3]: === Security Domain Configuration === [25/Apr/2017:11:10:42][http-bio-8443-exec-3]: Joining existing security domain [25/Apr/2017:11:10:42][http-bio-8443-exec-3]: Resolving security domain URL https://vm-058-114.ipa.example:443 [25/Apr/2017:11:10:42][http-bio-8443-exec-3]: Getting security domain cert chain [25/Apr/2017:11:10:42][http-bio-8443-exec-3]: ConfigurationUtils.importCertChain() [25/Apr/2017:11:10:42][http-bio-8443-exec-3]: ConfigurationUtils: GET https://vm-058-114.ipa.example:443/ca/admin/ca/getCertChain [25/Apr/2017:11:10:42][http-bio-8443-exec-3]: Server certificate: [25/Apr/2017:11:10:42][http-bio-8443-exec-3]: - subject: CN=vm-058-114.ipa.example,O=DOM-058-091.IPA.EXAMPLE [25/Apr/2017:11:10:42][http-bio-8443-exec-3]: - issuer: CN=Certificate Authority,O=DOM-058-091.IPA.EXAMPLE [25/Apr/2017:11:10:42][http-bio-8443-exec-3]: ConfigurationUtils: certificate chain: [25/Apr/2017:11:10:42][http-bio-8443-exec-3]: ConfigurationUtils: - CN=Certificate Authority,O=DOM-058-091.IPA.EXAMPLE [25/Apr/2017:11:10:42][http-bio-8443-exec-3]: Getting install token [25/Apr/2017:11:10:42][http-bio-8443-exec-3]: Getting install token [25/Apr/2017:11:10:43][http-bio-8443-exec-3]: Getting domain XML [25/Apr/2017:11:10:43][http-bio-8443-exec-3]: ConfigurationUtils: getting domain info [25/Apr/2017:11:10:43][http-bio-8443-exec-3]: ConfigurationUtils: GET https://vm-058-114.ipa.example:443/ca/admin/ca/getDomainXML [25/Apr/2017:11:10:43][http-bio-8443-exec-3]: Server certificate: [25/Apr/2017:11:10:43][http-bio-8443-exec-3]: - subject: CN=vm-058-114.ipa.example,O=DOM-058-091.IPA.EXAMPLE [25/Apr/2017:11:10:43][http-bio-8443-exec-3]: - issuer: CN=Certificate Authority,O=DOM-058-091.IPA.EXAMPLE [25/Apr/2017:11:10:43][http-bio-8443-exec-3]: ConfigurationUtils: status: 0 [25/Apr/2017:11:10:43][http-bio-8443-exec-3]: ConfigurationUtils: domain info: <?xml version="1.0" encoding="UTF-8" standalone="no"?><DomainInfo><Name>IPA</Name><CAList><CA><Host>vm-058-091.ipa.example</Host><UnSecurePort>80</UnSecurePort><SecurePort>443</SecurePort><SecureEEClientAuthPort>443</SecureEEClientAuthPort><SecureAgentPort>443</SecureAgentPort><SecureAdminPort>443</SecureAdminPort><Clone>FALSE</Clone><SubsystemName>CA vm-058-091.ipa.example 8443</SubsystemName><DomainManager>TRUE</DomainManager></CA><CA><Host>vm-058-114.ipa.example</Host><UnSecurePort>80</UnSecurePort><SecurePort>443</SecurePort><SecureEEClientAuthPort>443</SecureEEClientAuthPort><SecureAgentPort>443</SecureAgentPort><SecureAdminPort>443</SecureAdminPort><Clone>TRUE</Clone><SubsystemName>CA vm-058-114.ipa.example 8443</SubsystemName><DomainManager>TRUE</DomainManager></CA><CA><Host>vm-231.ipa.example</Host><UnSecurePort>80</UnSecurePort><SecurePort>443</SecurePort><SecureEEClientAuthPort>443</SecureEEClientAuthPort><SecureAgentPort>443</SecureAgentPort><SecureAdminPort>443</SecureAdminPort><Clone>TRUE</Clone><SubsystemName>CA vm-231.ipa.example 8443</SubsystemName><DomainManager>TRUE</DomainManager></CA><SubsystemCount>3</SubsystemCount></CAList><KRAList><KRA><Host>vm-058-091.ipa.example</Host><UnSecurePort>80</UnSecurePort><SecurePort>443</SecurePort><SecureEEClientAuthPort>443</SecureEEClientAuthPort><SecureAgentPort>443</SecureAgentPort><SecureAdminPort>443</SecureAdminPort><Clone>FALSE</Clone><SubsystemName>KRA vm-058-091.ipa.example 8443</SubsystemName><DomainManager>FALSE</DomainManager></KRA><KRA><Host>vm-058-114.ipa.example</Host><UnSecurePort>80</UnSecurePort><SecurePort>443</SecurePort><SecureEEClientAuthPort>443</SecureEEClientAuthPort><SecureAgentPort>443</SecureAgentPort><SecureAdminPort>443</SecureAdminPort><Clone>TRUE</Clone><SubsystemName>KRA vm-058-114.ipa.example 8443</SubsystemName><DomainManager>FALSE</DomainManager></KRA><SubsystemCount>2</SubsystemCount></KRAList><OCSPList><SubsystemCount>0</SubsystemCount></OCSPList><TKSList><SubsystemCount>0</SubsystemCount></TKSList><RAList><SubsystemCount>0</SubsystemCount></RAList><TPSList><SubsystemCount>0</SubsystemCount></TPSList></DomainInfo> [25/Apr/2017:11:10:43][http-bio-8443-exec-3]: len is 3 [25/Apr/2017:11:10:43][http-bio-8443-exec-3]: hostname: <vm-058-091.ipa.example> [25/Apr/2017:11:10:43][http-bio-8443-exec-3]: admin_port: <443> [25/Apr/2017:11:10:43][http-bio-8443-exec-3]: hostname: <vm-058-114.ipa.example> [25/Apr/2017:11:10:43][http-bio-8443-exec-3]: admin_port: <443> [25/Apr/2017:11:10:43][http-bio-8443-exec-3]: === Subsystem Configuration === [25/Apr/2017:11:10:43][http-bio-8443-exec-3]: SystemConfigService: validate clone URI: https://vm-058-114.ipa.example:443 [25/Apr/2017:11:10:43][http-bio-8443-exec-3]: SystemConfigService: get configuration entries from master [25/Apr/2017:11:10:43][http-bio-8443-exec-3]: updateNumberRange start host=vm-058-114.ipa.example adminPort=443 eePort=443 [25/Apr/2017:11:10:43][http-bio-8443-exec-3]: ConfigurationUtils: POST https://vm-058-114.ipa.example:443/kra/admin/kra/updateNumberRange [25/Apr/2017:11:10:43][http-bio-8443-exec-3]: Server certificate: [25/Apr/2017:11:10:43][http-bio-8443-exec-3]: - subject: CN=vm-058-114.ipa.example,O=DOM-058-091.IPA.EXAMPLE [25/Apr/2017:11:10:43][http-bio-8443-exec-3]: - issuer: CN=Certificate Authority,O=DOM-058-091.IPA.EXAMPLE [25/Apr/2017:11:10:49][http-bio-8443-exec-3]: content from admin interface =<HTML> <BODY BGCOLOR=white> <P> The Certificate System has encountered an unrecoverable error. <P> Error Message:<BR> <I>java.lang.NullPointerException</I> <P> Please contact your local administrator for assistance. </BODY> </HTML> [25/Apr/2017:11:10:49][http-bio-8443-exec-3]: updateNumberRange: Failed to contact master using admin portorg.xml.sax.SAXParseException; lineNumber: 2; columnNumber: 15; Open quote is expected for attribute "BGCOLOR" associated with an element type "BODY". [25/Apr/2017:11:10:49][http-bio-8443-exec-3]: updateNumberRange: Attempting to contact master using EE port [25/Apr/2017:11:10:49][http-bio-8443-exec-3]: ConfigurationUtils: POST https://vm-058-114.ipa.example:443/kra/ee/kra/updateNumberRange [25/Apr/2017:11:10:49][http-bio-8443-exec-3]: Server certificate: [25/Apr/2017:11:10:49][http-bio-8443-exec-3]: - subject: CN=vm-058-114.ipa.example,O=DOM-058-091.IPA.EXAMPLE [25/Apr/2017:11:10:49][http-bio-8443-exec-3]: - issuer: CN=Certificate Authority,O=DOM-058-091.IPA.EXAMPLE javax.ws.rs.NotFoundException: HTTP 404 Not Found
[25/Apr/2017:11:10:43][ajp-bio-127.0.0.1-8009-exec-13]: UpdateNumberRange: initializing... [25/Apr/2017:11:10:43][ajp-bio-127.0.0.1-8009-exec-13]: according to ccMode, authorization for servlet: kraUpdateNumberRange is LDAP based, not XML {1}, use default authz mgr: {2}. [25/Apr/2017:11:10:43][ajp-bio-127.0.0.1-8009-exec-13]: UpdateNumberRange: done initializing... [25/Apr/2017:11:10:43][ajp-bio-127.0.0.1-8009-exec-13]: CMSServlet:service() uri = /kra/admin/kra/updateNumberRange [25/Apr/2017:11:10:43][ajp-bio-127.0.0.1-8009-exec-13]: CMSServlet::service() param name='xmlOutput' value='true' [25/Apr/2017:11:10:43][ajp-bio-127.0.0.1-8009-exec-13]: CMSServlet::service() param name='sessionID' value='5437395391628274977' [25/Apr/2017:11:10:43][ajp-bio-127.0.0.1-8009-exec-13]: CMSServlet::service() param name='type' value='request' [25/Apr/2017:11:10:43][ajp-bio-127.0.0.1-8009-exec-13]: CMSServlet: kraUpdateNumberRange start to service. [25/Apr/2017:11:10:43][ajp-bio-127.0.0.1-8009-exec-13]: UpdateNumberRange: processing... [25/Apr/2017:11:10:43][ajp-bio-127.0.0.1-8009-exec-13]: UpdateNumberRange process: authentication starts [25/Apr/2017:11:10:43][ajp-bio-127.0.0.1-8009-exec-13]: IP: 10.34.78.231 [25/Apr/2017:11:10:43][ajp-bio-127.0.0.1-8009-exec-13]: AuthMgrName: TokenAuth [25/Apr/2017:11:10:43][ajp-bio-127.0.0.1-8009-exec-13]: CMSServlet: no client certificate found [25/Apr/2017:11:10:43][ajp-bio-127.0.0.1-8009-exec-13]: TokenAuthentication: start [25/Apr/2017:11:10:43][ajp-bio-127.0.0.1-8009-exec-13]: TokenAuthentication: content={hostname=[10.34.78.231], sessionID=[5437395391628274977]} [25/Apr/2017:11:10:43][ajp-bio-127.0.0.1-8009-exec-13]: ConfigurationUtils: POST https://vm-058-091.ipa.example:443/ca/admin/ca/tokenAuthenticate [25/Apr/2017:11:10:46][ajp-bio-127.0.0.1-8009-exec-13]: TokenAuthenticate: failed to contact admin host:port vm-058-091.ipa.example:443 javax.ws.rs.ProcessingException: Unable to invoke request [25/Apr/2017:11:10:46][ajp-bio-127.0.0.1-8009-exec-13]: TokenAuthenticate: attempting ee port 443 [25/Apr/2017:11:10:46][ajp-bio-127.0.0.1-8009-exec-13]: ConfigurationUtils: POST https://vm-058-091.ipa.example:443/ca/ee/ca/tokenAuthenticate [25/Apr/2017:11:10:49][ajp-bio-127.0.0.1-8009-exec-13]: TokenAuthenticate: failed to contact EE host:port vm-058-091.ipa.example:443 javax.ws.rs.ProcessingException: Unable to invoke request [25/Apr/2017:11:10:49][ajp-bio-127.0.0.1-8009-exec-13]: SignedAuditEventFactory: create() message created for eventType=AUTH_FAIL
Marking as critical, but prio is in the lower part of critical bucket.
Metadata Update from @pvoborni: - Issue priority set to: critical - Issue set to the milestone: FreeIPA 4.5.1 - Issue tagged with: bug
Metadata Update from @pvoborni: - Custom field rhbz adjusted to todo
I am still investigating, but after uninstallation of the 1st master, /etc/pki/pki-tomcat/ca/CS.cfg on the replica shows securitydomain.host=(new master), while /etc/pki/pki-tomcat/kra/CS.cfg contains securitydomain.host=(first master).
securitydomain.host is used in ToeknAuthentication.java function public IAuthToken authenticate() in order to find which server to use for authentication. It's likely that uninstall did not properly clean this setting.
Also found these errors in pki-kra-destroy.log on the 1st master:
2017-05-09 14:15:07 pkidestroy : INFO ....... contacting the CA to update the KRA connector 2017-05-09 14:15:07 pkidestroy : ERROR ....... unable to access security domain. Continuing .. HTTPSConnectionPool(host='vm-server.domain.com', port=443): Max retries exceeded with url: /ca/rest/securityDomain/domainInfo (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f91d6e95750>: Failed to establish a new connection: [Errno 111] Connection refused',)) 2017-05-09 14:15:07 pkidestroy : INFO ....... contacting the security domain master to update security domain 'IPA' 2017-05-09 14:15:07 pkidestroy : WARNING ....... this 'KRA' entry will NOT be deleted from security domain 'IPA'! 2017-05-09 14:15:07 pkidestroy : WARNING ....... security domain 'IPA' may be offline or unreachable! 2017-05-09 14:15:07 pkidestroy : ERROR ....... subprocess.CalledProcessError: Command '['/usr/bin/sslget', '-n', 'subsystemCert cert-pki-ca', '-p', '7Gl^5t3dJivjuMo$^R+]&>~%u2)+R5fO*tZ}(KfKQ', '-d', '/etc/pki/pki-tomcat/alias', '-e', 'name="/var/lib/pki/pki-tomcat"&type=KRA&list=kraList&host=vm-server.domain.com&sport=443&ncsport=443&adminsport=443&agentsport=443&operation=remove', '-v', '-r', '/ca/agent/ca/updateDomainXML', 'vm-server.domain.com:443']' returned non-zero exit status 6!
Not sure if it was this bug, but IIRC, @cheimes found out that after uninstallation there is still the original master in some LDAP entry present at first place then it is probably used by installer (not sure if IPA or PKI).
Metadata Update from @pvoborni: - Issue assigned to frenaud
Metadata Update from @frenaud: - Assignee reset - Custom field on_review adjusted to https://github.com/freeipa/freeipa/pull/788
Metadata Update from @pvoborni: - Custom field rhbz adjusted to https://bugzilla.redhat.com/show_bug.cgi?id=1451228 (was: todo)
Issue linked to bug 1451228
Metadata Update from @frenaud: - Issue assigned to frenaud
master:
ipa-4-5:
Metadata Update from @mbasti: - Issue close_status updated to: fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.