Cloned from https://pagure.io/dogtagpki/issue/2939 to address the IPA-specific parts.
Original description
Version-Release number of selected component (if applicable): ipa-server-4.5.0-22.el7_4.x86_64
How reproducible: always
Steps to Reproduce: 1. ipa-server-install --setup-dns --forwarder=FORWARDER --ip-address=12.13.14.15 -r TESTRELM.TEST -p Secret123 -a Secret123 -U --allow-zone-overlap
ipa-server-install --setup-dns --forwarder=FORWARDER --ip-address=12.13.14.15 -r TESTRELM.TEST -p Secret123 -a Secret123 -U --allow-zone-overlap
NOTE: 12.13.14.15 ip which is not present on the system
Actual results:
[1/29]: configuring certificate server instance ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to configure CA instance: Command '/usr/sbin/pkispawn -s CA -f /tmp/tmpn6lClp' returned non-zero exit status 1 ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the installation logs and the following files/directories for more information: ipa.ipaserver.install.cainstance.CAInstance: CRITICAL /var/log/pki/pki-tomcat [error] RuntimeError: CA configuration failed. ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR CA configuration failed. ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information
Initial reproduction notes and analysis by @frenaud:
Issue reproducible.
In order to reproduce, you need to add the machine IP address to /etc/hosts (the existing one, not the fake one).
In this case, pki-spawn fails with:
2017-11-23 09:37:48 pkispawn : INFO ....... executing 'systemctl daemon-reload' 2017-11-23 09:37:48 pkispawn : INFO ....... executing 'systemctl start pki-tomcatd@pki-tomcat.service' 2017-11-23 09:37:49 pkispawn : DEBUG ........... pki_protocol https hostname vm-110.abc.idm.lab.eng.brq.redhat.com port 8443 subsystem ca 2017-11-23 09:39:56 pkispawn : DEBUG ........... No connection - server may still be down 2017-11-23 09:39:56 pkispawn : DEBUG ........... No connection - exception thrown: ('Connection aborted.', error(110, 'Connection timed out')) 2017-11-23 09:39:57 pkispawn : ERROR ....... server failed to restart 2017-11-23 09:39:57 pkispawn : DEBUG ....... Error Type: Exception 2017-11-23 09:39:57 pkispawn : DEBUG ....... Error Message: server failed to restart 2017-11-23 09:39:57 pkispawn : DEBUG ....... File "/usr/sbin/pkispawn", line 533, in main scriptlet.spawn(deployer) File "/usr/lib/python2.7/site-packages/pki/server/deployment/scriptlets/configuration.py", line 374, in spawn raise Exception("server failed to restart")
The code shows that pki spawn is checking if the server is running (https://github.com/dogtagpki/pki/blob/DOGTAG_10_4_BRANCH/base/server/python/pki/server/deployment/pkihelper.py#L1020) by connecting to the url https://hostname:8443/ca/admin/ca/getStatus. Note that there is only one connection try.
hostname
Wireshark demonstrates that the fake IP address is used.
I noticed that if the timeout waiting for the server to come up is raised to 200s for instance, pki spawn finishes successfully (https://github.com/dogtagpki/pki/blob/DOGTAG_10_4_BRANCH/base/server/python/pki/server/deployment/scriptlets/configuration.py#L369).
So there are probably 2 timeouts that interact here: - the timeout set in deployer.instance.wait_for_startup(60), which allows to perform multiple times a get on https://hostname:8443/ca/admin/ca/getStatus (until timeout is exhausted) - the timeout used to establish the connection when get(url) is called, probably defined at the system level.
deployer.instance.wait_for_startup(60)
get(url)
When first timeout < second timeout, the get(url) can be performed only once and fails. If first timeout > 2nd timeout, the get(url) can be performed a second time and the second time succeeds.
master:
Metadata Update from @cheimes: - Issue close_status updated to: fixed - Issue status updated to: Closed (was: Open)
@ftweedal I'm seeing an increased amount of CI failures for a couple of days. CA installation is failing with timeout every now and then. Could this be related to this PR?
Possibly related to this PKI ticket: https://pagure.io/dogtagpki/issue/2973
Metadata Update from @stlaz: - Issue set to the milestone: FreeIPA 4.6.4 - Issue status updated to: Open (was: Closed)
ipa-4-6:
Metadata Update from @stlaz: - Issue close_status updated to: fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.