#8212 Nightly test failure in test_integration/test_backup_and_restore.py::TestReplicaInstallAfterRestore::test_replica_install_after_restore: ipa-restore does not detect PKI start failure
Opened 9 months ago by frenaud. Modified a month ago

The nightly test test_integration/test_backup_and_restore.py::TestReplicaInstallAfterRestore::test_replica_install_after_restore failed in PR #147 - testing_master_389ds.
The test failed during replica installation, trying to communicate with PKI server on the master:

[ipatests.pytest_ipa.integration.host.Host.replica1.cmd30]   [5/29]: configuring certificate server instance
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd30] Failed to configure CA instance: CalledProcessError(Command ['/usr/sbin/pkispawn', '-s', 'CA', '-f', '/tmp/tmpz1kefvsi'] returned non-zero exit status 1: 'WARNING : pkiparser      Unable to get security domain info: 503 Server Error: Service Unavailable for url: https://master.ipa.test:443/ca/rest/securityDomain/domainInfo\n')
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd30] See the installation logs and the following files/directories for more information:
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd30]   /var/log/pki/pki-tomcat
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd30]   [error] RuntimeError: CA configuration failed.
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd30] CA configuration failed.
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd30] The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd30] Your system may be partly configured.
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd30] Run /usr/sbin/ipa-server-install --uninstall to clean up.
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd30] 
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd30] Exit code: 1

The journal log on the master shows that PKI failed to start during the ipa-restore command:

Feb 29 21:17:23 master.ipa.test ipa-pki-wait-running[34880]: ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='master.ipa.test', port=8080): Read timed out. (read timeout=1.0)
Feb 29 21:17:24 master.ipa.test systemd[1]: pki-tomcatd@pki-tomcat.service: Start-post operation timed out. Stopping.
Feb 29 21:17:24 master.ipa.test systemd[1]: pki-tomcatd@pki-tomcat.service: Control process exited, code=killed, status=15/TERM
Feb 29 21:17:25 master.ipa.test systemd[1]: pki-tomcatd@pki-tomcat.service: Failed with result 'timeout'.
Feb 29 21:17:25 master.ipa.test systemd[1]: Failed to start PKI Tomcat Server pki-tomcat.
Feb 29 21:17:25 master.ipa.test audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pki-tomcatd@pki-tomcat comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Feb 29 21:17:25 master.ipa.test systemd[1]: pki-tomcatd@pki-tomcat.service: Consumed 34.599s CPU time.
Feb 29 21:17:25 master.ipa.test systemd[1]: Reached target PKI Tomcat Server.

Need to investigate how it is possible for ipa-restore to not detect the PKI start failure.


Are the full logs available anywhere?

The full IPA is started twice so having some idea which one failed could be useful (once as a start and once as a restart).

But it all happens in a big try/finally block so if a service failed to start then it should either raise an exception or have a non-zero returncode.

Login to comment on this ticket.

Metadata