#8497 Nightly failure in test_integration/test_installation.py::TestInstallWithCA_DNS2::test_replica2_with_dns_install
Closed: worksforme 17 days ago by frenaud. Opened 3 months ago by mpolovka.

test_replica2_with_dns_install is failing with

E subprocess.CalledProcessError: Command '['ipa-replica-install', '-w', 'Secret.123', '-p', 'Secret.123', '-U', '--setup-dns', '--forwarder', '192.168.122.1', '--ip-address', '192.168.122.31', '-r', 'IPA.TEST', '--domain', 'ipa.test']' returned non-zero exit status 1.

self = <ipatests.test_integration.test_installation.TestInstallWithCA_DNS2 object at 0x7f4ecc6d9550>

    def test_replica2_with_dns_install(self):
        tasks.install_replica(self.master, self.replicas[2], setup_ca=False,
>                             setup_dns=True)

test_integration/test_installation.py:127: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pytest_ipa/integration/tasks.py:444: in install_replica
    stdin_text=stdin_text)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <ipatests.pytest_ipa.integration.host.Host replica1.ipa.test (replica)>
argv = ['ipa-replica-install', '-w', 'Secret.123', '-p', 'Secret.123', '-U', ...]
set_env = True, stdin_text = None, log_stdout = True, raiseonerr = True
cwd = None, ok_returncode = 0

    def run_command(self, argv, set_env=True, stdin_text=None,
                    log_stdout=True, raiseonerr=True,
                    cwd=None, ok_returncode=0):
        """Wrapper around run_command to log stderr on raiseonerr=True

            :param ok_returncode: return code considered to be correct,
                                  you can pass an integer or sequence of integers
            """
        result = super(Host, self).run_command(
            argv, set_env=set_env, stdin_text=stdin_text,
            log_stdout=log_stdout, raiseonerr=False, cwd=cwd
        )
        try:
            result_ok = result.returncode in ok_returncode
        except TypeError:
            result_ok = result.returncode == ok_returncode
        if not result_ok and raiseonerr:
            result.log.error('stderr: %s', result.stderr_text)
            raise subprocess.CalledProcessError(
                result.returncode, argv,
>               result.stdout_text
            )
E           subprocess.CalledProcessError: Command '['ipa-replica-install', '-w', 'Secret.123', '-p', 'Secret.123', '-U', '--setup-dns', '--forwarder', '192.168.122.1', '--ip-address', '192.168.122.31', '-r', 'IPA.TEST', '--domain', 'ipa.test']' returned non-zero exit status 1.

pytest_ipa/integration/host.py:90: CalledProcessError
----------------------------- Captured stderr call -----------------------------

[ipatests.pytest_ipa.integration] Adding replica1.ipa.test:/var/log/ipareplica-conncheck.log to list of logs to collect
[ipatests.pytest_ipa.integration] Adding replica1.ipa.test:/var/log/ipareplica-ca-install.log to list of logs to collect
[ipatests.pytest_ipa.integration] Adding replica1.ipa.test:/var/log/ipaclient-install.log to list of logs to collect
[ipatests.pytest_ipa.integration] Adding replica1.ipa.test:/var/log/ipaserver-kra-install.log to list of logs to collect
[ipatests.pytest_ipa.integration] Adding replica1.ipa.test:/var/log/ipa-custodia.audit.log to list of logs to collect
[ipatests.pytest_ipa.integration] Adding replica1.ipa.test:/var/log/ipaclient-uninstall.log to list of logs to collect
[ipatests.pytest_ipa.integration] Adding replica1.ipa.test:/var/log/iparestore.log to list of logs to collect
[ipatests.pytest_ipa.integration] Adding replica1.ipa.test:/var/log/ipabackup.log to list of logs to collect
[ipatests.pytest_ipa.integration] Adding replica1.ipa.test:/var/log/kadmind.log to list of logs to collect
[ipatests.pytest_ipa.integration] Adding replica1.ipa.test:/var/log/krb5kdc.log to list of logs to collect
[ipatests.pytest_ipa.integration] Adding replica1.ipa.test:/var/log/httpd/error_log to list of logs to collect
[ipatests.pytest_ipa.integration] Adding replica1.ipa.test:/var/log/pki/ to list of logs to collect
[ipatests.pytest_ipa.integration] Adding replica1.ipa.test:/var/log/audit/audit.log to list of logs to collect
[ipatests.pytest_ipa.integration.host.Host.master.ParamikoTransport] RUN ['kinit', 'admin']
[ipatests.pytest_ipa.integration.host.Host.master.cmd36] RUN ['kinit', 'admin']
[ipatests.pytest_ipa.integration.host.Host.master.cmd36] Password for admin@IPA.TEST: 
[ipatests.pytest_ipa.integration.host.Host.master.cmd36] Exit code: 0
[ipatests.pytest_ipa.integration.host.Host.master.ParamikoTransport] RUN ['ipa', 'dnsconfig-mod', '--allow-sync-ptr=true']
[ipatests.pytest_ipa.integration.host.Host.master.cmd37] RUN ['ipa', 'dnsconfig-mod', '--allow-sync-ptr=true']
[ipatests.pytest_ipa.integration.host.Host.master.cmd37] ipa: ERROR: no modifications to be performed
[ipatests.pytest_ipa.integration.host.Host.master.cmd37] Exit code: 1
[ipatests.pytest_ipa.integration] Adding replica1.ipa.test:/var/log/ipaclient-install.log to list of logs to collect
[ipatests.pytest_ipa.integration.host.Host.replica1.ParamikoTransport] RUN ['true']
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd14] RUN ['true']
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd14] Exit code: 0
[ipatests.pytest_ipa.integration] Adding replica1.ipa.test:/ipatests/env.sh to list of logs to collect
[ipatests.pytest_ipa.integration.host.Host.replica1.ParamikoTransport] STAT /ipatests
[ipatests.pytest_ipa.integration.host.Host.replica1.ParamikoTransport] WRITE /ipatests/env.sh
[ipatests.pytest_ipa.integration.host.Host.replica1.ParamikoTransport] STAT /etc/hostname
[ipatests.pytest_ipa.integration.host.Host.replica1.ParamikoTransport] STAT /ipatests/file_backup/etc
[ipatests.pytest_ipa.integration.host.Host.replica1.ParamikoTransport] RUN ['cp', '-af', '/etc/hostname', '/ipatests/file_backup/etc/hostname']
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd15] RUN ['cp', '-af', '/etc/hostname', '/ipatests/file_backup/etc/hostname']
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd15] Exit code: 0
[ipatests.pytest_ipa.integration.host.Host.replica1.ParamikoTransport] WRITE /etc/hostname
[ipatests.pytest_ipa.integration.host.Host.replica1.ParamikoTransport] RUN ['hostname', 'replica1.ipa.test']
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd16] RUN ['hostname', 'replica1.ipa.test']
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd16] Exit code: 0
[ipatests.pytest_ipa.integration.host.Host.replica1.ParamikoTransport] RUN hostname > '/ipatests/backup_hostname'
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd17] RUN hostname > '/ipatests/backup_hostname'
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd17] Exit code: 0
[ipatests.pytest_ipa.integration.host.Host.master.ParamikoTransport] RUN ['kinit', 'admin']
[ipatests.pytest_ipa.integration.host.Host.master.cmd38] RUN ['kinit', 'admin']
[ipatests.pytest_ipa.integration.host.Host.master.cmd38] Password for admin@IPA.TEST: 
[ipatests.pytest_ipa.integration.host.Host.master.cmd38] Exit code: 0
[ipatests.pytest_ipa.integration.host.Host.master.ParamikoTransport] RUN ['ipa', 'dnsconfig-mod', '--allow-sync-ptr=true']
[ipatests.pytest_ipa.integration.host.Host.master.cmd39] RUN ['ipa', 'dnsconfig-mod', '--allow-sync-ptr=true']
[ipatests.pytest_ipa.integration.host.Host.master.cmd39] ipa: ERROR: no modifications to be performed
[ipatests.pytest_ipa.integration.host.Host.master.cmd39] Exit code: 1
[ipatests.pytest_ipa.integration.host.Host.master.ParamikoTransport] RUN ['ipa', 'dnszone-add', '122.168.192.in-addr.arpa.']
[ipatests.pytest_ipa.integration.host.Host.master.cmd40] RUN ['ipa', 'dnszone-add', '122.168.192.in-addr.arpa.']
[ipatests.pytest_ipa.integration.host.Host.master.cmd40] ipa: ERROR: DNS zone with name "122.168.192.in-addr.arpa." already exists
[ipatests.pytest_ipa.integration.host.Host.master.cmd40] Exit code: 1
ipa: WARNING: ipa: ERROR: DNS zone with name "122.168.192.in-addr.arpa." already exists


[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29] Update in progress, 4 seconds elapsed
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29] Update succeeded
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29] 
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [30/42]: prevent time skew after initial replication
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [31/42]: adding sasl mappings to the directory
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [32/42]: updating schema
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [33/42]: setting Auto Member configuration
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [34/42]: enabling S4U2Proxy delegation
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [35/42]: initializing group membership
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [36/42]: adding master entry
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [37/42]: initializing domain level
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [38/42]: configuring Posix uid/gid generation
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [39/42]: adding replication acis
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [40/42]: activating sidgen plugin
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [41/42]: activating extdom plugin
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [42/42]: configuring directory to start on boot
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29] Done configuring directory server (dirsrv).
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29] Configuring Kerberos KDC (krb5kdc)
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [1/5]: configuring KDC
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [2/5]: adding the password extension to the directory
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [3/5]: creating anonymous principal
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [4/5]: starting the KDC
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [5/5]: configuring KDC to start on boot
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29] Done configuring Kerberos KDC (krb5kdc).
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29] Configuring kadmin
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [1/2]: starting kadmin 
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [2/2]: configuring kadmin to start on boot
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29] Done configuring kadmin.
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29] Configuring directory server (dirsrv)
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [1/3]: configuring TLS for DS instance
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [2/3]: importing CA certificates from LDAP
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [3/3]: restarting directory server
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29] Done configuring directory server (dirsrv).
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29] Configuring the web interface (httpd)
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [1/22]: stopping httpd
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [2/22]: setting mod_nss port to 443
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [3/22]: setting mod_nss cipher suite
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [4/22]: setting mod_nss protocol list to TLSv1.2
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [5/22]: setting mod_nss password file
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [6/22]: enabling mod_nss renegotiate
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [7/22]: disabling mod_nss OCSP
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [8/22]: adding URL rewriting rules
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [9/22]: configuring httpd
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [10/22]: setting up httpd keytab
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [11/22]: configuring Gssproxy
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [12/22]: setting up ssl
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [13/22]: configure certmonger for renewals
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [14/22]: importing CA certificates from LDAP
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [15/22]: publish CA cert
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [16/22]: clean up any existing httpd ccaches
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [17/22]: configuring SELinux for httpd
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [18/22]: create KDC proxy config
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [19/22]: enable KDC proxy
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [20/22]: starting httpd
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [21/22]: configuring httpd to start on boot
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [22/22]: enabling oddjobd
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29] Done configuring the web interface (httpd).
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29] Configuring ipa-otpd
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [1/2]: starting ipa-otpd 
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [2/2]: configuring ipa-otpd to start on boot
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29] Done configuring ipa-otpd.
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29] Configuring ipa-custodia
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [1/4]: Generating ipa-custodia config file
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [2/4]: Generating ipa-custodia keys
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29]   [error] SERVER_DOWN: {'desc': "Can't contact LDAP server", 'errno': 107, 'info': 'Transport endpoint is not connected'}
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29] Your system may be partly configured.
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29] Run /usr/sbin/ipa-server-install --uninstall to clean up.
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29] 
[ipatests.pytest_ipa.integration.host.Host.replica1.cmd29] Exit code: 1
ipa: ERROR: stderr: ipaserver.install.server.replicainstall: ERROR    Could not resolve hostname replica1.ipa.test using DNS. Clients may not function properly. Please check your DNS setup. (Note that this check queries IPA DNS directly and ignores /etc/hosts.)
ipapython.admintool: ERROR    {'desc': "Can't contact LDAP server", 'errno': 107, 'info': 'Transport endpoint is not connected'}
ipapython.admintool: ERROR    The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

PR405, logs


The logs end with “OSError: [Errno 113] No route to host” and “paramiko.ssh_exception.SSHException: SSH session not active” in the middle of the installation phase. DS logs on master show “Unable to receive the response for a startReplication extended operation to consumer (Can't contact LDAP server).” Logs were not collected on replica2 due to the paramiko error above. I think this is a infra/network failure and nothing else (but could be wrong).

I support @fcami's assumption, the analysis sounds reasonable. I also ran that several times while testing another issue as a sanity check: https://github.com/freeipa-pr-ci2/freeipa/pull/442.

The test had 10+ green runs, issue happened only once.
It was probably an infrastucture issue, closing.

Metadata Update from @frenaud:
- Issue close_status updated to: worksforme
- Issue status updated to: Closed (was: Open)

17 days ago

Login to comment on this ticket.

Metadata