#8299 Nightly test failure in test_integration/test_external_ca.py::TestExternalCAInvalidCert::test_external_ca
Closed: worksforme 3 years ago by frenaud. Opened 3 years ago by fcami.

testing_master_previous Nightly PR #4610 failed in test_integration/test_external_ca.py::TestExternalCAInvalidCert::test_external_ca
logs

    def test_external_ca(self):
        # Step 1 of ipa-server-install.
        install_server_external_ca_step1(self.master)

        # Sign CA, transport it to the host and get ipa a root ca paths.
        root_ca_fname, ipa_ca_fname = tasks.sign_ca_and_transport(
            self.master, paths.ROOT_IPA_CSR, ROOT_CA, IPA_CA)

        # Step 2 of ipa-server-install.
        install_server_external_ca_step2(self.master, ipa_ca_fname,
                                         root_ca_fname)

        self.master.run_command([paths.IPA_CACERT_MANAGE, 'renew',
>                                '--external-ca'])

test_integration/test_external_ca.py:386: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <ipatests.pytest_ipa.integration.host.Host master.ipa.test (master)>
argv = ['/usr/sbin/ipa-cacert-manage', 'renew', '--external-ca'], set_env = True
stdin_text = None, log_stdout = True, raiseonerr = True, cwd = None, bg = False
encoding = 'utf-8', ok_returncode = 0

    def run_command(self, argv, set_env=True, stdin_text=None,
                    log_stdout=True, raiseonerr=True,
                    cwd=None, bg=False, encoding='utf-8', ok_returncode=0):
        """Wrapper around run_command to log stderr on raiseonerr=True

        :param ok_returncode: return code considered to be correct,
                              you can pass an integer or sequence of integers
        """
        result = super().run_command(
            argv, set_env=set_env, stdin_text=stdin_text,
            log_stdout=log_stdout, raiseonerr=False, cwd=cwd, bg=bg,
            encoding=encoding
        )
        # in FIPS mode SSH may print noise to stderr, remove the string
        # "FIPS mode initialized" + optional newline.
        result.stderr_bytes = FIPS_NOISE_RE.sub(b'', result.stderr_bytes)
        try:
            result_ok = result.returncode in ok_returncode
        except TypeError:
            result_ok = result.returncode == ok_returncode
        if not result_ok and raiseonerr:
            result.log.error('stderr: %s', result.stderr_text)
            raise subprocess.CalledProcessError(
                result.returncode, argv,
>               result.stdout_text, result.stderr_text
            )
E           subprocess.CalledProcessError: Command '['/usr/sbin/ipa-cacert-manage', 'renew', '--external-ca']' returned non-zero exit status 1.

That's another instance of '['/usr/sbin/ipa-cacert-manage', 'renew', '--external-ca']' failing due to a timeout in testing_master_previous (f30).


In PR #344 we can see the following:
- the lock is taken by ipa-server-guard process with pid 23435 (see renewal.lock)
- This process is started on Aug 13 00:43:51 by certmonger (in journal)
- the timestamp corresponds to a step in ipa-server-install: configure certmonger for renewals (in ipa-server-install.log)

Running in loop getcert modify-ca -c IPA -e '/usr/libexec/certmonger/ipa-server-guard /usr/libexec/certmonger/ipa-submit' can also reproduce the same issue (lock not being released). It is likely a certmonger problem as certmonger seems to send a SIGKILL signal to ipa-server-guard which prevents the process from correctly releasing the lock.
I suspect this is the same issue as in #8211 #8427 and #8425.

The test had 10+ green runs. The issue was probably fixed with certmonger's fix https://pagure.io/certmonger/c/00e948049acf0ca1b61ed9c2b8579b06b4bcb46a?branch=master which made it into certmonger 0.79.12
Issue can be closed

Metadata Update from @frenaud:
- Issue close_status updated to: worksforme
- Issue status updated to: Closed (was: Open)

3 years ago

Login to comment on this ticket.

Metadata