#8666 Nightly test failure (fed33) in test_simple_replication.py::TestSimpleReplication::test_replica_manage
Closed: fixed 3 years ago by frenaud. Opened 3 years ago by frenaud.

The nightly test test_simple_replication.py::TestSimpleReplication::test_replica_manage is failing in fedora 33. See PR #647 with the following logs and report:

self = <ipatests.test_integration.test_simple_replication.TestSimpleReplication object at 0x7f80a27b0520>

    def test_replica_manage(self):
        """Test ipa-replica-manage list

        Ensure that ipa-replica-manage list -v <node> does not print
        last init status: None
        last init ended: 1970-01-01 00:00:00+00:00
        when the node never had any total update.
        Test for ticket 7716.
        msg1 = "last init ended: 1970-01-01 00:00:00+00:00"
        msg2 = "last init status: None"
>       result = self.master.run_command(
            ["ipa-replica-manage", "list", "-v", self.replicas[0].hostname])

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <ipatests.pytest_ipa.integration.host.Host master.ipa.test (master)>
argv = ['ipa-replica-manage', 'list', '-v', 'replica0.ipa.test'], set_env = True
stdin_text = None, log_stdout = True, raiseonerr = True, cwd = None, bg = False
encoding = 'utf-8', ok_returncode = 0

    def run_command(self, argv, set_env=True, stdin_text=None,
                    log_stdout=True, raiseonerr=True,
                    cwd=None, bg=False, encoding='utf-8', ok_returncode=0):
        """Wrapper around run_command to log stderr on raiseonerr=True

        :param ok_returncode: return code considered to be correct,
                              you can pass an integer or sequence of integers
        result = super().run_command(
            argv, set_env=set_env, stdin_text=stdin_text,
            log_stdout=log_stdout, raiseonerr=False, cwd=cwd, bg=bg,
        # in FIPS mode SSH may print noise to stderr, remove the string
        # "FIPS mode initialized" + optional newline.
        result.stderr_bytes = FIPS_NOISE_RE.sub(b'', result.stderr_bytes)
            result_ok = result.returncode in ok_returncode
        except TypeError:
            result_ok = result.returncode == ok_returncode
        if not result_ok and raiseonerr:
            result.log.error('stderr: %s', result.stderr_text)
>           raise subprocess.CalledProcessError(
                result.returncode, argv,
                result.stdout_text, result.stderr_text
E           subprocess.CalledProcessError: Command '['ipa-replica-manage', 'list', '-v', 'replica0.ipa.test']' returned non-zero exit status 1.

pytest_ipa/integration/host.py:200: CalledProcessError
 ------------------------------Captured stderr call------------------------------ 
ipa: ERROR: stderr: Unknown host replica0.ipa.test: Host 'replica0.ipa.test' does not have corresponding DNS A/AAAA record

Metadata Update from @cheimes:
- Custom field blocking adjusted to 8544
- Issue marked as blocking: #8544

3 years ago

I added a DNS lookup before the failing test. Turns out master cannot resolve the replica hostname:

:transport.py:513 RUN ['dig', 'replica0.ipa.test']
:transport.py:557 ; <<>> DiG 9.11.27-RedHat-9.11.27-1.fc33 <<>> replica0.ipa.test
:transport.py:557 ;; global options: +cmd
:transport.py:557 ;; Got answer:
:transport.py:557 ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 27926
:transport.py:557 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
:transport.py:557 ;; OPT PSEUDOSECTION:
:transport.py:557 ; EDNS: version: 0, flags:; udp: 1232
:transport.py:557 ; COOKIE: fcc42f05843bef4827c08aad601801dfbd1377485560a286 (good)
:transport.py:557 ;; QUESTION SECTION:
:transport.py:557 ;replica0.ipa.test.       IN  A
:transport.py:557 ;; AUTHORITY SECTION:
:transport.py:557 ipa.test.     1   IN  SOA master.ipa.test. hostmaster.ipa.test. 1612186046 3600 900 1209600 3600
:transport.py:557 ;; Query time: 0 msec
:transport.py:557 ;; SERVER:
:transport.py:557 ;; WHEN: Mon Feb 01 13:27:59 UTC 2021
:transport.py:557 ;; MSG SIZE  rcvd: 128
:transport.py:217 Exit code: 0
:transport.py:391 RUN ['ipa-replica-manage', 'list', '-v', 'replica0.ipa.test']
:transport.py:513 RUN ['ipa-replica-manage', 'list', '-v', 'replica0.ipa.test']
:transport.py:557 Unknown host replica0.ipa.test: Host 'replica0.ipa.test' does not have corresponding DNS A/AAAA record
:transport.py:217 Exit code: 1

The root cause has been identified. PRCI is creating /etc/hosts file on the master/replica/client machines with a line for each test machine.
In fedora33, systemd-resolved is used and "dig@ client.ipa.test" on the client.ipa.test machine is able to answer (because systemd-resolved is configured by default with ReadEtcHosts=true, it also gets answers from the local /etc/hosts content).
As a consequence, ipa-client-install is tricked into believing that an A/AAAA record is already defined for the client machine and does not create the record (see https://github.com/freeipa/freeipa/blob/3a584803da697b80d49590223be632a7451e1b7e/ipaclient/install/client.py#L1564).

On fedora32, the client installer correctly sees that no DNS record exists for the client and adds the record in IPA DNS. That explains the difference between fedora32 and fodora33.

Fixed on the latest nightlies:
- testing_master_previous: PR 780, Report
- testing_master_latest: PR 779, Report
- testing_ipa-4-9_latest: PR 771, Report
- testing_ipa-4-9_previous: PR 773, Report
- testing_ipa-4.6: PR 776, Report

thanks to PR https://github.com/freeipa/freeipa/pull/5474 DNS resolvers management

Metadata Update from @frenaud:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

3 years ago

Log in to comment on this ticket.