#8481 Nightly test failure in rawhide in tasks.configure_dns_for_trust
Closed: fixed 2 years ago by frenaud. Opened 2 years ago by frenaud.

Issue

Multiple nightly tests failed in [testing_master_rawhide] in a call to tasks.configure_dns_for_trust. See PR #377:
- test_idviews: report
- test_ipahealthcheck_adtrust: report
- test_sssd: report
- test_trust: report
- test_winsyncmigrate: report

All the tests have similar logs:

cls = <class 'ipatests.test_integration.test_idviews.TestCertsInIDOverrides'>
mh = <pytest_multihost.plugin.MultihostFixture object at 0x7f4828867970>

    @classmethod
    def install(cls, mh):
        super(TestCertsInIDOverrides, cls).install(mh)
        cls.ad = config.ad_domains[0].ads[0]
        cls.ad_domain = cls.ad.domain.name
        cls.aduser = "testuser@%s" % cls.ad_domain

        master = cls.master
        # A setup for test_dbus_user_lookup
        master.run_command(['dnf', 'install', '-y', 'sssd-dbus'],
                           raiseonerr=False)
        master.run_command(
            "sed -i 's/= 7/= 0xFFF0/' %s" % paths.SSSD_CONF, raiseonerr=False)
        with tasks.remote_sssd_config(master) as sssd_config:
            try:
                sssd_config.new_service('ifp')
            except ServiceAlreadyExists:
                pass
            sssd_config.activate_service('ifp')

        master.run_command(['systemctl', 'restart', 'sssd.service'])
        # End of setup for test_dbus_user_lookup

        # AD-related stuff
        tasks.install_adtrust(master)
        tasks.sync_time(master, cls.ad)
>       tasks.configure_dns_for_trust(master, cls.ad)

test_integration/test_idviews.py:59: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pytest_ipa/integration/tasks.py:600: in configure_dns_for_trust
    master.run_command(['ipa', 'dnsforwardzone-add', ad.domain.name,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <ipatests.pytest_ipa.integration.host.Host master.ipa.test (master)>
argv = ['ipa', 'dnsforwardzone-add', 'ad.test', '--forwarder', '192.168.122.126', '--forward-policy', ...]
set_env = True, stdin_text = None, log_stdout = True, raiseonerr = True
cwd = None, bg = False, encoding = 'utf-8', ok_returncode = 0

    def run_command(self, argv, set_env=True, stdin_text=None,
                    log_stdout=True, raiseonerr=True,
                    cwd=None, bg=False, encoding='utf-8', ok_returncode=0):
        """Wrapper around run_command to log stderr on raiseonerr=True

        :param ok_returncode: return code considered to be correct,
                              you can pass an integer or sequence of integers
        """
        result = super().run_command(
            argv, set_env=set_env, stdin_text=stdin_text,
            log_stdout=log_stdout, raiseonerr=False, cwd=cwd, bg=bg,
            encoding=encoding
        )
        # in FIPS mode SSH may print noise to stderr, remove the string
        # "FIPS mode initialized" + optional newline.
        result.stderr_bytes = FIPS_NOISE_RE.sub(b'', result.stderr_bytes)
        try:
            result_ok = result.returncode in ok_returncode
        except TypeError:
            result_ok = result.returncode == ok_returncode
        if not result_ok and raiseonerr:
            result.log.error('stderr: %s', result.stderr_text)
>           raise subprocess.CalledProcessError(
                result.returncode, argv,
                result.stdout_text, result.stderr_text
            )
E           subprocess.CalledProcessError: Command '['ipa', 'dnsforwardzone-add', 'ad.test', '--forwarder', '192.168.122.126', '--forward-policy', 'only']' returned non-zero exit status 1.

pytest_ipa/integration/host.py:200: CalledProcessError
 -----------------------------Captured stderr setup------------------------------ 
ipa: ERROR: stderr: ipa: ERROR: an internal error has occurred

and httpd/error_log:

 ipa: INFO: Checking DNS domain ad.test., please wait ...
 ipa: ERROR: non-public: KeyError: 0
 Traceback (most recent call last):
   File "/usr/lib/python3.9/site-packages/ipaserver/rpcserver.py", line 395, in wsgi_execute
     result = command(*args, **options)
   File "/usr/lib/python3.9/site-packages/ipalib/frontend.py", line 471, in __call__
     return self.__do_call(*args, **options)
   File "/usr/lib/python3.9/site-packages/ipalib/frontend.py", line 499, in __do_call
     ret = self.run(*args, **options)
   File "/usr/lib/python3.9/site-packages/ipalib/frontend.py", line 821, in run
     return self.execute(*args, **options)
   File "/usr/lib/python3.9/site-packages/ipaserver/plugins/dns.py", line 4364, in execute
     self.obj._warning_if_forwarders_do_not_work(
   File "/usr/lib/python3.9/site-packages/ipaserver/plugins/dns.py", line 4310, in _warning_if_forwarders_do_not_work
     ipa_dns_ip = str(ans.rrset.items[0])
 KeyError: 0
 ipa: INFO: [jsonserver_kerb] admin@IPA.TEST: dnsforwardzone_add/1('ad.test', idnsforwarders=('192.168.122.126',), idnsforwardpolicy='only', version='2.239'): InternalError

It looks like there is a race condition between bind initialization and the time we do dnsforwardzone_add call. At least, http://freeipa-org-pr-ci.s3-website.eu-central-1.amazonaws.com/jobs/a122de4c-ea4e-11ea-b6b3-fa163e8ac44c/test_integration-test_trust.py-TestTrust-test_trustdomains_found_in_nonposix_trust/master.ipa.test/var/log/httpd/error_log.gz shows that the call happens at 00:37:08.120020 and in journal we can see bind telling us:

Aug 30 00:37:08 master.ipa.test named-pkcs11[31449]: forward zone 'ad.test': loaded

Also happened in PR #389, [testing_master_rawhide]

The issue is coming from dnspython 2.0 (rawhide ships python3-dns 2.0.0-1, while fedora 32 ships python3-dns 1.16.0).
With dnspython 1.16: ans.rrset.items is a list
With dnspython 2.0: ans.rrset.items is a dict

Metadata Update from @frenaud:
- Issue assigned to frenaud

2 years ago

Metadata Update from @frenaud:
- Custom field on_review adjusted to https://github.com/freeipa/freeipa/pull/5085

2 years ago

master:

  • dbc7881 dnsforwardzone-add: support dnspython 2.0

ipa-4-8:

  • fefaeb4 dnsforwardzone-add: support dnspython 2.0

Metadata Update from @frenaud:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata