Multiple nightly tests failed in [testing_master_rawhide] in a call to tasks.configure_dns_for_trust. See PR #377: - test_idviews: report - test_ipahealthcheck_adtrust: report - test_sssd: report - test_trust: report - test_winsyncmigrate: report
tasks.configure_dns_for_trust
test_idviews
test_ipahealthcheck_adtrust
test_sssd
test_trust
test_winsyncmigrate
All the tests have similar logs:
cls = <class 'ipatests.test_integration.test_idviews.TestCertsInIDOverrides'> mh = <pytest_multihost.plugin.MultihostFixture object at 0x7f4828867970> @classmethod def install(cls, mh): super(TestCertsInIDOverrides, cls).install(mh) cls.ad = config.ad_domains[0].ads[0] cls.ad_domain = cls.ad.domain.name cls.aduser = "testuser@%s" % cls.ad_domain master = cls.master # A setup for test_dbus_user_lookup master.run_command(['dnf', 'install', '-y', 'sssd-dbus'], raiseonerr=False) master.run_command( "sed -i 's/= 7/= 0xFFF0/' %s" % paths.SSSD_CONF, raiseonerr=False) with tasks.remote_sssd_config(master) as sssd_config: try: sssd_config.new_service('ifp') except ServiceAlreadyExists: pass sssd_config.activate_service('ifp') master.run_command(['systemctl', 'restart', 'sssd.service']) # End of setup for test_dbus_user_lookup # AD-related stuff tasks.install_adtrust(master) tasks.sync_time(master, cls.ad) > tasks.configure_dns_for_trust(master, cls.ad) test_integration/test_idviews.py:59: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ pytest_ipa/integration/tasks.py:600: in configure_dns_for_trust master.run_command(['ipa', 'dnsforwardzone-add', ad.domain.name, _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <ipatests.pytest_ipa.integration.host.Host master.ipa.test (master)> argv = ['ipa', 'dnsforwardzone-add', 'ad.test', '--forwarder', '192.168.122.126', '--forward-policy', ...] set_env = True, stdin_text = None, log_stdout = True, raiseonerr = True cwd = None, bg = False, encoding = 'utf-8', ok_returncode = 0 def run_command(self, argv, set_env=True, stdin_text=None, log_stdout=True, raiseonerr=True, cwd=None, bg=False, encoding='utf-8', ok_returncode=0): """Wrapper around run_command to log stderr on raiseonerr=True :param ok_returncode: return code considered to be correct, you can pass an integer or sequence of integers """ result = super().run_command( argv, set_env=set_env, stdin_text=stdin_text, log_stdout=log_stdout, raiseonerr=False, cwd=cwd, bg=bg, encoding=encoding ) # in FIPS mode SSH may print noise to stderr, remove the string # "FIPS mode initialized" + optional newline. result.stderr_bytes = FIPS_NOISE_RE.sub(b'', result.stderr_bytes) try: result_ok = result.returncode in ok_returncode except TypeError: result_ok = result.returncode == ok_returncode if not result_ok and raiseonerr: result.log.error('stderr: %s', result.stderr_text) > raise subprocess.CalledProcessError( result.returncode, argv, result.stdout_text, result.stderr_text ) E subprocess.CalledProcessError: Command '['ipa', 'dnsforwardzone-add', 'ad.test', '--forwarder', '192.168.122.126', '--forward-policy', 'only']' returned non-zero exit status 1. pytest_ipa/integration/host.py:200: CalledProcessError -----------------------------Captured stderr setup------------------------------ ipa: ERROR: stderr: ipa: ERROR: an internal error has occurred
and httpd/error_log:
ipa: INFO: Checking DNS domain ad.test., please wait ... ipa: ERROR: non-public: KeyError: 0 Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/ipaserver/rpcserver.py", line 395, in wsgi_execute result = command(*args, **options) File "/usr/lib/python3.9/site-packages/ipalib/frontend.py", line 471, in __call__ return self.__do_call(*args, **options) File "/usr/lib/python3.9/site-packages/ipalib/frontend.py", line 499, in __do_call ret = self.run(*args, **options) File "/usr/lib/python3.9/site-packages/ipalib/frontend.py", line 821, in run return self.execute(*args, **options) File "/usr/lib/python3.9/site-packages/ipaserver/plugins/dns.py", line 4364, in execute self.obj._warning_if_forwarders_do_not_work( File "/usr/lib/python3.9/site-packages/ipaserver/plugins/dns.py", line 4310, in _warning_if_forwarders_do_not_work ipa_dns_ip = str(ans.rrset.items[0]) KeyError: 0 ipa: INFO: [jsonserver_kerb] admin@IPA.TEST: dnsforwardzone_add/1('ad.test', idnsforwarders=('192.168.122.126',), idnsforwardpolicy='only', version='2.239'): InternalError
It looks like there is a race condition between bind initialization and the time we do dnsforwardzone_add call. At least, http://freeipa-org-pr-ci.s3-website.eu-central-1.amazonaws.com/jobs/a122de4c-ea4e-11ea-b6b3-fa163e8ac44c/test_integration-test_trust.py-TestTrust-test_trustdomains_found_in_nonposix_trust/master.ipa.test/var/log/httpd/error_log.gz shows that the call happens at 00:37:08.120020 and in journal we can see bind telling us:
dnsforwardzone_add
Aug 30 00:37:08 master.ipa.test named-pkcs11[31449]: forward zone 'ad.test': loaded
Also happened in PR #389, [testing_master_rawhide]
The issue is coming from dnspython 2.0 (rawhide ships python3-dns 2.0.0-1, while fedora 32 ships python3-dns 1.16.0). With dnspython 1.16: ans.rrset.items is a list With dnspython 2.0: ans.rrset.items is a dict
Metadata Update from @frenaud: - Issue assigned to frenaud
Metadata Update from @frenaud: - Custom field on_review adjusted to https://github.com/freeipa/freeipa/pull/5085
master:
ipa-4-8:
Metadata Update from @frenaud: - Issue close_status updated to: fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.