TestCASpecificRUVs.test_replica_uninstall_deletes_ruvs start failing with assertion error :
Error Message
AssertionError: Replica RUVs were not clean during replica uninstallation assert 'vm-171-238....q.redhat.com' not in 'Replica Upda...com:389: 6\n' 'vm-171-238.abc.idm....eng.brq.redhat.com' is contained here: Replica Update Vectors: vm-171-229.abc.idm.lab.eng.brq.redhat.com:389: 4 vm-069.abc.idm.lab.eng.brq.redhat.com:389: 3 vm-171-238.abc.idm.lab.eng.brq.redhat.com:389: 7 Certificate Server Replica Update Vectors: vm-171-229.abc.idm.lab.eng.brq.redhat.com:389: 6
Stacktrace
self = <ipatests.test_integration.test_topology.TestCASpecificRUVs object at 0x7fd880058668> def test_replica_uninstall_deletes_ruvs(self): """ http://www.freeipa.org/page/V4/Manage_replication_topology_4_4/Test_Plan #Test_case:_.2A-ruv_subcommands_of_ipa-replica-manage_are_extended _to_handle_CA-specific_RUVs """ master = self.master replica = self.replicas[1] res1 = master.run_command(['ipa-replica-manage', 'list-ruv', '-p', master.config.dirman_password]).stdout_text assert(res1.count(replica.hostname) == 2), ( "Did not find proper number of replica hostname (%s) occurrencies" " in the command output: %s" % (replica.hostname, res1)) master.run_command(['ipa-replica-manage', 'del', replica.hostname, '-p', master.config.dirman_password]) tasks.uninstall_master(replica) res2 = master.run_command(['ipa-replica-manage', 'list-ruv', '-p', master.config.dirman_password]).stdout_text > assert(replica.hostname not in res2), ( "Replica RUVs were not clean during replica uninstallation") E AssertionError: Replica RUVs were not clean during replica uninstallation E assert 'vm-171-238....q.redhat.com' not in 'Replica Upda...com:389: 6\n' E 'vm-171-238.abc.idm....eng.brq.redhat.com' is contained here: E Replica Update Vectors: E vm-171-229.abc.idm.lab.eng.brq.redhat.com:389: 4 E vm-069.abc.idm.lab.eng.brq.redhat.com:389: 3 E vm-171-238.abc.idm.lab.eng.brq.redhat.com:389: 7 E Certificate Server Replica Update Vectors: E vm-171-229.abc.idm.lab.eng.brq.redhat.com:389: 6 /usr/lib/python3.6/site-packages/ipatests/test_integration/test_topology.py:248: AssertionError
Test start failing potentially after commit 8a8b641
Metadata Update from @gkaihoro: - Issue tagged with: test-failure, tests
<img alt="test_logs.zip" src="/freeipa/issue/raw/files/4734e878c4e44b5de5dd887514c4962a2b061355d8585e8302580cffa11759f1-test_logs.zip" />
Metadata Update from @fbarreto: - Issue assigned to fbarreto
Metadata Update from @slaykovsky: - Issue priority set to: normal - Issue set to the milestone: FreeIPA 4.6.4
test_topology.py is green on PR CI (check nightly test PR https://github.com/freeipa/freeipa/pull/1921) and here https://fedorapeople.org/groups/freeipa/prci/jobs/ce459b9c-579f-11e8-93b8-fa163e15c266/.
test_topology.py
I'm still uploading the new vagrant box (updated f27). Once it's done I will rerun the test with it.
Metadata Update from @rcritten: - Issue set to the milestone: FreeIPA 4.6.5 (was: FreeIPA 4.6.4)
Failure still happens randomly, for instance see test output from 2018-10-30
The issue happens sometimes, when the master is too slow to clean the ruv. For reference, the test scenario is: 1. ipa-replica-manage del $replica 2. (on replica) ipa-server-install --uninstall -U 3. ipa-replica-manage list-ruv to check that the replica is not present any more.
ipa-replica-manage del $replica
ipa-server-install --uninstall -U
ipa-replica-manage list-ruv
ipa-replica-manage del is removing the host entry, which in turns triggers the topology plugin. The plugin creates clean-ruv tasks (one for the domain, one for o=ipaca if CA is installed on the replica). 389-ds tasks are asynchronous, meaning that ipa-replica-manage del command may return before the task has finished. As the test does not check the task status, it may also call ipa-replica-manage list-ruv before the clean-ruv task has finished. We should update the test so that it checks if the task has finished, waiting for up to a configured timeout, and consider that the test failed if the task did not finish or if list-ruv still displays the replica after the task has finished.
ipa-replica-manage del
Metadata Update from @frenaud: - Custom field on_review adjusted to https://github.com/freeipa/freeipa/pull/2574 - Issue assigned to frenaud (was: fbarreto)
master:
ipa-4-7:
ipa-4-6:
Metadata Update from @frenaud: - Issue close_status updated to: fixed - Issue status updated to: Closed (was: Open)
Same issue is seen in testing_master_latest PR 1653 Report
testing_master_latest
I suspect this might be a regression. Hence reopening this issue.
Metadata Update from @sumedhs: - Issue set to the milestone: None (was: FreeIPA 4.6.5)
Metadata Update from @sumedhs: - Issue status updated to: Open (was: Closed)
Failure was observed in [testing_master_previous] Nightly PR 1654 Report https://github.com/freeipa-pr-ci2/freeipa/pull/1654 Report http://freeipa-org-pr-ci.s3-website.eu-central-1.amazonaws.com/jobs/5b2da426-c049-11ec-814f-fa163e02831a/report.html
Sorry I jumped the gun. The re-run resulted in a pass and the test is passing now. @gkaihoro can you re-run the failing test in !1654 ?
Sure. Re-running
Metadata Update from @sumedhs: - Custom field affects_doc adjusted to on - Custom field knownissue adjusted to on - Issue close_status updated to: fixed - Issue status updated to: Closed (was: Open)
Closing this as fixed since both re-runs in PR 1653 and PR 1654 have passed. Sorry for the noise
http://freeipa-org-pr-ci.s3-website.eu-central-1.amazonaws.com/jobs/e516334e-c0a6-11ec-bffb-fa163ef8383a
Login to comment on this ticket.