I run the IPA test suite ( base + xmlrpc ) against the latest pytest (by now 4.6.3). Build is available here ( https://dev.azure.com/slev0400/slev/_build/results?buildId=161 ) There are many errors ( logs are attached ):
pytest
4.6.3
testsuite errors="41" failures="9" name="pytest" skipped="13" tests="2341"
Though the same test suite passes well against pytest 3.9.3).
Version bisecting points me to https://github.com/pytest-dev/pytest/commit/0f918b1a9dd14a2d046d19be6ec239e8e2df4cb2 :
3094: Classic xunit-style functions and methods now obey the scope of autouse fixtures. Now the xunit-style functions are integrated with the fixture mechanism and obey the proper scope rules of fixtures involved in the call.
3094: Classic xunit-style functions and methods now obey the scope of autouse fixtures.
Now the xunit-style functions are integrated with the fixture mechanism and obey the proper scope rules of fixtures involved in the call.
That introduced in pytest 4.2 new hidden (internal only) fixtures:
which are auto-injected on the corresponding levels. How does this feature affect the IPA test suite?
Let's take a look at one of the errors.
ipa-run-tests test_xmlrpc/test_sudocmd_plugin.py::TestSudoCmd
==================================== ERRORS ==================================== ___________ ERROR at teardown of TestSudoCmd.test_update_and_verify ____________ tp = <class 'AttributeError'>, value = None, tb = None def reraise(tp, value, tb=None): try: if value is None: value = tp() if value.__traceback__ is not tb: raise value.with_traceback(tb) > raise value root/.local/lib/python3.7/site-packages/six.py:693: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ root/.local/lib/python3.7/site-packages/six.py:693: in reraise raise value root/.local/lib/python3.7/site-packages/six.py:693: in reraise raise value usr/lib/python3.7/site-packages/ipatests/test_xmlrpc/tracker/base.py:253: in cleanup del_command() usr/lib/python3.7/site-packages/ipatests/test_xmlrpc/tracker/base.py:71: in run_command result = cmd(*args, **options) usr/lib/python3.7/site-packages/ipalib/frontend.py:450: in __call__ return self.__do_call(*args, **options) usr/lib/python3.7/site-packages/ipalib/frontend.py:478: in __do_call ret = self.run(*args, **options) usr/lib/python3.7/site-packages/ipalib/frontend.py:801: in run return self.forward(*args, **options) usr/lib/python3.7/site-packages/ipalib/frontend.py:824: in forward *args, **kw) usr/lib/python3.7/site-packages/ipalib/rpc.py:1139: in forward command = getattr(self.conn, name) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = ipaclient.plugins.rpcclient.rpcclient() def __get_conn(self): """ Return thread-local connection. """ if not hasattr(context, self.id): raise AttributeError( "{0} is not connected ({1} in {2})".format( self.name, self.id, > threading.currentThread().getName() ) ) E AttributeError: rpcclient is not connected (rpcclient_140106773738160 in MainThread) usr/lib/python3.7/site-packages/ipalib/backend.py:108: AttributeError ----------------------------- Captured stdout call ----------------------------- Ran command: ipaserver.plugins.sudocmd.sudocmd_mod()('/usr/bin/sudotestcmd1', description='Updated sudo command 1', version='2.232'): OK Ran command: ipaserver.plugins.sudocmd.sudocmd_show()('/usr/bin/sudotestcmd1', all=False, version='2.232'): OK --------------------------- Captured stdout teardown --------------------------- Ran command: ipaserver.plugins.sudocmd.sudocmd_del()('/usr/bin/sudotestcmd1', version='2.232'): OK Ran command: ipaserver.plugins.sudocmd.sudocmd_del()('/usr/bin/sudoTestCmd1', version='2.232'): AttributeError: rpcclient is not connected (rpcclient_140106773738160 in MainThread) ====================== 5 passed, 1 error in 4.09 seconds ======================
So, an error happens in teardown code. Let's look at setup plan:
teardown
ipa-run-tests test_xmlrpc/test_sudocmd_plugin.py::TestSudoCmd --setup-only usr/lib/python3.7/site-packages/ipatests/test_xmlrpc/test_sudocmd_plugin.py SETUP C _Class__pytest_setup_class SETUP C sudocmd1 SETUP C sudocmd2 usr/lib/python3.7/site-packages/ipatests/test_xmlrpc/test_sudocmd_plugin.py::TestSudoCmd::test_create (fixtures used: _Class__pytest_setup_class, sudocmd1, sudocmd2) usr/lib/python3.7/site-packages/ipatests/test_xmlrpc/test_sudocmd_plugin.py::TestSudoCmd::test_create_duplicates (fixtures used: _Class__pytest_setup_class, sudocmd1, sudocmd2) usr/lib/python3.7/site-packages/ipatests/test_xmlrpc/test_sudocmd_plugin.py::TestSudoCmd::test_retrieve (fixtures used: _Class__pytest_setup_class, sudocmd1) usr/lib/python3.7/site-packages/ipatests/test_xmlrpc/test_sudocmd_plugin.py::TestSudoCmd::test_search (fixtures used: _Class__pytest_setup_class, sudocmd1, sudocmd2) usr/lib/python3.7/site-packages/ipatests/test_xmlrpc/test_sudocmd_plugin.py::TestSudoCmd::test_update_and_verify (fixtures used: _Class__pytest_setup_class, sudocmd1) TEARDOWN C sudocmd1 TEARDOWN C _Class__pytest_setup_class TEARDOWN C sudocmd2E =========================================== ERRORS ============================================ ___________________ ERROR at teardown of TestSudoCmd.test_update_and_verify ___________________ ...
SETUPs are OK, but TEARDOWN C _Class__pytest_setup_class should follow TEARDOWN C sudocmd2. The cause is that teardown_ functions are now usual fixtures and follow corresponding rules:
SETUP
TEARDOWN C _Class__pytest_setup_class
TEARDOWN C sudocmd2
teardown_
Within a function request for features, fixture of higher-scopes (such as session) are instantiated first than lower-scoped fixtures (such as function or class). The relative order of fixtures of same scope follows the declared order in the test function and honours dependencies between fixtures.
Thus, there is only one legal way to arrange the fixtures within one scope (level) in a strict order - the specification for fixtures interdependencies. This means that all the class fixtures should depend on class setup/teardown ones. But the latter is hidden from usage. Hence, all the setup_class/teardown_class methods should be replaced with the corresponding auto fixtures and should be linked from the other class fixtures.
the specification for fixtures interdependencies
setup/teardown
setup_class/teardown_class
The overall problem is mixing of idioms of different test frameworks (nose, unittest, pytest) in the test suite. Yes, pytest supports all of them, but doesn't provide the full power:
https://docs.pytest.org/en/latest/nose.html https://docs.pytest.org/en/latest/unittest.html
My proposal is to have only one style for test writing - pytest. Other ones should be refactored into the latter. The implementation plan might be as follows:
Replace
def setup_class(cls) def setup(self)
with
@pytest.fixture(autouse=True, scope="class") @pytest.fixture(autouse=True) def somefixture_setup(self, request)
def teardown_class(cls) def teardown(self) def teardown_method(self, method)
def fin(): ... request.addfinalizer(fin)
This part is ready to preview: https://github.com/stanislavlevin/freeipa/tree/pytest4_errors_xmlrpc_run_pytest4 https://dev.azure.com/slev0400/slev/_build/results?buildId=163
def setUp(self) def setUpClass(cls)
@pytest.fixture(autouse=True) @pytest.fixture(autouse=True, scope="class") def somefixture_setup(self, request)
def tearDown(self) def tearDownClass(cls)
Add corresponding dependencies of base class fixtures
Replace unittest asserts with analogues
So, my questions are: Does anybody need it? Does anybody need pytest4 compatibility? Thanks in advance!
Metadata Update from @abbra: - Issue set to the milestone: FreeIPA 4.8.1
@slev the proposal sounds good to me.
@abbra, thanks for the feedback. I'm going to resume work on it after vacation (next two weeks).
Hello @slev , the PKI team runs the IPA test suite in our nightly test [1] as well as per commit tests. Your proposal will help fix our CI environment. Thank you
[1] https://travis-ci.org/SilleBille/pki-nightly-test/jobs/596143907
Hi @dmoluguw, I'm sorry I broke that promise. I would like to get a second chance.
Metadata Update from @pcech: - Issue tagged with: Raven
https://github.com/freeipa/freeipa/pull/3783
Login to comment on this ticket.