There is a problem with calculating nodeids by ipa-run-tests.
ipa-run-tests
About nodeid:
Each test is assigned a unique nodeid which is rooted at the rootdir and takes in account full path, class name, function name and parametrization (if any).
For example,
test_mod.py::TestClass::test_method
About rootdir:
pytest determines a rootdir for each test run which depends on the command line arguments (specified test files, paths) and on the existence of ini-files. The determined rootdir and ini-file are printed as part of the pytest header during startup. Here’s a summary what pytest uses rootdir for: Construct nodeids during collection; each test is assigned a unique nodeid which is rooted at the rootdir and takes in account full path, class name, function name and parametrization (if any). Is used by plugins as a stable location to store project/test run specific information; for example, the internal cache plugin creates a .pytest_cache subdirectory in rootdir to store its cross-test run state. The rootdir is used as a reference directory for constructing test addresses (“nodeids”) and can be used also by plugins for storing per-testrun information.
pytest determines a rootdir for each test run which depends on the command line arguments (specified test files, paths) and on the existence of ini-files. The determined rootdir and ini-file are printed as part of the pytest header during startup.
Here’s a summary what pytest uses rootdir for:
The rootdir is used as a reference directory for constructing test addresses (“nodeids”) and can be used also by plugins for storing per-testrun information.
Let's take a look at running of
ipa-run-tests test_cmdline -s
and setting of corresponding breakpoints in
/usr/lib/python3.7/site-packages/_pytest/config/findpaths.py(105)determine_setup() /usr/lib/python3.7/site-packages/ipatests/conftest.py(132)pytest_collection_modifyitems()
a) cwd - '/freeipa'
> /usr/lib/python3.7/site-packages/_pytest/config/findpaths.py(105)determine_setup() -> dirs = get_dirs_from_args(args) (Pdb) p args ['test_cmdline', '--with-xunit'] <= not mangled paths
Paths come as they were passed on command line. Go ahead.
(Pdb) return -> return rootdir, inifile, inicfg or {} (Pdb) rootdir local('/freeipa')
The rootdir is set here and then will not be changed.
/usr/lib/python3.7/site-packages/ipatests/conftest.py(132)pytest_collection_modifyitems()->None (Pdb) config.rootdir local('/freeipa') (Pdb) items[0].nodeid 'test_cmdline/test_cli.py::TestCLIParsing::()::test_ping'
b) cwd - '/'
> /usr/lib/python3.7/site-packages/ipatests/conftest.py(132)pytest_collection_modifyitems()->None -> breakpoint() (Pdb) config.rootdir local('/') (Pdb) items[0].nodeid 'usr/lib/python3.7/site-packages/ipatests/test_cmdline/test_cli.py::TestCLIParsing::()::test_ping'
So, we have different modules paths:
/freeipa + test_cmdline/test_cli.py / + usr/lib/python3.7/site-packages/ipatests/test_cmdline/test_cli.py
For pytest 4 this is even more worse:
/freeipa + test_cli.py / + usr/lib/python3.7/site-packages/ipatests/test_cmdline/test_cli.py
This behaviour is legitimate, but not suitable:
Finding the rootdir Here is the algorithm which finds the rootdir from args: determine the common ancestor directory for the specified args that are recognised as paths that exist in the file system. If no such paths are found, the common ancestor directory is set to the current working directory.
Finding the rootdir
Here is the algorithm which finds the rootdir from args:
Some of disadvantages of current implementation of ipa-run-tests script: 1) the stability of nodeids outcome is broken, because it depends on current working directory. 2) pytest_load_initial_conftests hook (as it was shown above) is not intended for changing paths in current way 3) not all of the Unix glob patterns are supported 4) it needs to mangle paths rather than using the default conventions for Python test discovery (which is more stable)
pytest_load_initial_conftests
I suggest to use a chdir to ipatests module directory and then execv (sort of pre https://github.com/freeipa/freeipa/commit/93c158b05812b55383043e320cac89550993b8b2). The motivation of that change says about 2 problems at least. New "old" implementation don't have them (checked against pytest 3.9.3 and 4.6.2): 1) collect and run by nodeid is working
ipa-run-tests test_cmdline/test_ipagetkeytab.py::test_ipagetkeytab::test_1_run ... ============================ 1 passed ...
2) coverage report (pytest-cov) is the same as for bare pytest
ipa-run-tests --cov=ipatests.test_cmdline test_cmdline ... Name Stmts Miss Cover ------------------------------------------------------- test_cmdline/__init__.py 2 0 100% test_cmdline/cmdline.py 30 12 60% test_cmdline/test_cli.py 153 108 29% test_cmdline/test_help.py 98 75 23% test_cmdline/test_ipagetkeytab.py 226 160 29% ------------------------------------------------------- TOTAL 509 355 30%
Today I will open the corresponding PR.
master:
Metadata Update from @abbra: - Issue set to the milestone: FreeIPA 4.8.1
ipa-4-8:
Metadata Update from @abbra: - Issue close_status updated to: fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.