#8007 Not stable nodeids within pytest
Closed: fixed 4 years ago by abbra. Opened 4 years ago by slev.

There is a problem with calculating nodeids by ipa-run-tests.

About nodeid:

Each test is assigned a unique nodeid which is rooted at the rootdir and takes in account full path, class name, function name and parametrization (if any).

For example,

test_mod.py::TestClass::test_method

About rootdir:

pytest determines a rootdir for each test run which depends on the command line arguments (specified test files, paths) and on the existence of ini-files. The determined rootdir and ini-file are printed as part of the pytest header during startup.

Here’s a summary what pytest uses rootdir for:

  • Construct nodeids during collection; each test is assigned a unique nodeid which is rooted at the rootdir and takes in account full path, class name, function name and parametrization (if any).
  • Is used by plugins as a stable location to store project/test run specific information; for example, the internal cache plugin creates a .pytest_cache subdirectory in rootdir to store its cross-test run state.

The rootdir is used as a reference directory for constructing test addresses (“nodeids”) and can be used also by plugins for storing per-testrun information.

Let's take a look at running of

ipa-run-tests test_cmdline -s

and setting of corresponding breakpoints in

/usr/lib/python3.7/site-packages/_pytest/config/findpaths.py(105)determine_setup()
/usr/lib/python3.7/site-packages/ipatests/conftest.py(132)pytest_collection_modifyitems()

a) cwd - '/freeipa'

> /usr/lib/python3.7/site-packages/_pytest/config/findpaths.py(105)determine_setup()
-> dirs = get_dirs_from_args(args)
(Pdb) p args
['test_cmdline', '--with-xunit']   <= not mangled paths

Paths come as they were passed on command line.
Go ahead.

(Pdb) return
-> return rootdir, inifile, inicfg or {}
(Pdb) rootdir
local('/freeipa')

The rootdir is set here and then will not be changed.

/usr/lib/python3.7/site-packages/ipatests/conftest.py(132)pytest_collection_modifyitems()->None
(Pdb) config.rootdir
local('/freeipa')
(Pdb) items[0].nodeid
'test_cmdline/test_cli.py::TestCLIParsing::()::test_ping'

b) cwd - '/'

> /usr/lib/python3.7/site-packages/ipatests/conftest.py(132)pytest_collection_modifyitems()->None
-> breakpoint()
(Pdb) config.rootdir
local('/')
(Pdb) items[0].nodeid
'usr/lib/python3.7/site-packages/ipatests/test_cmdline/test_cli.py::TestCLIParsing::()::test_ping'

So, we have different modules paths:

/freeipa + test_cmdline/test_cli.py
/ + usr/lib/python3.7/site-packages/ipatests/test_cmdline/test_cli.py

For pytest 4 this is even more worse:

/freeipa + test_cli.py
/ + usr/lib/python3.7/site-packages/ipatests/test_cmdline/test_cli.py

This behaviour is legitimate, but not suitable:

Finding the rootdir

Here is the algorithm which finds the rootdir from args:

  • determine the common ancestor directory for the specified args that are recognised as paths that exist in the file system. If no such paths are found, the common ancestor directory is set to the current working directory.

Some of disadvantages of current implementation of ipa-run-tests script:
1) the stability of nodeids outcome is broken, because it depends on current working directory.
2) pytest_load_initial_conftests hook (as it was shown above) is not intended for changing paths in current way
3) not all of the Unix glob patterns are supported
4) it needs to mangle paths rather than using the default conventions for Python test discovery (which is more stable)

I suggest to use a chdir to ipatests module directory and then execv (sort of pre https://github.com/freeipa/freeipa/commit/93c158b05812b55383043e320cac89550993b8b2).
The motivation of that change says about 2 problems at least.
New "old" implementation don't have them (checked against pytest 3.9.3 and 4.6.2):
1) collect and run by nodeid is working

ipa-run-tests test_cmdline/test_ipagetkeytab.py::test_ipagetkeytab::test_1_run
...
============================ 1 passed
...

2) coverage report (pytest-cov) is the same as for bare pytest

ipa-run-tests --cov=ipatests.test_cmdline test_cmdline
...
Name                                Stmts   Miss  Cover
-------------------------------------------------------
test_cmdline/__init__.py                2      0   100%
test_cmdline/cmdline.py                30     12    60%
test_cmdline/test_cli.py              153    108    29%
test_cmdline/test_help.py              98     75    23%
test_cmdline/test_ipagetkeytab.py     226    160    29%
-------------------------------------------------------
TOTAL                                 509    355    30%

Today I will open the corresponding PR.


master:

  • 2312b38 Simplify ipa-run-tests script
  • a2d4e2a Make use of Azure Pipeline slicing
  • c750022 Avoid use of '/tmp' for pip operations

Metadata Update from @abbra:
- Issue set to the milestone: FreeIPA 4.8.1

4 years ago

ipa-4-8:

  • 8aa3ef0 Simplify ipa-run-tests script
  • 5780d6a Make use of Azure Pipeline slicing
  • 0ec908b Avoid use of '/tmp' for pip operations

Metadata Update from @abbra:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

4 years ago

Login to comment on this ticket.

Metadata