#4610 Refactor test suite to use pytest
Closed: Fixed None Opened 9 years ago by mkosek.

Current test suite and integration test framework use python-nose. While the current test suite and test framework for in our case, it may become limiting factor in future growth of the test suite.

For example, with growing number of tests, we will soon need to be able to run only selected range of test in a selected test run. Developer may need to run just quick unit tests without running whole test suite. Or CI pre-review test may need to run all single server tests. This and other more complicated tasks are difficult to do with python-nose.

The proposal is to switch to pytest framework and update the test infrastructure so that:

  • New tests can be easily added
  • There is clearer distinction between unit tests and other functional tests
  • We can configure classes/tiers of test and then only run subsection of them.

See initial discussion on freeipa-devel. This is something we will want to do in the beginning of 4.2 cycle.

master:

  • 375e9f7 tests: Use PEP8-compliant setup/teardown method names
  • 2b24faf tests: Add configuration for pytest
  • f3409ea ipatests.util.ClassChecker: Raise AttributeError in get_subcls
  • 84bd4c1 test_automount_plugin: Fix test ordering
  • 93c69b5 Use setup_class/teardown_class in Declarative tests
  • b64f91f dogtag plugin: Don't use doctest syntax for non-doctest examples
  • 82e41dc test_webui: Don't use init for test classes
  • 387b8b4 test_ipapython: Use functions instead of classes in test generators
  • dca259a Configure pytest to run doctests
  • 07def0b Declarative tests: Move cleanup to setup_class/teardown_class
  • 0cb12f3 Declarative tests: Switch to pytest
  • 3a9a98b Integration tests: Port the ordering plugin to pytest
  • bca6a14 Switch make-test to pytest
  • 48de128 Add local pytest plugin for --with-xunit and --logging-level
  • eaad0a9 Switch ipa-run-tests to pytest
  • 0ad5c57 Switch integration testing config to a fixture
  • 29c2878 Integration tests: Port the BeakerLib plugin and log collection to pytest
  • d42c26c test_integration: Adjust tests for pytest

master:

  • 74f7d67 test_integration: Use python-pytest-multihost
  • d9ab11a test_integration: Use collect_log from the host, not the testing class
  • a97d61d test_integration: Parametrize test instead of using a generator
  • 3e406f9 ipatests: Increase required version for pytest-multihost plugin

IPA now uses pytest, and an example of how to proceed is in place for the host plugin.

Tests can always use more work, though...

Right. I think that follow up work can be tracked in separate tickets, RFEs. For now, I think we can mark this work as completed. Thanks Petr!

Metadata Update from @mkosek:
- Issue assigned to someone
- Issue set to the milestone: FreeIPA 4.2

7 years ago

Login to comment on this ticket.

Metadata