PR-CI doesn't mark a test run with unexpected passes as failed. For example test_installation.py::TestInstallWithCA_DNS1::()::test_replica0_ca_less_install from http://freeipa-org-pr-ci.s3-website.eu-central-1.amazonaws.com/jobs/9369fcca-7f18-11e8-aae8-fa163ef99e01/report.html is a XPassed.
test_installation.py::TestInstallWithCA_DNS1::()::test_replica0_ca_less_install
Another example: http://freeipa-org-pr-ci.s3-website.eu-central-1.amazonaws.com/jobs/59805778-83d4-11e8-9987-fa163e430211/report.html for fedora-28/test_installation_TestInstallWithCA1 has 8 unexpected passes, e.g. test_installation.py::TestInstallWithCA1::()::test_replica2_ipa_dns_install.
fedora-28/test_installation_TestInstallWithCA1
test_installation.py::TestInstallWithCA1::()::test_replica2_ipa_dns_install
FreeIPA's test suite is using xfail to mark tests as known failures. A failing test doesn't cause the test suite to fail. Instead it's recored as XFAIL (expected failure). It's also possible that an xfail test passes, Pytest calls this XPASS (unexpected pass). PR-CI doesn't consider an unexpected pass as an error.
With @pytest.mark.xfail(strict=True), pytest turns an XPASS into an error. See https://pytest.readthedocs.io/en/reorganize-docs/new-docs/user/xfail.html for more details
Metadata Update from @cheimes: - Issue assigned to cheimes
Metadata Update from @cheimes: - Custom field on_review adjusted to https://github.com/freeipa/freeipa/pull/2131
master:
Metadata Update from @cheimes: - Issue close_status updated to: fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.