allow wiki reporting per test module, report FreeIPA_web_ui
Summary:
This tweaks the wiki result reporting bits to be a bit more
flexible and allow for reporting the QA:Testcase_FreeIPA_web_ui
results. We added that test case as a single, non-fatal test
module within the realmd_join_cockpit test suite. That means
that when openQA runs that test suite, it will run the FreeIPA
web UI test module and record the result of that module, but
if that module fails, the openQA job will not stop nor will it
be considered to have failed; if all the other modules pass,
the job is considered to have passed. So for wiki reporting
purposes we need to be able to say 'report a pass for this
Wikitcms test case only if this specific test module within the
job passed'.
We allow this by letting entries in the TESTSUITES dict be
dicts instead of lists, with a special key for the normal
'these test cases passed if the job passed' list and other keys
representing individual modules, with lists of test cases that
passed only if those specific test modules passed. There's a
bit more detail in the comments.
Test Plan:
Generate ResTups for a run of realmd_join_cockpit
where both the job and the freeipa_webui_postinstall module
passed - e.g. https://openqa.stg.fedoraproject.org/tests/24944
- and make sure a ResTup for QA:Testcase_FreeIPA_web_ui is
generated. If possible, also generate ResTups for a run where
the job passed but the module failed, and make sure there are
ResTups for the *other* test cases but *not* that one, and
also generate ResTups for a run where the job failed and make
sure we get none.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D932