5ac9f3e allow wiki reporting per test module, report FreeIPA_web_ui

Authored and Committed by adamwill 7 years ago
    allow wiki reporting per test module, report FreeIPA_web_ui
    
    Summary:
    This tweaks the wiki result reporting bits to be a bit more
    flexible and allow for reporting the QA:Testcase_FreeIPA_web_ui
    results. We added that test case as a single, non-fatal test
    module within the realmd_join_cockpit test suite. That means
    that when openQA runs that test suite, it will run the FreeIPA
    web UI test module and record the result of that module, but
    if that module fails, the openQA job will not stop nor will it
    be considered to have failed; if all the other modules pass,
    the job is considered to have passed. So for wiki reporting
    purposes we need to be able to say 'report a pass for this
    Wikitcms test case only if this specific test module within the
    job passed'.
    
    We allow this by letting entries in the TESTSUITES dict be
    dicts instead of lists, with a special key for the normal
    'these test cases passed if the job passed' list and other keys
    representing individual modules, with lists of test cases that
    passed only if those specific test modules passed. There's a
    bit more detail in the comments.
    
    Test Plan:
    Generate ResTups for a run of realmd_join_cockpit
    where both the job and the freeipa_webui_postinstall module
    passed - e.g. https://openqa.stg.fedoraproject.org/tests/24944
     - and make sure a ResTup for QA:Testcase_FreeIPA_web_ui is
    generated. If possible, also generate ResTups for a run where
    the job passed but the module failed, and make sure there are
    ResTups for the *other* test cases but *not* that one, and
    also generate ResTups for a run where the job failed and make
    sure we get none.
    
    Reviewers: jskladan, garretraziel
    
    Reviewed By: garretraziel
    
    Subscribers: tflink
    
    Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D932