5d8a217 add a script for sanity checking conf_test_suites

Authored and Committed by adamwill 7 years ago
    add a script for sanity checking conf_test_suites
    
    Summary:
    It bugs me that it's not very visible if we've messed up wiki
    reporting: we really have to be paying attention to spot that
    we don't have reporting hooked up for a test, or if we have it
    hooked up but we didn't quite get the settings right and
    wikitcms won't manage to report the result. The latter case is
    partly logged, but that still means we have to go poke through
    the server logs to spot it, and there are in fact cases where
    wikitcms won't report insufficient / incorrect restups (I
    should probably fix that), and it means we don't catch the
    issue till after we've committed the changes; we can't check
    before commit.
    
    So I fixed it! Here's a script that checks the TESTSUITES and
    TESTCASES dicts correspond, that we have an entry in TESTSUITES
    for every test suite in the openQA templates, and that wikitcms
    can actually find a resultrow and environment for a set of
    faked-up 'passed jobs' derived from the openQA templates. Now
    we can use this to sanity check commits that add wiki reports.
    
    We also fix an error it discovered, for KDE_Package_Install.
    The result would actually have gotten reported anyway in this
    case (as wikitcms doesn't bother checking the restup's 'env'
    value if the resultrow only has *one* env), but it's good to
    be correct.
    
    I guess this is *one* way to spend an 8 hour train ride...
    
    Test Plan:
    Run the script, see what it tells you. On current
    develop it should spot that there is no reporting of the
    graphical update tests yet. You can introduce intentional
    errors to conf_test_suites to see if it catches them.
    
    Reviewers: jskladan, garretraziel
    
    Reviewed By: garretraziel
    
    Subscribers: tflink
    
    Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1013
    
        
file added
+194