add a script for sanity checking conf_test_suites
Summary:
It bugs me that it's not very visible if we've messed up wiki
reporting: we really have to be paying attention to spot that
we don't have reporting hooked up for a test, or if we have it
hooked up but we didn't quite get the settings right and
wikitcms won't manage to report the result. The latter case is
partly logged, but that still means we have to go poke through
the server logs to spot it, and there are in fact cases where
wikitcms won't report insufficient / incorrect restups (I
should probably fix that), and it means we don't catch the
issue till after we've committed the changes; we can't check
before commit.
So I fixed it! Here's a script that checks the TESTSUITES and
TESTCASES dicts correspond, that we have an entry in TESTSUITES
for every test suite in the openQA templates, and that wikitcms
can actually find a resultrow and environment for a set of
faked-up 'passed jobs' derived from the openQA templates. Now
we can use this to sanity check commits that add wiki reports.
We also fix an error it discovered, for KDE_Package_Install.
The result would actually have gotten reported anyway in this
case (as wikitcms doesn't bother checking the restup's 'env'
value if the resultrow only has *one* env), but it's good to
be correct.
I guess this is *one* way to spend an 8 hour train ride...
Test Plan:
Run the script, see what it tells you. On current
develop it should spot that there is no reporting of the
graphical update tests yet. You can introduce intentional
errors to conf_test_suites to see if it catches them.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1013