So I have this kinda long-standing idea to do some update testing with openQA. The FreeIPA tests would make a good place to start: in theory we could make it so that whenever any update to a FreeIPA-related package was submitted, we'd run the FreeIPA tests for that release version with the current stable package set plus the packages from that update, and submit the results to Bodhi for that update.

This would be neat, but it's quite a lot of tricky integration work and would probably be a solid two, three week project. Which I don't have time for right now.

Talking to the pki folks today, though, it occurred to me that there's a middle-ground bodge we could probably do quite a lot faster. It'd be quite simple. We just create a new openQA product which would run only test suites that would be useful for testing updates - e.g. the FreeIPA and desktop tests. Then we set things up such that a run of those tests is triggered every time there's a push to updates-testing. Then we'd at least have the tests run on the entire contents of each updates-testing push, and we'd notice when one of the tests suddenly started failing and probably be able to manually pinpoint the offending update and address the problem. We'd probably want to do a check-compose style email so it'd be easy to spot when a test started failing.

There'd be some refactoring needed to make this fly, of course, but it's probably all stuff that could be bodged up in a few hours or days. I'm gonna try and make time to give this a shot soon, as I think it's something that could be useful...