I floated this on devel@ and in this week's QA meeting and it got a generally positive response, so I thought it might be worth a ticket.
We don't currently 'enforce' AutoQA test results because the tests aren't really reliable enough: we know there are timing/ordering issues with upgradepath, and known cases of both false failures (e.g. explicit conflicts, internal dependencies between multiple binary RPMs from a single source RPM) and false successes for depcheck.
Still, both tests have spotted significant failures which we've ended up shipping: see https://admin.fedoraproject.org/updates/FEDORA-2014-3092/dnf-0.4.15-1.fc20 , and more generally all the ones listed in https://lists.fedoraproject.org/pipermail/devel/2014-February/196072.html .
So, as a halfway proposal until Taskotron is ready for production (still probably a month away at least), can we consider disabling autokarma for an update if either of the AutoQA checks fails? I think this could be implemented quite easily; it might have to be a bit hacky (like parsing the AutoQA bot's comments for "FAILED"), but that should be okay for a short-term stopgap. It would be easily reversible by the package maintainer if the test result was incorrect; they could just turn automatism right back on again. And it doesn't flat prevent the update going out in any case, just should help avoid broken updates going out via automatism before the maintainer can spot the problem.
This was approved at today's meeting. I will work with AutoQA folks and Luke to get it implemented; we'll link to any list discussions / trac tickets / etc here.
From the 2014-03-05 FESCo meeting:
AGREED: Proposal in ticket to disable autokarma if AutoQA fails accepted (+:9,-:0,0:0) (notting, 19:20:22)
Removing meeting keyword. Please update if there are issues.
tflink has filed a ticket against Bodhi for this: https://github.com/fedora-infra/bodhi/issues/36
The issue has been fixed, it's just pending deployment now.
Closing this ticket now.
to comment on this ticket.