So I did a bit of poking around here in response to some discussion on releng IRC today. This is all still fairly mysterious, though.

@kevin reported that we're seeing errors like this (I assume from Bodhi?):

greenwave.api_v1 ERROR Cannot find any applicable policies for koji_build subjects at gating point bodhi_update_push_stable in fedora-34

What that error means, according to greenwave, is "I couldn't find any policy that applies to the subject.type "koji_build" and the decision_context "bodhi_update_push_stable" for the product_version "fedora-34".

However, this seems pretty mysterious, because looking at our Greenwave config at the time, the policy "taskotron_release_critical_tasks_for_stable" should certainly have matched that query.

@kevin tried to fix this by wiping out the "taskotron" policies and adding current mainline Fedora dists to the "no_requirements" policies, but I can't see any reason that ought to help, and it seems definitely wrong to me. If it did help, something seems decidedly broken.

The reason I think it's wrong is that I think it effectively disables per-package gating policies. The only practical difference in the older version between the "taskotron_release_critical" policies and the "no_requirements" policies is that the former policies applied the rule !RemoteRule {}, while the latter policies applied no rules. That RemoteRule is the special sauce that makes policies in the package repos work: see greenwave docs for more detail. All that @kevin 's change ought to have done, AFAICS, is disable that special sauce for fedora-rawhide, fedora-34, fedora-33 and fedora-32; that seems both a) wrong and b) like it shouldn't fix the actual problem.

This doesn't do anything in particular to fix the bug, but it does change the file to be more or less how I think it should be, if I'm understanding things right, and explain stuff better. And use better names for the policies, because the ones with "taskotron" in their names didn't really have anything to do with taskotron at all any more.

There are still some open questions. Why do we apparently only want to let per-package gating policies apply to the 'mainline' dists? IIUC, with the policy as it stood before @kevin 's changes and as it would stand after this PR, per-package gating policies will always be ignored for EPEL, modules, containers and flatpaks. Is that really what we wanted?

Also, should we make product_versions for the "no_requirements" policies just be - fedora-*? I think that ought to work, and it would save a lot of tedious editing and would also mean we aren't broken for new releases when they show up. The job that rule is actually doing, AFAICT, is "make sure we have at least one policy that will match any query run for the koji_build subject type", because as Greenwave is currently configured, a query for that subject type (in fact, queries for any subject type other than "bodhi_update") that matches no policies is an error. (That's the error we're hitting). "bodhi_update" is treated specially in Greenwave for some reason, it is the only subject type for which the config setting "ignore_missing_policy" defaults to True. That's why it's not a problem that we don't have policies for modular, containers, flatpaks, epel and eln for the bodhi_update subject type (this took me a while to figure out).

If anyone thought this policy actually enforced that no requirements would be applied at all to the versions included in it, it doesn't. That's because policies are additive (as I mentioned in one of the comments added in this PR). If we had e.g. this:

--- !Policy
id: "kojibuild_no_requirements_testing"
product_versions:
  - fedora-*
decision_context: bodhi_update_push_testing
subject_type: koji_build
rules: []

id: "fedora_mainline_kojibuild_remoterule"
product_versions:
  - fedora-rawhide
  - fedora-34
  - fedora-33
  - fedora-32
decision_context: bodhi_update_push_testing
subject_type: koji_build
rules:
  - !RemoteRule {}

then there isn't any kind of conflict or precedence question or anything; per Greenwave's documentation it simply adds up all the rules in all the applicable policies. So in this case it'd apply all the rules in the two policies combined. The "no_requirements" policy doesn't prevent the other policy from taking effect, or anything.

Ah ha. See, I didn't know anything about the RemoteRule thing. ;)

@adamwill note that greenwave also has an issue with how it applies the remoterule which can lead to the same error message: https://pagure.io/greenwave/issue/603

Aha - that could absolutely be the case we were running into, and would also explain why effectively not applying RemoteRule to fedora-34 any more "fixed" the problem. Thanks.