Hi,

Thanks for the report.

I was trying to figure out how to suppress bogus errors reported by fedora-ci.koji-build.installability.functional for systemd. I already have a .zuul.yaml file that was supposed to suppress the test which cannot be satisfied, but it doesn't seem to work.

Could you please link to the bogus job / Zuul build ? Is there an issue opened that describe that bug on https://pagure.io/fedora-ci/general/issues ? The Zuul CI does not trigger the fedora-ci.koji-build.installability.functional test.

I found this:
https://fedoraproject.org/wiki/Zuul-based-ci#For_distgit_projects_on_src.fedoraproject.org:

For every distgit projects we believe that the Zuul pipeline should be similar, thus, by default, we do not allow package maintainer to add custom Zuul configurations in-distgit.

This is fundamentally misguided. If the goal is for packagers to take the test results seriously, just make them report things correctly, i.e. reduce the false positive rate as much as possible. Do not make packagers jump through hoops to suppress tests that are broken or cannot be immediately fixed. In particular, packagers shouldn't have to write yaml in some complex configuration space they have no idea about.

Removing the capability for a packager to provide in-repo (in-distgit) Zuul configuration is to avoid having to handle a myriad of custom config (with side effects, zuul config questions, and so on). The Zuul configuration for all distgit is then centralized 1 2. This still give capability for a packager to override default templates for instance here 3 the rpm-linter is set as non-voting for the pypy distgit.

However I see that for systemd, there is already an exception to that rule 3. Your .zuul.yaml sets the Zuul var install_repo_exclude which is used here. So this only have impact on the Zuul job rpm-install-test which is not the fedora-ci.koji-build.installability.functional test. I will redirect to @msrb or @mvadkert they maybe known more about this job ?

In this issue, see a discussion about the removal of rpm-install-test in favor of the installability test managed by Testing Farm

My proposal:
- drop the policy quoted above and always honour .zuul.yaml if present in dist-git.

See explanation above. It is still possible for a project to enable in dist-git Zuul configuration (like for systemd)

• in any test output, always link to a page in docs which explains how the test is configured and how to suppress it.

We Set a link to the Wiki page in the Zuul comment

• profit from packagers actually looking at CI results.
  

Could you please link to the bogus job / Zuul build?

https://artifacts.dev.testing-farm.io/badf5dc8-6193-4377-88b4-7919c8705163/

Is there an issue opened that describe that bug on https://pagure.io/fedora-ci/general/issues ?

I don't think so. I think there was some issue about the installabillity tests for systemd, but I cannot find it.

Removing the capability for a packager to provide in-repo (in-distgit) Zuul configuration is to avoid having to handle a myriad of custom config (with side effects, zuul config questions, and so on). The Zuul configuration for all distgit is then centralized 1 2. This still give capability for a packager to override default templates for instance here 3 the rpm-linter is set as non-voting for the pypy distgit.

Hmm, so I think we'll have to agree to disagree. I very much agree that common config should be provided, and that packages should be subject to the standard tests by default, but I don't agree that exceptions to this should be managed centrally. It's much nicer for individual contributors if they can put the overrides for dist-git and manage them without waiting for something external.

However I see that for systemd, there is already an exception to that rule 3. Your .zuul.yaml sets the Zuul var install_repo_exclude which is used here. So this only have impact on the Zuul job rpm-install-test which is not the fedora-ci.koji-build.installability.functional test. I will redirect to @msrb or @mvadkert they maybe known more about this job ?

Thanks. I think somebody explained this to me before already, but I keep forgetting. When I try to google for this, I'm guided to the zuul tests, which are more documented than the other stuff.

OK, I'll close this issue then. The problem I have with installability test is elsewhere.

Though if somebody knows how to make fedora-ci.koji-build.installability.functional ignore uninstallable packages for systemd, please let me know. The docs at https://docs.fedoraproject.org/en-US/ci/gating/ and https://gating-greenwave.readthedocs.io/en/latest/package-specific-policies.html are not comprehensible for me.