fedora-ci / metadata

#13 Support defining a generic test
Closed by psss. Opened by kparal.

Closed
Reopen Issue

Per our discussion regarding moving Taskotron tests to Fedora CI, this is a request for CI metadata to be able to define a generic test. A generic test is such a test that is not tied to a certain package (e.g. ssh), i.e. it is not stored in a package distgit. Rather it is defined (and possibly stored) server-side and is executed every time a matching event happens.

For most of our tests, we'd need to run them every time a new Koji build is finished (and perhaps we can also extend this to scratch builds or pull requests as well).

For rpmdeplint test, we'd need to run it each time a specific Koji tag is modified (a new build is tagged into it). In Taskotron, we currently execute rpmdeplint on Koji tag changes matching this regex:

$regex: '/^f[0-9]{2}-updates(-testing)?-pending$$/'

And so we'd need a similar matching mechanism for Koji tag events even in Fedora CI. I imagine this filtering mechanism could be generic for any event type (so e.g. filter package names for koji builds, filter tag names for koji tag changes), and not just tied to the one specific use case. See Test subject blacklist/whitelist section below, this can be essentially the very same thing described there.

Extended requests

I have a few more requests related to the generic test definition. I can file them as separate tickets, or keep them here, whatever you prefer. Some of these might apply in general to all tests and not just generic tests.

Architecture handling

We need to think about different architectures when scheduling a generic test. Some tests might be able to test all available test subject architectures in a single execution (like rpmlint which just performs static tests on the RPM files), some might require running on a host of the same architecture as the test subject architecture and therefore require several executions (per-arch).

In Taskotron, for simplicity-sake, we always run generic tests per-arch, because it increases test reliability (some tests take a long time on large packages and can easily time out when trying to evaluate all arches at once), but the test itself can override it and schedule just a single "noarch" execution (that makes sense for a test that e.g. only works with src rpms, or just with dnf metadata, or is very quick in general).

We most probably need ci-metadata to allow for a similar definition of "how the test should be executed regarding system architectures".

Test subject blacklist/whitelist

Certainly not essential, but it might be nice to support a blacklist/whitelist of test subjects (e.g. packages) for a certain test. For example, we maintain a blacklist of packages for which we simply can't run abicheck, because it requires far too much memory or time to execute (think libreoffice or kernel). This way we save resources for tests that are otherwise guaranteed to fail/time out (blacklist), or we allow for easy experimentation for a brand new test (whitelist).

Having such a capability in ci-metadata could certainly help in such cases.

It seems it would be good to sync on these issues in person as I've got feeling that our expected use cases are not completely aligned. But what makes clear sense to me is the possibility to allow blacklisting or arch configuration for a package to be stored in the CI metadata (if it makes sense to have this stored in component's dist git repo). Similarly as we have drafted the example for the rpmdiff configuration.

We've had an in-person discussion and concluded that we'd like to have a repo with generic tests and a matching metadata configuration for them (that might require some new fields). The package maintainers might be able to configure/override certain aspects for them (or perhaps opt-in/opt-out). The test discovery process (FMF) would be able to look at both places (distgit and generic tests location) and schedule all matching tests.

Hi, are there any developments on this topic? From what I understand so far, the

[...snip...] we'd like to have a repo with generic tests and a matching metadata configuration for them [...snip...] The test discovery process (FMF) would be able to look at both places (distgit and generic tests location) and schedule all matching tests.

part is kind-of-sort-of there with discover https://tmt.readthedocs.io/en/latest/spec/steps.html#discover used with a git-repo-url.

The part that seems to be missing (or I might have misunderstood/missed something) is a way to pass externaly-defined parameters (I can see an ENV variable, or a command line parameter) to replace a "placeholder" in the FMF metadata on execution.

Say we have a tests/example/main.fmf file containing something like:

summary: My awesome generic test for all the builds!
contact: Mr Coconut <coconut@fp.o>
test: ./run_test.py --item={CMDLINE: ITEM} --item_type={ENVVAR: I_TYPE}

And then could run something like

I_TYPE='koji_build' tmt run --ITEM='htop-2.0.2-4.fc27'

To have the tests/example/main.fmf processed like this "on runtime"

summary: My awesome generic test
contact: Mr Coconut <coconut@fp.o>
test: ./test.sh  --item='htop-2.0.2-4.fc27' --item_type=koji_build

I'm using the env-vars/cmdline params just as an example (since this would probably be quite easy to use on the consumer side), but I can easily see something like variables.json file providing the key-values. But hopefully the requirement/intent is clear.

All the tests I've seen so far are package-specific, and do not need to solve this particular issue. I could have obviously easily missed something between all the docs and repos, so if "something like this" is already possible, I'd be glad for a relevant example/link.
Edited by mvadkert

@jskladan meh, we last time discussed with @psss that it would be good time to inform the the metadata definition is now in https://tmt.readthedocs.io/en/latest/spec.html. We should migrate all the issues there :(

[...snip...] we'd like to have a repo with generic tests and a matching metadata configuration for them [...snip...] The test discovery process (FMF) would be able to look at both places (distgit >> and generic tests location) and schedule all matching tests.

part is kind-of-sort-of there with discover https://tmt.readthedocs.io/en/latest/spec/steps.html#discover used with a git-repo-url.

Indeed, this location is not yet clear, but this will be on OSCI team to handle where these tests will be.

The part that seems to be missing (or I might have misunderstood/missed something) is a way to pass externaly-defined parameters (I can see an ENV variable, or a command line parameter) to replace a "placeholder" in the FMF metadata on execution.

Say we have a tests/example/main.fmf file containing something like:
summary: My awesome generic test for all the builds! contact: Mr Coconut <coconut@fp.o> test: ./run_test.py --item={CMDLINE: ITEM} --item_type={ENVVAR: I_TYPE}

And then could run something like
I_TYPE='koji_build' tmt run --ITEM='htop-2.0.2-4.fc27'

To have the tests/example/main.fmf processed like this "on runtime"
summary: My awesome generic test contact: Mr Coconut <coconut@fp.o> test: ./test.sh --item='htop-2.0.2-4.fc27' --item_type=koji_build

I'm using the env-vars/cmdline params just as an example (since this would probably be quite > easy to use on the consumer side), but I can easily see something like variables.json file providing the key-values. But hopefully the requirement/intent is clear.

All the tests I've seen so far are package-specific, and do not need to solve this particular issue. I could have obviously easily missed something between all the docs and repos, so if "something like this" is already possible, I'd be glad for a relevant example/link.

I believe these generic tests will be on level 2 and the idea is they will look currently like this:

https://github.com/psss/tmt/pull/96

I believe this addresses the concern how you pass the package to tmt run. We believe this is as generic as it can be.

Let's continue the discussion there.

Example generic test configuration can be found here:
https://github.com/fedora-ci/rpmdeplint-pipeline

External parameters can be passed as environment variables:

tmt run --environment VAR=VAL

Is this issue obsolete? Or do we have any open questions to solve here?

Well, Fedora QA is no longer in immediate need of this (@tflink, feel free to correct me). We (or somebody else) will likely want to run generic tests on Fedora packages in the future. I haven't studied the provided examples in detail because I don't have an immediate use case, so it's really up to you do decide whether this needs more work and more tracking in tickets, or not. Thanks.

Ok, let's close this one for now. We can revisit later when we have specific use cases at hand.

Metadata Update from @psss:
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.