fedora-ci / general

#38 Extend test results format to specify test subject tested
Opened by kparal. Modified

Open
Close Issue

The test result format was recently substantially improved (#18). Yet, in order to move Taskotron tests to Fedora CI (#36), we need another addition. This use case is specific to rpmdeplint, a test which gets scheduled on Koji tag changes, but produces results for Koji builds (pass/fail whether they have satisfied dependencies inside that tag/repo). Because there's no 1:1 mapping of the test subject the CI knows about (the koji tag name) and of the test subjects the rpmdeplint produces results for (all Koji builds inside that tag, determined dynamically), the existing result format is not sufficient:

     - {result: pass, test: test1, log: /path/to/test1.log}
     - {result: fail, test: test2, log: /path/to/test2.log}

because it doesn't specify what was tested. We would need to extend the format to something like this:

     - {result: pass, test: rpmdeplint, subject: NVR1, log: /path/to/nvr1.log}
     - {result: fail, test: rpmdeplint, subject: NVR2, log: /path/to/nvr2.log}

where subject: foo is optional (if it is not present, that CI system assumes the test result applies to the original test subject the test was scheduled for).

In fact, this might need one more keyword, because subject: NVR1 might be confusing. Is NVR1 a Koji package build, or a Koji module build, or a Bodhi update ID, or a Koji tag name, or a compose ID, or something else? So additional keyword like this subject_type: koji_build might be needed to fully identify what was tested.

     - {result: pass, test: rpmdeplint, subject: NVR1, subject_type: koji_build, log: /path/to/nvr1.log}
     - {result: fail, test: rpmdeplint, subject: NVR2, subject_type: koji_build, log: /path/to/nvr2.log}

Here's an example of how test results look like for rpmdeplint in Taskotron:

- arch: x86_64
  artifact: /var/lib/taskotron/artifacts/499480d4-4015-11e9-be29-525400fc9f92/tests.yml/itemlogs/pungi-4.1.33-1.fc28.x86_64.log
  checkname: dist.rpmdeplint
  item: pungi-4.1.33-1.fc28
  outcome: PASSED
  scenario: x86_64
  type: koji_build
- arch: x86_64
  artifact: /var/lib/taskotron/artifacts/499480d4-4015-11e9-be29-525400fc9f92/tests.yml/itemlogs/ckeditor-4.11.2-1.fc28.x86_64.log
  checkname: dist.rpmdeplint
  item: ckeditor-4.11.2-1.fc28
  outcome: PASSED
  scenario: x86_64
  type: koji_build

It's very similar to what you have, just a different terminology: outcome->result, checkname->test, artifact->log, and the new ones are item->subject and type->subject_type. FYI, here's a list of subject types Taskotron knows about (i.e. a test can report results for them).

I'm not saying Fedora CI needs to implement the same structure Taskotron uses, but if we want to run rpmdeplint inside it, or any other test that produces results for multiple different subjects in a single run (especially when the list of such subjects is not known in advance to the CI system), the test result format must have a way to distinguish which result applies to which subject.

I note in passing that this proposal uses the "type" koji_build, as Taskotron resultsdb results do. (CI pipeline resultsdb results do not identify an artifact/subject type). The equivalent artifact type in the fedora-ci/messages spec is koji-build.

Once again my theory that dashes and underscores were invented by the devil to cause us pain and misery is borne out. :P

Just a clarification note: The results.yml format proposed in #18 is supposed to be used by Standard Test Interface tests, that is by tests enabled using tests.yml files in dist git. I'm not sure whether we want to enable rpmdeplint using STI and thus I'm a bit hesitant to introduce additional fields to the format.

Shall we perhaps start from the use case? I mean first find the right way/place how rpmdeplint will be integrated into Fedora CI and how exactly will be results processed and only then find what else is needed?
Edited by psss

My assumption was that STI is the universal format to be used for literally everything. If you want to take a different path, that's completely fine, and will likely make some of my tickets invalid :) Let's sync up in person tomorrow and update this ticket with our conclusions.

The outcome of our meeting was that we'll agree on a results format which will be produced by generic tests and will be used by the CI tool to distribute results messages on the bus. It will support defining multiple test results for multiple test subjects (the rpmdeplint use case). The results format doesn't have to be necessarily the same as is used for STI-based distgit tests (which I referenced here). It can be an xunit, etc. In my personal opinion, it would be nice to keep the results format the same, though (whether it's a generic test or a distgit test), so that we don't end up with too many formats unnecessarily.

Per my update in #36, is this also still valid after taskotron's EOL?

As a general concept, yes, I think it's still valid. I believe you'll hit this issue with other generic tests similar to rpmdeplint (i.e. operating on a large number of test subjects at once) in the future.
Specifically for rpmdeplint, I think this is no longer valid, because rpmdeplint was implemented as a custom pipeline in Fedora CI, so this issue is either somehow resolved or sidestepped.

I provided a longer update in:
https://pagure.io/fedora-ci/general/issue/36#comment-650628
Metadata
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.