PR#550: docs: Describe how scenario, system_architecture and system_variant are handled - greenwave

greenwave

#550 docs: Describe how scenario, system_architecture and system_variant are handled

Merged 4 years ago by lholecek. Opened 4 years ago by lholecek.

lholecek/greenwave doc-scenario-and-system into master

docs: Describe how scenario, system_architecture and system_variant are handled

Lukas Holecek • 4 years ago

513cc4b

docs/policies.rst

file modified

+9 -2

		`@@ -163,8 +163,15 @@`
		`-------------------`

		`For this rule to be satisfied, there must be a result in ResultsDB for the`
		- given ``test_case_name`` with an outcome of ``PASS``, or there must be a
		`- corresponding waiver in WaiverDB for the given test case.`
		+ given ``test_case_name`` with an outcome of ``PASSED`` or ``INFO``, or
		`+ there must be a corresponding waiver in WaiverDB for the given test case.`
		`+`
		`+ The rule requires all matching latest test results with distinct triplets`
		+ ``system_architecture``, ``system_variant`` and ``scenario`` (which are
		`+ defined in result data) to pass or be waived.`
		`+`
		+ Optional ``scenario`` property can be specified to consider only results
		`+ with a given scenario name.`

		`.. _remote-rule:`

lholecek commented 4 years ago

JIRA: FACTORY-5904
Signed-off-by: Lukas Holecek hluk@email.cz

rebased onto cff675bbdf66e9e073a0cb0f22f417e4d3c3b675

4 years ago

gnaponie commented on line 10 of docs/policies.rst 4 years ago

Maybe we should explain a bit what "scenario" is (or can be) and maybe provide an example.

lholecek commented 4 years ago

Maybe we should explain a bit what "scenario" is (or can be) and maybe provide an example.

Sorry, I'm still unsure why scenario is used instead of e.g. more specific test case name.

It has something to do with https://pagure.io/greenwave/pull-request/96.

gnaponie commented 4 years ago

I'm not an expert either. But for what I understood is useful to differentiate different scenarios (you don't say :D) for the same testcase. It is used for Fedora openQA... and it is used mainly for composes.
Some discussion happened in here in the past: https://pagure.io/greenwave/pull-request/393
and examples are in the fedora policies: https://greenwave.fedoraproject.org/api/v1.0/policies

Maybe let's just say it can help to distinguish different use case for the same test case.

rebased onto b43d930ef95b1aa49c5835139aaab661dedb103e

4 years ago

lholecek commented 4 years ago

Can you help me put together a sentence that would make it more clear?

gnaponie commented on line 10 of docs/policies.rst 4 years ago

Can you help me put together a sentence that would make it more clear?

What about something like this here:
Optionally, an additional scenario field can be specified to distinguish different use cases that have the same test case. It can be useful to have a more granular set of tests.
In case of two results with the same test case, but different scenario, these results will be treated independently.
The same thing will happen for system_architecture and system_variant.

--
feel free to reword this

lholecek commented on line 10 of docs/policies.rst 4 years ago

What about something like this here:

Hmm, I'm not sure if adds any additional information.

Is it unclear from my description that the scenario is used to distinguish results further?

gnaponie commented 4 years ago

Mmm yeah, but you first introduce scenarios, without telling what they are and what's the purpose, just "to filter results by scenario name"... which I find confusing... Maybe you can write:
"an additional scenario field can be specified to distinguish different use cases that have the same test case. It can be useful to have a more granular set of tests."
and then just:
"The rule requires all matching latest test results with distinct triplets system_architecture, system_variant and scenario (unless specific value is requested) to pass or be waived."