PR#103: Use separate field for test scenario description. - fedora-ci/messages

fedora-ci / messages

#103 Use separate field for test scenario description.

Merged 3 years ago by mvadkert. Opened 3 years ago by pholica.

fedora-ci/ pholica/messages scenario_support into master

Use separate field for test scenario description.

Pavel Holica • 3 years ago

ad416c2

schemas/test-common.yaml

file modified

+22 -8

		`@@ -81,12 +81,12 @@`
		`type:`
		`description:`
		`Test type. Identifies the test(s) being run in this execution.`
		`- Depending on your test system this may indicate a single test`
		`- (possibly with 'scenario' information as well), a group of tests,`
		`- or you may only ever use one value here if your test system does`
		`- not emit separate messages per test or test group. A good way to`
		`- think of this is "the information a consumer would need to identify`
		`- runs of 'the same test(s)' on two different artifacts".`
		`+ Depending on your test system this may indicate a single test,`
		`+ a group of tests, or you may only ever use one value here if your`
		`+ test system does not emit separate messages per test or test`
		`+ group. A good way to think of this is "the information a consumer`
		`+ would need to identify runs of 'the same test(s)' on two different`
		`+ artifacts".`
		`examples:`
		`- tier1`
		`- tier2`
		`@@ -96,10 +96,24 @@`
		`- covscan`
		`- pipeline`
		`- autocloud`
		`- - base_selinux KDE-live-iso x86_64 64bit`
		`+ - base_selinux`
		`- Any other identifier that makes sense for your test system`
		`type: string`
		`-`
		`+ scenario:`
		`+ description:`
		`+ Test scenario. Identifies scenario under which the test(s) are`
		`+ executed. This is useful in case of artifacts consisting of`
		`+ multiple items which are subject of the same testsuite. For example`
		`+ "productmd-compose" artifact contains multiple variants and`
lholecek commented 3 years ago The variant and architecture is already included in `productmd-compose.test.complete` messages (see "schemas/productmd-compose.test.complete.yaml" and "schemas/productmd-compose-system.yaml"). These values are used by Greenwave similarly as "scenario". So using "scenario" in this case seems redundant if it would contain the same variant+arch values.
		`+ architectures all with their own repositories and installation`
		`+ media on which the same testsuite is executed. The variant and`
		`+ architecture would be in such case the scenario. The scenario is`
		`+ free form text where the tested item identifier is encoded.`
		`+ examples:`
		`+ - KDE-live-iso x86_64 64bit`
		`+ - Server x86_64`
		`+ - Any other identifier`
		`+ type: string`
		`required:`
		`- category`
		`- namespace`

pholica commented 3 years ago

The reason for this change is to move currently complex value encoded to string (scenario in type) to separate own field.

This change will be probably controversial and changes the way how messages with scenarios should be processed. Fortunately, it should be possible to have a processing code handling both situations at the same time (in case of separate scenario field just use that one and don't do anything with the type, if the scenario is not provided process the message the same way as up until this change).

lholecek commented on line 34 of schemas/test-common.yaml 3 years ago

The variant and architecture is already included in productmd-compose.test.complete messages (see "schemas/productmd-compose.test.complete.yaml" and "schemas/productmd-compose-system.yaml").

These values are used by Greenwave similarly as "scenario". So using "scenario" in this case seems redundant if it would contain the same variant+arch values.

pholica commented 3 years ago

The variant and architecture is already included in productmd-compose.test.complete messages (see "schemas/productmd-compose.test.complete.yaml" and "schemas/productmd-compose-system.yaml").
These values are used by Greenwave similarly as "scenario". So using "scenario" in this case seems redundant if it would contain the same variant+arch values.

Yes, they are there, but unfortunately, they don't provide enough scalability as the system doesn't provide enough scalability for the scenarios. For example, one can use one type of system to test both BIOS and UEFI platforms and execute the same testsuite on it.

I'm thinking now the "label" in the system could be used for this but still I feel that may not be the good way to proceed. Also since the scenario is already in the test-common I believe it would be beneficial to use the more common approach which would provide bigger scalability options.

The plan we have is to define different combinations of scenarios also along with different tests (which don't use scenarios) in the greenwave policies. E.g. have just only "x86_64" arch, "Server" variant, "a" result together with test result of "b" and act when both of those are pass for one artifact.

lholecek commented 3 years ago

Metadata Update from @lholecek:
- Request assigned

3 years ago

Metadata Update from @lholecek:
- Request assigned

3 years ago

mvadkert commented 3 years ago

@pholica makes sense to me and I like it. Thanks for the review @lholecek

mvadkert commented 3 years ago

@adamwill @ralph hey fellaz, are you ok with this also.

@adamwill I believe you can still use the scenario in the test type if you want ... so I see no breakage of your use case

adamwill commented 3 years ago

uh, my brain is quite full of cheese at the moment...but I don't think I have any problem with this. I don't think I have anything that actually consumes this ATM, openQA results just put the scenario info in that field because it seemed like the closest fit in the spec when we onboarded openQA, IIRC.