PR#97: MTF (meta test family) standard test role - standard-test-roles

standard-test-roles

#97 MTF (meta test family) standard test role

Opened 6 years ago by jscotka. Modified 6 years ago

jscotka/standard-test-roles str_mtf into master

MTF (meta test family) standard test role

Jan Scotka • 6 years ago

e238ad4

roles/standard-test-mtf/README.md

file added

+12

		`@@ -0,0 +1,12 @@`
		`+ # Ansible role for MTF tests`
		`+`
		`+ This role is intended for [Meta Test Family](https://github.com/fedora-modularity/meta-test-family) tests`
		`+ Test set has to be defined via [metadata definition](https://github.com/fedora-modularity/meta-test-family/tree/devel/mtf/metadata)`
		`+`
		`+`
		`+ * Put this role in your test_local.yml playbook.`
		`+ * You'll need to have the following variables defined:`
		+ * `type`: A string representing which artifact type will be tested (eg. docker, nspawn)
		+ * `testsubject_location`: string represeting repository or docker image name
		+ * `artifacts`: An artifacts directory
		`+`

roles/standard-test-mtf/defaults/main.yml

file added

		`@@ -0,0 +1,2 @@`
		`+ ---`
		`+ artifacts: "{{ lookup('env', 'TEST_ARTIFACTS')\|default('./artifacts', true) }}"`

roles/standard-test-mtf/tasks/main.yml

file added

+22

		`@@ -0,0 +1,22 @@`
		`+ ---`
		`+ - name: Enable Copr repo with MTF`
		`+ shell: dnf -y copr enable phracek/meta-test-family-devel`
		`+`
		`+ - name: Install the MTF requirements`
		`+ package: name={{item}} state=latest`
		`+ with_items:`
		`+ - meta-test-family`
		`+`
		`+ - name: Make artifacts directory`
		`+ file: path={{ artifacts }} state=directory owner=root mode=755 recurse=yes`
		`+`
		`+ - block:`
		`+ - name: Execute the MTF tests`
		`+ shell: exec 2>>{{artifacts}}/test.log 1>>{{artifacts}}/test.log; mtf --setup --metadata --module {{type}} --url {{testsubject_location}} --job-results-dir {{artifacts}}`
ttomecek commented 6 years ago the main difference b/w your role and the one I propose that you explicitly exec `mtf` while I want to leave this up to the user so the user is not locked
		`+`
		`+ - always:`
		`+ - name: Pull out the logs`
		`+ fetch:`
		`+ dest: "{{artifacts}}/"`
		`+ src: "{{artifacts}}/"`
		`+ flat: yes`

roles/standard-test-mtf/vars/main.yml

file added

		`@@ -0,0 +1,3 @@`
		`+ ---`
		`+ type: ""`
		`+ testsubject_location: ""`

jscotka commented 6 years ago

Initial draft how MTF can be handled via standard test roles.
Unclear part for me is how testsubject_location will be pasted, because I expect to test for example docker_image from outside.

Workflow for docker test subject is:
* start docker daemon
* import there test subject (docker image defined via testsubject_location)
* be able to test test subject as blackbox and optionaly as whitebox, means that I'm not able to test test subject (image) itself from inside, because container provide some services based on various environment definition and I have to be able to start in its own turn.

maybe @bgoncalv could be interested because it is same also for module testing.

astepano commented 6 years ago

@jscotka how do you think, whether it possible to extend some existing role to handle this case?
Does it make sense to spawn more roles for each individual case?

astepano commented 6 years ago

MTF can be run with help of standard-test-basic role

jscotka commented 6 years ago

yes, that could be part of some another role in case it make sense. I'm not sure what it better solution. if there will be more specialized roles, or less universal ones, but harder to config them - need to setup more variables there.
I'm not so familiar with standard test roles. so I let it up to you.

from that perspective there are 3. important parts:
* install additional requirement
* use test subject as an parameter (I dont know how to do it now - {{testsubject_location}} variable set maybe via TEST_SUBJECT env var in case it everytime contains this), and test subject is docker cotainer image (somewhere built)
* run special command: mtf --setup --metadata --module {{type}} --url {{testsubject_location}} --job-results-dir {{artifacts}}

and there is another important part, and this is test metadata - our tool solves this via --metadata option, so a do not expect to have test set defined via standard test role. But this is more less implementation detail how we gather test sets

Edited 6 years ago by jscotka

astepano commented 6 years ago

Hi.

Installing additional packages is very often task. It is part of common role now. Will be released with next build.

About "TEST_SUBJECT" can we discuss it face-to-face?

ttomecek commented on line 16 of roles/standard-test-mtf/tasks/main.yml 6 years ago

the main difference b/w your role and the one I propose that you explicitly exec mtf while I want to leave this up to the user so the user is not locked