#14 WIP: Example for CI of tests
Opened 2 years ago by lzachar. Modified 2 years ago
fedora-ci/ lzachar/metadata master  into  master

@@ -0,0 +1,32 @@ 

+ # CI for changes of tests found in PR

+ /test:

+     # Detects what tests, preparation steps for each of them  

+     discover:

+         how: beakerlib-test

+     execute:

+         how: restraint

+         checkers: # All available are enabled by default

+             # Sends FAF report if happened during test run

+             abrt: False

+             # Compares SUT before/after test is run, fails when SUT has changed

+             inspector: False

+     /dist-git-pr:

+         /rhel-7-normal:

+             distro: RHEL-7-latest

+         /rhel-7-rh-python36:

+             distro: RHEL-7-latest

+             discover:

+                 how: scl-beakerlib-test

+                 options:

+                     collection: rh-python36

+         /rhel-8-platform:

+             distro: RHEL-8-latest

+             environment:

+                 PYTHON: /usr/libexec/platform-python

+                 PACKAGES: "platform-python python36"

+         /rhel-8-python27:

+             distro: RHEL-8-latest

+             environment:

+                 PYTHON: python2

+                 PACKAGES: "python27"

+ 

First iteration, please propose better names ;)

Pull-Request has been closed by lzachar

2 years ago

This would be restraint - sclrun is involved in "discovery" stage, where it yields a job XML. After that, it plays no role in running the tests.

How is beakerlib-test expected to work? IIUIC, the test to run is the test changed by PR, right? So it's something like "itself"?

Pull-Request has been reopened by lzachar

2 years ago

It specifies 'discovery' - on PR can change multiple files, between multiple tests. So it should detect all changed tests based on touched files. It is beakerli-test type because that means ' one test is one Makefile per directory in current / parent directory(ies) of changed file(s)'

Once again, I have no idea how to react on comment on comment in code view.

How is beakerlib-test expected to work? IIUIC, the test to run is the test changed by PR, right? So it's something like "itself"?

It specifies 'discovery' - on PR can change multiple files, between multiple tests. So it should detect all changed tests based on touched files. It is beakerli-test type because that means ' one test is one Makefile per directory in current / parent directory(ies) of changed file(s)'

Not really. Scl run specifies SUT deployment and fills test environment variables.

Not really. Scl run specifies SUT deployment and fills test environment variables.

True, but that in my world is still called "discovery" :) Find the tests, write down what environment (compose, env variables, RAM size, etc.) they need, maybe create groups of them if you can, to save some resources, and pass the paper down the stream. Following stages take care of provisioning necessary environments, setting things up, and "execution" would just make the prescription happen - running what needs to be run, setting env variables that must be set. Why wouldn't it need to call sclrun once again? It has all required information for running the tests. Would you have a use in which it would be necessary to involve sclrun in the execution stage?

True, but that in my world is still called "discovery" :)

If you spell it out this way... It makes sense. But discovery for me is 'what tests should be run'. Not HOW they should be run, that's environment deployment step, imho.

True, but that in my world is still called "discovery" :)

If you spell it out this way... It makes sense. But discovery for me is 'what tests should be run'. Not HOW they should be run, that's environment deployment step, imho.

Understood. However, I strongly believe that FOO=bar /test.sh is a different test than FOO=baz /test.sh, which is part of WHAT tests should run. In my mind, this is a responsibility of "discovery" stage, to write down the necessary, hm, "pseudo-instructions", if you will, that form each test: "the test needs at least 8GB of RAM. Relevant for x86_64 only. Set envvar FOO to bar. Execute file /test.sh".

HOW is the responsibility of the "execution" stage - it may need to connect to the remote machine, maybe over SSH, or using restraint? Is it going to run something like ssh 'FOO=bar /test.sh' root@remote.com, or would it prepare a shell script somewhere in /tmp which does more things before/after executing the script? Or it may create whole Beaker job just to execute a simple shell script. Who knows. That's HOW as I understand it. But it would always follow our "pseudo-instructions", prepared by its "discovery" predecesor, to set variables and run things.

Imagine we'd involve sclrun in the "execute" stage, to "fill test environment variables" - you'd have to teach each execution method you'd wish to use with this "discovery" step how to use sclrun and how to connect it with execution's specific way of setting variables for tests. On the other hand, I'm proposing to draw a line there, to let these two stages communicate in a sort of lingua franca - no matter HOW you want to run the "test", these methods will always share common properties: variables to set, path to chdir to, hardware requirements, required packages/collections/containers/..., the actual test, something to execute, and so on. Specify this in a transparent, agnostic way, and you can use any runner and provisioner to run your test. No need to tweak runner to talk to sclrun.

(Sorry for the long post, but I strongly believe we should avoid the temptation for finding shortcuts or making some tools more closer than others... Sort of spreading the religion. Not related to your PR per se, I'm rather trying to make my ideas clear, to avoid being suspected of nitpicking. I really have some very specific picture in my mind, not trying find typos in your code :)

Understood. However, I strongly believe that FOO=bar /test.sh is a different test than FOO=baz /test.sh, which is part of WHAT tests should run. In my mind, this is a responsibility of "discovery" stage, to write down the necessary, hm, "pseudo-instructions", if you will, that form each test: "the test needs at least 8GB of RAM. Relevant for x86_64 only. Set envvar FOO to bar. Execute file /test.sh".

OK, with this definition of "discovery" I agree. Does 'scl enable <scl> -- ' as necessary step before executing the test will fall into this category?

What about adding DEPLOY phase? E.g. I need to run this commands or ansible playbook before tests are executed (e.g. non-trivial product installation).
Similar for DESTROY (to be prepared for e.g. de-registration)

Also, we plan to support 'test-inspector', 'abrt-checker' - but one should be able to disable them - what configuration should we have for these?

rebased onto d23ea04

2 years ago

...

OK, with this definition of "discovery" I agree. Does 'scl enable <scl> -- ' as necessary step before executing the test will fall into this category?

I think it does, it should be part of the "instructions" yielded by the discovery step, as a sort of a requirement. Just like we want to install some required package, we might need to execute commands, or maybe even explicitly enable collections. Discovery should make a corresponding note in its report.

What about adding DEPLOY phase? E.g. I need to run this commands or ansible playbook before tests are executed (e.g. non-trivial product installation).

Yup, we have it, provision and prepare steps. Again, discovery step should write down what needs to be done.

Similar for DESTROY (to be prepared for e.g. de-registration)

This one we don't have yet. I see the point. Although I think it does not have to be as step-ish as "discovery", "provision", "prepare", "execute" and "report", maybe just a "responsibility" of the "execute" step. The steps probably don't map 1:1 to existing pipelines, e.g. there should be a counterpart for "provision" as well, defined as a step. Maybe it'd be good to specify it as well, hm. Needs discussion.

Also, we plan to support 'test-inspector', 'abrt-checker' - but one should be able to disable them - what configuration should we have for these?

I don't know how test-inspector works. I have some idea about abrt-checker, is it a "post-test" step, checking for coredumps & other stuff, marking test "FAIL" if it found anything wrong? I'd suppose this would be part of the "execution" step, similar to what we Restraint does for us when it comes to AVC denials. Does that make sense?

@lzachar the first version of the steps definition (in progress) here:
https://pagure.io/fedora-ci/metadata/blob/18774b8/f/l2/README.md
It might help you to get a better idea about the individual steps.

@lzachar, will there be som update here? or is this all set now?

Sorry, I couldn't get to this. Hopefully after PTO I can provide an update.

Metadata