| |
@@ -4,7 +4,7 @@
|
| |
|
| |
Standard Discovery, Staging and Invocation of Integration Tests.
|
| |
|
| |
- Version: `1.1.0`
|
| |
+ Version: `2.0.0`
|
| |
|
| |
== Summary ==
|
| |
|
| |
@@ -28,7 +28,9 @@
|
| |
that allows varied test frameworks as well as CI system infrastructure to interoperate.
|
| |
The integration tests outlast the implementation details of either the frameworks they're written in or the CI systems running them.
|
| |
|
| |
- == Terminology ==
|
| |
+ == Definitions ==
|
| |
+
|
| |
+ === Terminology ===
|
| |
|
| |
Test Subject::
|
| |
The items that are to be tested.
|
| |
@@ -76,6 +78,30 @@
|
| |
For example ansible is run on the _test runner_ and tests are executed on the managed host.
|
| |
Usually a stable version of OS is used for _test runner_.
|
| |
|
| |
+ === Results Format ===
|
| |
+
|
| |
+ The following format should be used to report results of individual tests in the `results.yml` file:
|
| |
+
|
| |
+ results:
|
| |
+ - {result: pass, test: test1, logs: [test1.log]}
|
| |
+ - {result: fail, test: test2, logs: [test2.log, debug.log]}
|
| |
+ - {result: error, test: test3, logs: [test3.log, debug.log, error.log]}
|
| |
+
|
| |
+ result:: Test result. One of `pass`, `fail` or `error`. Mandatory.
|
| |
+ test:: Test name. A unique identifier. Mandatory.
|
| |
+ logs::
|
| |
+ One or more logs with detailed test output. Optional.
|
| |
+ Path should be relative to the artifacts directory.
|
| |
+ Some user interfaces might show only a single log by default.
|
| |
+ In that case first log from the list should be presented to the user.
|
| |
+
|
| |
+ The `result` field can contain following values:
|
| |
+
|
| |
+ pass:: Test has successfully finished and passed.
|
| |
+ fail:: Test has successfully finished and failed.
|
| |
+ error:: There has been a problem with test execution.
|
| |
+
|
| |
+
|
| |
== Responsibilities ==
|
| |
|
| |
The *Testing System* is responsible to:
|
| |
@@ -186,7 +212,8 @@
|
| |
. MUST execute Ansible with inventory set to the full path of the file or directory `tests/inventory` if it exists.
|
| |
.. If the `tests/inventory` file doesn't exist, then the following inventory SHOULD be used as a default:<br> `/usr/share/ansible/inventory`
|
| |
. MUST execute the playbook as root.
|
| |
- . MUST examine the exit code of the playbook. A zero exit code is successful _test result_, non-zero is failure.
|
| |
+ . MUST examine the exit code of the playbook. A zero exit code means tests completed successfully, non-zero means a problem with running tests.
|
| |
+ . MUST examine the file `results.yml` in the `artifacts` folder to detect whether tests passed of failed.
|
| |
. MUST treat the file `test.log` in the `artifacts` folder as the main readable output of the test.
|
| |
. SHOULD place the textual stdout/stderr of the `ansible-playbook` command in the `ansible.log` file in the `artifacts` folder.
|
| |
. SHOULD treat the contents of the `artifacts` folder as the _test artifacts_.
|
| |
@@ -198,7 +225,8 @@
|
| |
. MUST provision the _test subject_ listed in the `subjects` variable appropriately for its playbook name (described above) and MUST fail if this is not possible.
|
| |
. MUST place the main readable output of the _test suite_ into a `test.log` file in the `artifacts` variable folder. This MUST happen even if some of the test suites fail.
|
| |
. SHOULD place additional _test artifacts_ in the folder defined in the `artifacts` variable.
|
| |
- . MUST return a zero exit code of the playbook if the _test result_ is a pass, or a non-zero exit code if the _test result_ is a fail.
|
| |
+ . MUST return a zero exit code of the playbook if tests have been executed successfully, or a non-zero exit code if failed to run any test (e.g. because of an infrastructure error).
|
| |
+ . MUST create a `results.yml` file in the `artifacts` directory with test results in the _results format_ defined above.
|
| |
|
| |
If an inventory file or script exists, it:
|
| |
|
| |
psss
commented 5 years ago
Introduces a new file
results.ymlwhich allows to clearlydistinguish test failures from infrastructure errors. See the
following ticket for the whole context and motivation:
https://pagure.io/fedora-ci/general/issue/18