There has been a discussion about how to improve reporting results in the Standard Test Interface so that it is more easy to distinguish test failure from infrastructure error:

• https://pagure.io/fedora-ci/general/issue/18

Let's update the standard-test-roles according to the agreed result: New file results.yml has been defined which should be stored in the artifacts directory as the test.log file and will contain test results in the following format:

results:
- {result: pass, test: shell/smoke}
- {result: fail, test: shell/login}
- {result: error, test: shell/func}

Or we can support the extended version including links to logs:

results:
- {result: pass, test: shell/smoke, log: /tmp/artifacts/PASS_shell-smoke.log}
- {result: fail, test: shell/login, log: /tmp/artifacts/PASS_shell-login.log}
- {result: error, test: shell/func, log: /tmp/artifacts/FAIL_shell-func.log}

Plus, ansible should return non-zero status only for infrastructure errors. Here are some examples to make the expected implementation clear:

• It was not possible to execute test ---> playbook returns 1
• Unabled to install packages required by test ---> playbook returns 1
• Test successfully executed and fails ---> playbook returns 0
• Test successfully executed and passes ---> playbook returns 0
• Test successfully executed and errors ---> playbook returns 0