There has been a discussion about how to improve reporting results in the Standard Test Interface so that it is more easy to distinguish test failure from infrastructure error:
Let's update the specification according to the agreed result: New file results.yml has been defined which will contain test results in the following format:
results.yml
results: - {result: pass, test: shell/smoke} - {result: fail, test: shell/login} - {result: error, test: shell/func}
Extended version with links to logs:
results: - {result: pass, test: shell/smoke, log: /tmp/artifacts/PASS_shell-smoke.log} - {result: fail, test: shell/login, log: /tmp/artifacts/PASS_shell-login.log} - {result: error, test: shell/func, log: /tmp/artifacts/FAIL_shell-func.log}
And, for completeness, the expanded multiline version:
results: - result: pass test: shell/smoke log: /tmp/artifacts/PASS_shell-smoke.log
Plus, ansible should return non-zero status only for infrastructure errors. For example, when the test is successfully executed and fails, running the playbook should return 0.
Metadata Update from @psss: - Issue assigned to psss
Metadata Update from @psss: - Issue tagged with: Standard Test Interface
Standard Test Interface update ready for review: https://pagure.io/fedora-ci/docs/pull-request/5
Updated test examples ready for review: https://pagure.io/fedora-ci/docs/pull-request/26
Both pull requests have been merged.
Metadata Update from @psss: - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.