#3 Standard Test Interface: Improve results reporting
Closed 5 years ago by psss. Opened 5 years ago by psss.

There has been a discussion about how to improve reporting results in the Standard Test Interface so that it is more easy to distinguish test failure from infrastructure error:

Let's update the specification according to the agreed result: New file results.yml has been defined which will contain test results in the following format:

results:
- {result: pass, test: shell/smoke}
- {result: fail, test: shell/login}
- {result: error, test: shell/func}

Extended version with links to logs:

results:
- {result: pass, test: shell/smoke, log: /tmp/artifacts/PASS_shell-smoke.log}
- {result: fail, test: shell/login, log: /tmp/artifacts/PASS_shell-login.log}
- {result: error, test: shell/func, log: /tmp/artifacts/FAIL_shell-func.log}

And, for completeness, the expanded multiline version:

results:
- result: pass
  test: shell/smoke
  log: /tmp/artifacts/PASS_shell-smoke.log

Plus, ansible should return non-zero status only for infrastructure errors. For example, when the test is successfully executed and fails, running the playbook should return 0.


Metadata Update from @psss:
- Issue assigned to psss

5 years ago

Metadata Update from @psss:
- Issue tagged with: Standard Test Interface

5 years ago

Both pull requests have been merged.

Metadata Update from @psss:
- Issue status updated to: Closed (was: Open)

5 years ago

Log in to comment on this ticket.

Metadata