PR#344: Clearly show overall result of the testing - standard-test-roles

standard-test-roles

#344 Clearly show overall result of the testing

Merged 4 years ago by astepano. Opened 4 years ago by psss.

psss/standard-test-roles result into master

Clearly show overall result of the testing

Petr Šplíchal • 4 years ago

820ab1f

roles/standard-test-basic/tasks/main.yml

file modified

+18 -6

		`@@ -25,15 +25,27 @@`
		`# Can't go in block. See`
		`# https://github.com/ansible/ansible/issues/20736`
		`- name: Check the results`
		`- shell: grep "^ERROR" {{ remote_artifacts }}/test.log`
		`- register: test_error`
		`- # Never fail at this step. Just store result of tests.`
		`- failed_when: False`
		`+ shell: \|`
		`+ log="{{ remote_artifacts }}/test.log"`
		`+ if [ ! -f "$log" ]; then`
		`+ echo ERROR`
		`+ echo "Test results not found." 1>&2`
		`+ elif grep ^ERROR "$log" 1>&2; then`
		`+ echo ERROR`
		`+ elif grep ^FAIL "$log" 1>&2; then`
		`+ echo FAIL`
		`+ elif grep -q ^PASS "$log"; then`
		`+ echo PASS`
		`+ else`
		`+ echo ERROR`
		`+ echo "No test results found." 1>&2`
		`+ fi`
		`+ register: test_results`

		`- name: Set role result`
		`set_fact:`
		`- role_result_error: "{{ (test_error.stdout\|d\|length > 0) or (test_error.stderr\|d\|length > 0) }}"`
		`- role_result_msg: "{{ test_error.stdout\|d('test execution error.') }}"`
		`+ role_result: "{{ test_results.stdout }}"`
		`+ role_message: "{{ test_results.stderr\|d('test execution error.') }}"`

		`- include_role:`
		`name: str-common-final`

roles/standard-test-beakerlib/tasks/main.yml

file modified

+18 -6

		`@@ -63,15 +63,27 @@`
		`# Can't go in block. See`
		`# https://github.com/ansible/ansible/issues/20736`
		`- name: Check the results`
		`- shell: grep "^ERROR" {{ remote_artifacts }}/test.log`
		`- register: test_error`
		`- # Never fail at this step. Just store result of tests.`
		`- failed_when: False`
		`+ shell: \|`
		`+ log="{{ remote_artifacts }}/test.log"`
		`+ if [ ! -f "$log" ]; then`
		`+ echo ERROR`
		`+ echo "Test results not found." 1>&2`
		`+ elif grep ^ERROR "$log" 1>&2; then`
		`+ echo ERROR`
		`+ elif grep ^FAIL "$log" 1>&2; then`
		`+ echo FAIL`
		`+ elif grep -q ^PASS "$log"; then`
		`+ echo PASS`
		`+ else`
		`+ echo ERROR`
		`+ echo "No test results found." 1>&2`
		`+ fi`
		`+ register: test_results`

		`- name: Set role result`
		`set_fact:`
		`- role_result_error: "{{ (test_error.stdout\|d\|length > 0) or (test_error.stderr\|d\|length > 0) }}"`
		`- role_result_msg: "{{ test_error.stdout\|d('test execution error.') }}"`
		`+ role_result: "{{ test_results.stdout }}"`
		`+ role_message: "{{ test_results.stderr\|d('test execution error.') }}"`

		`- include_role:`
		`name: str-common-final`

roles/str-common-final/tasks/main.yml

file modified

+3 -3

		`@@ -11,8 +11,8 @@`
		`- name: Report role result`
		`vars:`
		`msg: \|`
		`- Tests error: {{ role_result_error\|d('Undefined') }}`
		`- Tests msg: {{ role_result_msg\|d('None') }}`
		`+ Result: {{ role_result\|d('Undefined') }}`
		`+ {{ role_message\|d('None') }}`
		`debug:`
		`msg: "{{ msg.split('\n') }}"`
		`- failed_when: "role_result_error\|bool"`
		`+ failed_when: role_result == 'ERROR'`

psss commented 4 years ago

Report role result task has been updated to show actual result of
testing, which means: ERROR upon any test execution problem, FAIL
when any test fails and PASS if there is at least one test passed.

psss commented 4 years ago

Here's a couple of example outputs. No test results:

fatal: [localhost]: FAILED! => {
    "msg": [
        "Result: ERROR",
        "Test results not found.",
        ""
    ]
}

Tests passed:

ok: [localhost] => {
    "msg": [
        "Result: PASS",
        ""
    ]
}

Tests failed:

ok: [localhost] => {
    "msg": [
        "Result: FAIL",
        "FAIL wizard/bad",
        ""
    ]
}

Test execution error:

fatal: [localhost]: FAILED! => {
    "msg": [
        "Result: ERROR",
        "ERROR wizard/problem",
        ""
    ]
}

astepano commented 4 years ago

Looks reasonable. Waiting for CI check.

bgoncalv commented 4 years ago

thanks, it worked well for me.

astepano commented 4 years ago

Commit 5f7cc58 fixes this pull-request

Pull-Request has been merged by astepano

4 years ago

Pull-Request has been merged by astepano

4 years ago

Metadata

Assignee

None

Tags

No Tags

Flags

jenkins

success (100%)

Build #391 successful (commit: 820ab1fe)

4 years ago

Changes Summary 3

+18 -6

file changed

roles/standard-test-basic/tasks/main.yml

+18 -6

file changed

roles/standard-test-beakerlib/tasks/main.yml

+3 -3

file changed

roles/str-common-final/tasks/main.yml

standard-test-roles

Source Code

#344 Clearly show overall result of the testing Merged 4 years ago by astepano. Opened 4 years ago by psss. psss/standard-test-roles result into master

Metadata

Flags

Changes Summary 3

#344 Clearly show overall result of the testing

Merged 4 years ago by astepano. Opened 4 years ago by psss.

psss/standard-test-roles result into master