PR#345: Include logs in results.yml, use consistent prefix - standard-test-roles

standard-test-roles

#345 Include logs in results.yml, use consistent prefix

Merged 4 years ago by astepano. Opened 4 years ago by psss.

psss/standard-test-roles logs into master

Include logs in results.yml, use consistent prefix

Petr Šplíchal • 4 years ago

4fb1d4c

roles/standard-test-basic/files/run-basic-test

file modified

+13 -6

		`@@ -116,15 +116,22 @@`
		`fi`
		`local run_journal="$STR_ARTIFACTS_DIR/test.log"`
		`echo "${status} $STR_TEST_NAME" >> "$run_journal"`
		`- # Handle results.yml file`
		`+ # Handle results.yml file, rename logs`
		`local results="$STR_ARTIFACTS_DIR/results.yml"`
		`local result=$(echo $status \| tr '[:upper:]' '[:lower:]')`
		`test -f "$results" \|\| echo 'results:' > "$results"`
		`- echo "- {result: $result, test: $STR_TEST_NAME}" >> "$results"`
		`- if [ -e "$logfile_stdout" ]; then`
		`- local new_logfile_stdout="$(dirname "$logfile_stdout")/${status}_$(basename "$logfile_stdout")"`
		`- mv -f "$logfile_stdout" "$new_logfile_stdout"`
		`- fi`
		`+ printf '%s\n' '' \`
		`+ "- test: $STR_TEST_NAME" \`
		`+ " result: $result" \`
		`+ " logs:" \`
		`+ >> "$results"`
		`+ for log in "$logfile_stdout" "$logfile_stderr"; do`
		`+ if [ -f "$log" ]; then`
		`+ local prefixed_log="$STR_ARTIFACTS_DIR/${status}-$(basename $log)"`
		`+ mv -f "$log" "$prefixed_log"`
		`+ echo " - $(basename $prefixed_log)" >> "$results"`
		`+ fi`
		`+ done`
		`exit 0`
		`}`
		`trap clean_exit SIGINT SIGTERM SIGABRT EXIT`

roles/standard-test-beakerlib/files/run-beakerlib-test

file modified

+13 -3

		`@@ -115,12 +115,22 @@`
		`status="ERROR"`
		`fi`
		`echo "$status $STR_BKR_TEST" >> "$STR_ARTIFACTS_DIR/test.log"`
		`- mv "$log_file_path" "$STR_ARTIFACTS_DIR/${status}_${log_file_name}"`
		`- # Handle results.yml file`
		`+ # Handle results.yml file, rename logs`
		`local results="$STR_ARTIFACTS_DIR/results.yml"`
		`local result=$(echo $status \| tr '[:upper:]' '[:lower:]')`
		`test -f "$results" \|\| echo 'results:' > "$results"`
		`- echo "- {result: $result, test: $STR_BKR_TEST}" >> "$results"`
		`+ printf '%s\n' '' \`
		`+ "- test: $STR_BKR_TEST" \`
		`+ " result: $result" \`
		`+ " logs:" \`
		`+ >> "$results"`
		`+ for log in "$log_file_path" "$OUTPUTFILE" "$logfile_stdout" "$logfile_stderr"; do`
		`+ if [ -f "$log" ]; then`
		`+ local prefixed_log="$STR_ARTIFACTS_DIR/${status}-$(basename $log)"`
		`+ mv -f "$log" "$prefixed_log"`
		`+ echo " - $(basename $prefixed_log)" >> "$results"`
		`+ fi`
		`+ done`
		`# Exit code == 0, no matter of the test result.`
		`exit 0`
		`}`

psss commented 4 years ago

Include all log file names in the results.yml file.
Prefix all log files so that investigation is easier.

psss commented 4 years ago

Example results.yml file:

results:

    - test: wizard/bad
      result: fail
      logs:
        - FAIL-wizard-bad.log
        - FAIL-wizard-bad-out.log
        - FAIL-wizard-bad-err.log

    - test: wizard/good
      result: pass
      logs:
        - PASS-wizard-good.log
        - PASS-wizard-good-out.log
        - PASS-wizard-good-err.log

    - test: wizard/problem
      result: error
      logs:
        - ERROR-wizard-problem.log
        - ERROR-wizard-problem-out.log
        - ERROR-wizard-problem-err.log

bgoncalv commented 4 years ago

Thanks, it works for me and the YAML is valid, I'm just wondering if the indentation is the best one.

I'd suggest to do something like:

    test -f "$results" || echo 'results:' > "$results"
    printf '%s\n' \
        "  - test: $STR_BKR_TEST" \
        "    result: $result"      \
        "    logs:"  \
        >> "$results"
    for log in "$log_file_path" "$OUTPUTFILE" "$logfile_stdout" "$logfile_stderr"; do
        if [ -f "$log" ]; then
            local prefixed_log="$STR_ARTIFACTS_DIR/${status}-$(basename $log)"
            mv -f "$log" "$prefixed_log"
            echo "      - $(basename $prefixed_log)" >> "$results"
        fi
    done

so the results.yml would be shift to left by 2 white spaces (but it might need some change if we want extra line between each test case):

results:
  - test: shell/func
    result: pass
    logs:
      - PASS-shell-func.log
      - PASS-shell-func-out.log
      - PASS-shell-func-err.log
  - test: shell/login
    result: pass
    logs:
      - PASS-shell-login.log
      - PASS-shell-login-out.log
      - PASS-shell-login-err.log
  - test: shell/smoke
    result: pass
    logs:
      - PASS-shell-smoke.log
      - PASS-shell-smoke-out.log
      - PASS-shell-smoke-err.log

rebased onto 4fb1d4c

4 years ago

psss commented 4 years ago

Thanks for the review and feedback, @bgoncalv. Makes sense to get rid of the extra spaces. Pull request updated. This is how results.yml looks now for beakerlib:

results:

- test: wizard/bad
  result: fail
  logs:
  - FAIL-wizard-bad.log
  - FAIL-wizard-bad-out.log
  - FAIL-wizard-bad-err.log

- test: wizard/good
  result: pass
  logs:
  - PASS-wizard-good.log
  - PASS-wizard-good-out.log
  - PASS-wizard-good-err.log

- test: wizard/problem
  result: error
  logs:
  - ERROR-wizard-problem.log
  - ERROR-wizard-problem-out.log
  - ERROR-wizard-problem-err.log

and for the basic role:

results:

- test: good
  result: pass
  logs:
  - PASS-str_good.log
  - PASS-str_good-err.log

- test: bad
  result: fail
  logs:
  - FAIL-str_bad.log
  - FAIL-str_bad-err.log

- test: missing
  result: error
  logs:
  - ERROR-str_missing.log
  - ERROR-str_missing-err.log

I've added an extra line between test cases to make the output a bit more readable.

Edited 4 years ago by psss

bgoncalv commented 4 years ago

thanks, it looks good to me

psss commented 4 years ago

Thanks for quick review, @bgoncalv.