Second, and follow test-runs takes into consideration a result from previous tests run.
Consider next scenario:
Problem is in:
- name: Check the results shell: grep "^FAIL" {{ remote_artifacts }}/test.log register: test_fails failed_when: test_fails.stdout or test_fails.stderr
In my case, results for all test runs are accumulated at:
cat /tmp/artifacts/test.log PASS test PASS test FAIL test FAIL test
Each test run should not take into consideration previous test-runs.
standard-test-roles-2.4-1.fc26.noarch
[root@andrei-2017-09-22-141943 tests]# cat tests.yml --- # Tests for Classic - hosts: localhost roles: - role: standard-test-beakerlib tags: - classic tests: - test
.
[root@andrei-2017-09-22-141943 tests]# cat test/runtest.sh #!/bin/bash # vim: dict=/usr/share/beakerlib/dictionary.vim cpt=.,w,b,u,t,i,k # Include Beaker environment . /usr/bin/rhts-environment.sh || exit 1 . /usr/share/beakerlib/beakerlib.sh || exit 1 rlIsRHEL 7 && OPTIONS="--insecure" rlJournalStart rlPhaseStart "FAIL" "$TEST: Dummy test." # Use command: false or true. rlRun -t -l "true" # run with true + run tests, change to false + run tests, change to true + run test rlPhaseEnd rlJournalPrintText rlJournalEnd
[root@andrei-2017-09-22-141943 tests]# cat test/Makefile export TEST=/CoreOS/basictest export TESTVERSION=1.0 BUILT_FILES= FILES=$(METADATA) runtest.sh Makefile PURPOSE .PHONY: all install download clean run: $(FILES) build ./runtest.sh build: $(BUILT_FILES) test -x runtest.sh || chmod a+x runtest.sh clean: rm -f *~ $(BUILT_FILES) include /usr/share/rhts/lib/rhts-make.include $(METADATA): Makefile @echo "Owner: RedHat <RedHat@redhat.com>" > $(METADATA) @echo "Name: $(TEST)" >> $(METADATA) @echo "TestVersion: $(TESTVERSION)" >> $(METADATA) @echo "Path: $(TEST_DIR)" >> $(METADATA) @echo "Description: various download methods with non-root user" >> $(METADATA) @echo "Type: Sanity" >> $(METADATA) @echo "TestTime: 5m" >> $(METADATA) @echo "RunFor: curl" >> $(METADATA) @echo "Requires: curl" >> $(METADATA) @echo "Priority: Normal" >> $(METADATA) @echo "License: GPLv2" >> $(METADATA) @echo "Confidential: no" >> $(METADATA) @echo "Destructive: no" >> $(METADATA) rhts-lint $(METADATA)
Hey @astepano. The test specification requires that the artifacts directory is empty when invoking the playbook.
Nevertheless this is indeed annoying. I tried to fix this before ... and it didn't work, because multiple standard-test-beakerlib (and other) roles can be invoked within a single test suite.
@merlinm Do you think we should have an inventory script clear out the artifacts? That would be a major hack.
What do you think about a handy invocation script (just a few lines) in standard-test-roles ... that performs these tasks for people hacking on the tests?
The problem is that, even I specify TEST_ARTIFACTS="$PWD/artifacts" it always uses next directory: /tmp/artifacts/
# cat /tmp/artifacts/test.log PASS test PASS test FAIL test FAIL test
Currently it is necessary to cleanup /tmp/artifacts
Ah yes, because when running the tests locally there's the intermediate step there.
@merlinm ... what do you think about having remote_artifacts always mirror the main artifacts directory. It seems that within an Atomic Host, or container the {{ remote_artifacts }} can be an arbitrary path ... even if {{ artifacts }} it's rather deep and convoluted.
@stefw I really don't care for the idea of mirroring artifacts and remote_artifacts. They were originally mirrored, but separated because doing so was causing many problems.
artifacts
remote_artifacts
However, we can probably hack the roles a bit to clear the remote_artifacts directory and then set a variable such as remote_artifacts_have_already_been_cleared if that variable has not already been set. Perhaps we could even extend that concept by creating a new role that does such preliminary cleanup in case we want to include additional initialization tasks in the future.
remote_artifacts_have_already_been_cleared
I also got FAILS. Spent 2 days for debugging. I believe we should fix it. <img alt="Fails" src="/standard-test-roles/issue/raw/files/ff3f4d17737ec9ccd0a8df284e6b90a81a4bfc1f815e269cd10f462ebc58f4ff-Fails" />
Login to comment on this ticket.