#65 standard-test-beakerlib role, "Check the results" takes into consideration old test-runs.
Opened 6 years ago by astepano. Modified 6 years ago

Description of problem

Second, and follow test-runs takes into consideration a result from previous tests run.

Consider next scenario:

  1. ansible-playbook --tags classic tests.yml (suppose this test run rightfully fails)
  2. ansible-playbook --tags classic tests.yml (this one doesn't fail, but ansible will report is a failed test)

Problem is in:

  - name: Check the results
    shell: grep "^FAIL" {{ remote_artifacts }}/test.log
    register: test_fails
    failed_when: test_fails.stdout or test_fails.stderr

Actual results

In my case, results for all test runs are accumulated at:

cat /tmp/artifacts/test.log
PASS test
PASS test
FAIL test
FAIL test

Expected results

Each test run should not take into consideration previous test-runs.

Version-Release number of selected component

standard-test-roles-2.4-1.fc26.noarch

Steps to Reproduce

[root@andrei-2017-09-22-141943 tests]# cat tests.yml 
---
# Tests for Classic
- hosts: localhost
  roles:
  - role: standard-test-beakerlib
    tags:
    - classic
    tests:
    - test

.

[root@andrei-2017-09-22-141943 tests]# cat test/runtest.sh 
#!/bin/bash
# vim: dict=/usr/share/beakerlib/dictionary.vim cpt=.,w,b,u,t,i,k

# Include Beaker environment
. /usr/bin/rhts-environment.sh || exit 1
. /usr/share/beakerlib/beakerlib.sh || exit 1

rlIsRHEL 7 && OPTIONS="--insecure"

rlJournalStart

    rlPhaseStart "FAIL" "$TEST: Dummy test."
        # Use command: false or true.
        rlRun -t -l "true"
        # run with true + run tests, change to false + run tests, change to true + run test
    rlPhaseEnd

    rlJournalPrintText

rlJournalEnd

.

[root@andrei-2017-09-22-141943 tests]# cat test/Makefile 
export TEST=/CoreOS/basictest
export TESTVERSION=1.0

BUILT_FILES=

FILES=$(METADATA) runtest.sh Makefile PURPOSE

.PHONY: all install download clean

run: $(FILES) build
    ./runtest.sh

build: $(BUILT_FILES)
    test -x runtest.sh || chmod a+x runtest.sh

clean:
    rm -f *~ $(BUILT_FILES)


include /usr/share/rhts/lib/rhts-make.include

$(METADATA): Makefile
    @echo "Owner:           RedHat <RedHat@redhat.com>" > $(METADATA)
    @echo "Name:            $(TEST)" >> $(METADATA)
    @echo "TestVersion:     $(TESTVERSION)" >> $(METADATA)
    @echo "Path:            $(TEST_DIR)" >> $(METADATA)
    @echo "Description:     various download methods with non-root user" >> $(METADATA)
    @echo "Type:            Sanity" >> $(METADATA)
    @echo "TestTime:        5m" >> $(METADATA)
    @echo "RunFor:          curl" >> $(METADATA)
    @echo "Requires:        curl" >> $(METADATA)
    @echo "Priority:        Normal" >> $(METADATA)
    @echo "License:         GPLv2" >> $(METADATA)
    @echo "Confidential:    no" >> $(METADATA)
    @echo "Destructive:     no" >> $(METADATA)

    rhts-lint $(METADATA)

Hey @astepano. The test specification requires that the artifacts directory is empty when invoking the playbook.

Nevertheless this is indeed annoying. I tried to fix this before ... and it didn't work, because multiple standard-test-beakerlib (and other) roles can be invoked within a single test suite.

@merlinm Do you think we should have an inventory script clear out the artifacts? That would be a major hack.

What do you think about a handy invocation script (just a few lines) in standard-test-roles ... that performs these tasks for people hacking on the tests?

The problem is that, even I specify TEST_ARTIFACTS="$PWD/artifacts" it always uses next directory: /tmp/artifacts/

# cat /tmp/artifacts/test.log 
PASS test
PASS test
FAIL test
FAIL test

Currently it is necessary to cleanup /tmp/artifacts

Ah yes, because when running the tests locally there's the intermediate step there.

@merlinm ... what do you think about having remote_artifacts always mirror the main artifacts directory. It seems that within an Atomic Host, or container the {{ remote_artifacts }} can be an arbitrary path ... even if {{ artifacts }} it's rather deep and convoluted.

@stefw I really don't care for the idea of mirroring artifacts and remote_artifacts. They were originally mirrored, but separated because doing so was causing many problems.

However, we can probably hack the roles a bit to clear the remote_artifacts directory and then set a variable such as remote_artifacts_have_already_been_cleared if that variable has not already been set. Perhaps we could even extend that concept by creating a new role that does such preliminary cleanup in case we want to include additional initialization tasks in the future.

Login to comment on this ticket.

Metadata
Attachments 1
Attached 6 years ago View Comment