#274 tests: Distribute and invoke our own self-tests in Jenkins
Merged a month ago by stefw. Opened a month ago by stefw.
stefw/standard-test-roles tests-upstream  into  master

@@ -12,7 +12,7 @@ 

    - name: Execute the avocado test

      shell: exec 2>>{{ remote_artifacts }}/test.log 1>>{{ remote_artifacts }}/test.log; MODULE=rpm python -m avocado run --job-results-dir {{ remote_artifacts }}/ {{tests|join(' ') }}

  

- - always:

+   always:

    - name: Pull out the logs

      synchronize:

        dest: "{{ artifacts }}/"

@@ -79,7 +79,7 @@ 

  

      merged = Inventory()

  

-     for i in os.listdir(inventory_dir):

+     for i in os.path.exists(inventory_dir) and os.listdir(inventory_dir) or []:

          ipath = os.path.join(inventory_dir, i)

          if not i.startswith("standard-inventory-"):

              continue

file added
+24

@@ -0,0 +1,24 @@ 

+ # Tests for avocado role

+ - hosts: localhost

+   tags:

+   - classic

+   - container

+   roles:

+   - role: standard-test-avocado

+     tests:

+     - /bin/true

+   - role: standard-test-avocado

+     tests:

+     - /bin/false

+     ignore_errors: yes

+   tasks:

+     - name: Read test.log on test environment

+       set_fact:

+         test_log: "{{ lookup('file', artifacts + '/test.log') }}"

+     - name: Check for FAIL on test.log on test environment

+       fail: msg="Could not find expected text on test log"

+       when: >

+           (test_log.find('/bin/false:  FAIL') == -1) or

+           (test_log.find('/bin/true:  PASS') == -1)

+       # We should fail when we can not find FAIL string on log

+       # -1 means string not found

file added
+38

@@ -0,0 +1,38 @@ 

+ - import_playbook: prepare.yml

+ 

+ # Tests for basic role

+ - hosts: localhost

+   tags:

+   - atomic

+   - classic

+   - container

+   roles:

+   - role: standard-test-basic

+     tests:

+     - test-basic-simple

+     - test-basic-parameters:

+         dir: ./

+         run: echo "check parameters on basic role" | grep "check parameters on basic role"

+     required_packages:

+     # Test if we can install required packages

+     - "{{req_pkg}}"

+   tasks:

+   - import_tasks: shared-tasks/artifacts_test_env.yml

+   - import_tasks: shared-tasks/artifacts_test_runner.yml

+ 

+ # Make sure the role behaves correctly if test fails

+ - hosts: localhost

+   tags:

+   - atomic

+   - classic

+   - container

+   roles:

+   - role: standard-test-basic

+     tests:

+     - test-basic-fail

+     ignore_errors: yes

+   tasks:

+   # 'verify_failed_test' tasks should run after 'test-basic-fail'

+   - import_tasks: shared-tasks/verify_failed_test.yml

+   - import_tasks: shared-tasks/artifacts_test_env.yml

+   - import_tasks: shared-tasks/artifacts_test_runner.yml

file added
+35

@@ -0,0 +1,35 @@ 

+ - import_playbook: prepare.yml

+ 

+ # Tests for beakerlib role

+ - hosts: localhost

+   tags:

+   - atomic

+   - classic

+   - container

+   roles:

+   - role: standard-test-beakerlib

+     tests:

+     - test-beakerlib-simple

+     required_packages:

+     - "{{req_pkg}}"

+   tasks:

+   - import_tasks: shared-tasks/artifacts_test_env.yml

+   - import_tasks: shared-tasks/artifacts_test_runner.yml

+ 

+ 

+ # Make sure the role behaves correctly if test fails

+ - hosts: localhost

+   tags:

+   - atomic

+   - classic

+   - container

+   roles:

+   - role: standard-test-beakerlib

+     tests:

+     - test-beakerlib-fail

+     ignore_errors: yes

+   tasks:

+   # 'tests_verify_failed_test' tasks should run after 'test-beakerlib-fail'

+   - import_tasks: shared-tasks/verify_failed_test.yml

+   - import_tasks: shared-tasks/artifacts_test_env.yml

+   - import_tasks: shared-tasks/artifacts_test_runner.yml

file added
+14

@@ -0,0 +1,14 @@ 

+ - hosts: localhost

+   tags:

+     - always

+   tasks:

+     - name: Get inventory with empty input

+       command: merge-standard-inventory

+       environment:

+         # Reset vars to default values

+         TEST_SUBJECTS:

+       register: inventory

+       delegate_to: localhost

+     - name: Assert merge-standard-inventory produces correct JSON inventory (empty)

+       fail:

+       when: inventory.stdout != '{}'

@@ -0,0 +1,13 @@ 

+ #!/bin/sh

+ 

+ #

+ # This invokes the tests using the local inventory and roles rather

+ # than what's installed on system. This is aslight deviation from

+ # what's specified in the standard test interface

+ #

+ 

+ cd $(dirname $0)/..

+ 

+ rm -rf /tmp/artifacts/ /tmp/local-artifacts/

+ TEST_ARTIFACTS=/tmp PATH=$PWD/scripts:$PATH ANSIBLE_INVENTORY=$PWD/inventory \

+ 	ansible-playbook --extra-vars=artifacts=/tmp/local-artifacts --tags=classic tests/tests.yml

file added
+13

@@ -0,0 +1,13 @@ 

+ # Preparation steps

+ - hosts: localhost

+   vars:

+     # Must be the same as in str-common/defaults/main.yml

+     artifacts: "{{ lookup('env', 'TEST_ARTIFACTS')|default('./artifacts', true) }}"

+     remote_artifacts: /tmp/artifacts/

+   tasks:

+     - import_tasks: shared-tasks/artifacts.yml

+       tags:

+         - always

+     - import_tasks: shared-tasks/req-pkg.yml

+       tags:

+         - always

file added
+33

@@ -0,0 +1,33 @@ 

+ - import_playbook: prepare.yml

+ 

+ # Tests for rhts role only runs on classic tag

+ - hosts: localhost

+   tags:

+   - classic

+   roles:

+   - role: standard-test-rhts

+     tests:

+     - test-rhts-simple

+     required_packages:

+     # Test if we can install required packages

+     - "{{req_pkg}}"

+   tasks:

+   - import_tasks: shared-tasks/artifacts_test_env.yml

+   - import_tasks: shared-tasks/artifacts_test_runner.yml

+ 

+ # Make sure the role behaves correctly if test fails

+ - hosts: localhost

+   tags:

+   - classic

+   roles:

+   - role: standard-test-rhts

+     tags:

+     - classic

+     tests:

+     - test-rhts-fail

+     ignore_errors: yes

+   tasks:

+   # 'verify_failed_test' tasks should run after 'testi-rhts-fail'

+   - import_tasks: shared-tasks/verify_failed_test.yml

+   - import_tasks: shared-tasks/artifacts_test_env.yml

+   - import_tasks: shared-tasks/artifacts_test_runner.yml

@@ -0,0 +1,1 @@ 

+ ../roles 

\ No newline at end of file

@@ -0,0 +1,9 @@ 

+ # STR requires that artifacts dir must be absent

+ - name: "Check if {{artifacts}} is absent at test-environment."

+   stat:

+     path: "{{remote_artifacts}}"

+   register: st_te

+ 

+ - fail:

+     msg: "Remove {{remote_artifacts}} before test run at test-environment."

+   when: st_te.stat.exists

@@ -0,0 +1,9 @@ 

+ # These tasks should run after as test that fails

+ - name: "Check if {{remote_artifacts}} was created properly on test environment"

+   shell: "ls {{remote_artifacts}}/test.log"

+ 

+ - name: "Clean up {{remote_artifacts}} to not affect other tests on test environment"

+   file:

+     state: absent

+     path: "{{remote_artifacts}}/"

+ 

@@ -0,0 +1,10 @@ 

+ # These tasks should run after as test that fails

+ - name: "Check if {{artifacts}} was created properly on test runner"

+   shell: "ls {{artifacts}}/test.log"

+   delegate_to: localhost

+ 

+ - name: "Clean up {{artifacts}} to not affect other tests on test runner"

+   file:

+     state: absent

+     path: "{{artifacts}}/"

+   delegate_to: localhost

@@ -0,0 +1,12 @@ 

+ - import_role:

+     name: str-common-pkgs

+     tasks_from: inspect.yml

+ 

+ - set_fact:

+     req_pkg: zsh

+ 

+ - name: Remove a test package from test-environment

+   package:

+     name: "{{req_pkg}}"

+     state: absent

+   when: not is_atomic

@@ -0,0 +1,11 @@ 

+ # These tasks should run after as test that fails

+ - name: Read test.log on test environment

+   shell: "cat {{remote_artifacts}}/test.log"

+   register: test_log

+ 

+ - name: Check for FAIL on test.log on test environment

+   fail: msg="Could not find FAIL on test log"

+   when: test_log.stdout.find("FAIL") == -1

+   # We should fail when we can not find FAIL string on log

+   # -1 means string not found

+ 

@@ -0,0 +1,5 @@ 

+ #!/bin/bash

+ echo "Running a command that should fail"

+ 

+ false

+ 

@@ -0,0 +1,5 @@ 

+ #!/bin/bash

+ PACKAGE="bash"

+ 

+ echo "Using $PACKAGE version:"

+ rpm -q $PACKAGE

@@ -0,0 +1,14 @@ 

+ #!/bin/bash

+ 

+ # Include Beaker environment

+ . /usr/share/beakerlib/beakerlib.sh || exit 1

+ 

+ PACKAGE="bash"

+ 

+ rlJournalStart

+     rlPhaseStartTest "Simple beakerlib test"

+         # Running a command that should fail

+         rlRun "false"

+     rlPhaseEnd

+ rlJournalPrintText

+ rlJournalEnd

@@ -0,0 +1,12 @@ 

+ #!/bin/bash

+ # Include Beaker environment

+ . /usr/share/beakerlib/beakerlib.sh || exit 1

+ 

+ PACKAGE="bash"

+ 

+ rlJournalStart

+     rlPhaseStartTest "Simple beakerlib test"

+         rlRun "ls /"

+     rlPhaseEnd

+ rlJournalPrintText

+ rlJournalEnd

@@ -0,0 +1,34 @@ 

+ export TESTVERSION=1.0

+ 

+ BUILT_FILES=

+ 

+ FILES=$(METADATA) runtest.sh Makefile

+ 

+ .PHONY: all install clean

+ 

+ run: $(FILES) build

+ 	./runtest.sh

+ 

+ build: $(BUILT_FILES)

+ 	test -x runtest.sh || chmod a+x runtest.sh

+ 

+ clean:

+ 	rm -f *~ $(BUILT_FILES)

+ 

+ 

+ include /usr/share/rhts/lib/rhts-make.include

+ 

+ $(METADATA): Makefile

+ 	@echo "Owner:           Bruno Goncalves <bgoncalv@redhat.com>" > $(METADATA)

+ 	@echo "Name:            $(TEST)" >> $(METADATA)

+ 	@echo "TestVersion:     $(TESTVERSION)" >> $(METADATA)

+ 	@echo "Path:            $(TEST_DIR)" >> $(METADATA)

+ 	@echo "Description:     fail rhts test role" >> $(METADATA)

+ 	@echo "Type:            Sanity" >> $(METADATA)

+ 	@echo "TestTime:        10m" >> $(METADATA)

+ 	@echo "Priority:        Normal" >> $(METADATA)

+ 	@echo "License:         GPLv2" >> $(METADATA)

+ 	@echo "Confidential:    no" >> $(METADATA)

+ 	@echo "Destructive:     no" >> $(METADATA)

+ 

+ 	rhts-lint $(METADATA)

@@ -0,0 +1,14 @@ 

+ #!/bin/bash

+ 

+ # Include Beaker environment

+ . /usr/share/beakerlib/beakerlib.sh || exit 1

+ 

+ PACKAGE="bash"

+ 

+ rlJournalStart

+     rlPhaseStartTest "Simple beakerlib test"

+         # Running a command that should fail

+         rlRun "false"

+     rlPhaseEnd

+ rlJournalPrintText

+ rlJournalEnd

@@ -0,0 +1,34 @@ 

+ export TESTVERSION=1.0

+ 

+ BUILT_FILES=

+ 

+ FILES=$(METADATA) runtest.sh Makefile

+ 

+ .PHONY: all install clean

+ 

+ run: $(FILES) build

+ 	./runtest.sh

+ 

+ build: $(BUILT_FILES)

+ 	test -x runtest.sh || chmod a+x runtest.sh

+ 

+ clean:

+ 	rm -f *~ $(BUILT_FILES)

+ 

+ 

+ include /usr/share/rhts/lib/rhts-make.include

+ 

+ $(METADATA): Makefile

+ 	@echo "Owner:           Bruno Goncalves <bgoncalv@redhat.com>" > $(METADATA)

+ 	@echo "Name:            $(TEST)" >> $(METADATA)

+ 	@echo "TestVersion:     $(TESTVERSION)" >> $(METADATA)

+ 	@echo "Path:            $(TEST_DIR)" >> $(METADATA)

+ 	@echo "Description:     simple rhts test role" >> $(METADATA)

+ 	@echo "Type:            Sanity" >> $(METADATA)

+ 	@echo "TestTime:        10m" >> $(METADATA)

+ 	@echo "Priority:        Normal" >> $(METADATA)

+ 	@echo "License:         GPLv2" >> $(METADATA)

+ 	@echo "Confidential:    no" >> $(METADATA)

+ 	@echo "Destructive:     no" >> $(METADATA)

+ 

+ 	rhts-lint $(METADATA)

@@ -0,0 +1,12 @@ 

+ #!/bin/bash

+ # Include Beaker environment

+ . /usr/share/beakerlib/beakerlib.sh || exit 1

+ 

+ PACKAGE="bash"

+ 

+ rlJournalStart

+     rlPhaseStartTest "Simple beakerlib test"

+         rlRun "ls /"

+     rlPhaseEnd

+ rlJournalPrintText

+ rlJournalEnd

file added
+5

@@ -0,0 +1,5 @@ 

+ ---

+ - import_playbook: inventory.yml

+ - import_playbook: basic.yml

+ - import_playbook: beakerlib.yml

+ - import_playbook: avocado.yml

When we open a pull request to this repo we should be testing our own tests. Lets move the tests here, and gate in our own upstream repo ... before we even get to an RPM.

Obviously we can use these tests again when we get to dist-git.

I have a corresponding branch for the rpms/standard-test-roles ... so that the tests are not duplicated... but am unable to make a pull request due to my fork being stuck in the PENDING status.

3 new commits added

  • tests: Distribute and invoke our own self-tests in Jenkins
  • scripts: merge-standard-inventory ignore non-existant directory
  • roles: Fix broken standard-test-avocado role
a month ago

Depends on https://pagure.io/standard-test-roles/pull-request/275 to be first merged and deployed ... before this one will even be validated.

rebased onto 52bf0c8

a month ago

Rebased now that #275 is merged

This now blocks on https://pagure.io/standard-test-roles/pull-request/276 . We need ansible in the testing container for the tests to pass.

New image is ready for usage. That image has #276.
Image is located at: https://console.apps.ci.centos.org:8443/console/project/fedora-atomic-process/browse/images/str-tester
It has package ansible.noarch 0:2.7.0-1.el7

rebased onto e4d631b

a month ago

This is now dependent on https://pagure.io/standard-test-roles/pull-request/278

I had to do more work to get ansible running correctly in the container. cIn addition I had to switch the container to Fedora so that FMF is available.

rebased onto 390ad3a

a month ago

This pull request is blocked on asking for a ServiceAccount to be able to create root pods. standard-test-roles needs to run as root.

3 new commits added

  • tests: Distribute and invoke our own self-tests in Jenkins
  • scripts: merge-standard-inventory ignore non-existant directory
  • roles: Fix broken standard-test-avocado role
a month ago

pretty please pagure-ci rebuild

pretty please pagure-ci rebuild

pretty please pagure-ci rebuild

rebased onto e917e1b

a month ago

Awesome. Thank you. Merging.

Commit bfa7471 fixes this pull-request

Pull-Request has been merged by stefw

a month ago

Pull-Request has been merged by stefw

a month ago