PR#274: tests: Distribute and invoke our own self-tests in Jenkins - standard-test-roles

standard-test-roles

#274 tests: Distribute and invoke our own self-tests in Jenkins

Merged 5 years ago by stefw. Opened 5 years ago by stefw.

stefw/standard-test-roles tests-upstream into master

tests: Distribute and invoke our own self-tests in Jenkins

Stef Walter • 5 years ago

f472e03

scripts: merge-standard-inventory ignore non-existant directory

Stef Walter • 5 years ago

235f8c0

roles: Fix broken standard-test-avocado role

Stef Walter • 5 years ago

e917e1b

roles/standard-test-avocado/tasks/main.yml

file modified

+1 -1

		`@@ -12,7 +12,7 @@`
		`- name: Execute the avocado test`
		`shell: exec 2>>{{ remote_artifacts }}/test.log 1>>{{ remote_artifacts }}/test.log; MODULE=rpm python -m avocado run --job-results-dir {{ remote_artifacts }}/ {{tests\|join(' ') }}`

		`- - always:`
		`+ always:`
		`- name: Pull out the logs`
		`synchronize:`
		`dest: "{{ artifacts }}/"`

scripts/merge-standard-inventory

file modified

+1 -1

		`@@ -79,7 +79,7 @@`

		`merged = Inventory()`

		`- for i in os.listdir(inventory_dir):`
		`+ for i in os.path.exists(inventory_dir) and os.listdir(inventory_dir) or []:`
		`ipath = os.path.join(inventory_dir, i)`
		`if not i.startswith("standard-inventory-"):`
		`continue`

tests/avocado.yml

file added

+24

		`@@ -0,0 +1,24 @@`
		`+ # Tests for avocado role`
		`+ - hosts: localhost`
		`+ tags:`
		`+ - classic`
		`+ - container`
		`+ roles:`
		`+ - role: standard-test-avocado`
		`+ tests:`
		`+ - /bin/true`
		`+ - role: standard-test-avocado`
		`+ tests:`
		`+ - /bin/false`
		`+ ignore_errors: yes`
		`+ tasks:`
		`+ - name: Read test.log on test environment`
		`+ set_fact:`
		`+ test_log: "{{ lookup('file', artifacts + '/test.log') }}"`
		`+ - name: Check for FAIL on test.log on test environment`
		`+ fail: msg="Could not find expected text on test log"`
		`+ when: >`
		`+ (test_log.find('/bin/false: FAIL') == -1) or`
		`+ (test_log.find('/bin/true: PASS') == -1)`
		`+ # We should fail when we can not find FAIL string on log`
		`+ # -1 means string not found`

tests/basic.yml

file added

+38

		`@@ -0,0 +1,38 @@`
		`+ - import_playbook: prepare.yml`
		`+`
		`+ # Tests for basic role`
		`+ - hosts: localhost`
		`+ tags:`
		`+ - atomic`
		`+ - classic`
		`+ - container`
		`+ roles:`
		`+ - role: standard-test-basic`
		`+ tests:`
		`+ - test-basic-simple`
		`+ - test-basic-parameters:`
		`+ dir: ./`
		`+ run: echo "check parameters on basic role" \| grep "check parameters on basic role"`
		`+ required_packages:`
		`+ # Test if we can install required packages`
		`+ - "{{req_pkg}}"`
		`+ tasks:`
		`+ - import_tasks: shared-tasks/artifacts_test_env.yml`
		`+ - import_tasks: shared-tasks/artifacts_test_runner.yml`
		`+`
		`+ # Make sure the role behaves correctly if test fails`
		`+ - hosts: localhost`
		`+ tags:`
		`+ - atomic`
		`+ - classic`
		`+ - container`
		`+ roles:`
		`+ - role: standard-test-basic`
		`+ tests:`
		`+ - test-basic-fail`
		`+ ignore_errors: yes`
		`+ tasks:`
		`+ # 'verify_failed_test' tasks should run after 'test-basic-fail'`
		`+ - import_tasks: shared-tasks/verify_failed_test.yml`
		`+ - import_tasks: shared-tasks/artifacts_test_env.yml`
		`+ - import_tasks: shared-tasks/artifacts_test_runner.yml`

tests/beakerlib.yml

file added

+35

		`@@ -0,0 +1,35 @@`
		`+ - import_playbook: prepare.yml`
		`+`
		`+ # Tests for beakerlib role`
		`+ - hosts: localhost`
		`+ tags:`
		`+ - atomic`
		`+ - classic`
		`+ - container`
		`+ roles:`
		`+ - role: standard-test-beakerlib`
		`+ tests:`
		`+ - test-beakerlib-simple`
		`+ required_packages:`
		`+ - "{{req_pkg}}"`
		`+ tasks:`
		`+ - import_tasks: shared-tasks/artifacts_test_env.yml`
		`+ - import_tasks: shared-tasks/artifacts_test_runner.yml`
		`+`
		`+`
		`+ # Make sure the role behaves correctly if test fails`
		`+ - hosts: localhost`
		`+ tags:`
		`+ - atomic`
		`+ - classic`
		`+ - container`
		`+ roles:`
		`+ - role: standard-test-beakerlib`
		`+ tests:`
		`+ - test-beakerlib-fail`
		`+ ignore_errors: yes`
		`+ tasks:`
		`+ # 'tests_verify_failed_test' tasks should run after 'test-beakerlib-fail'`
		`+ - import_tasks: shared-tasks/verify_failed_test.yml`
		`+ - import_tasks: shared-tasks/artifacts_test_env.yml`
		`+ - import_tasks: shared-tasks/artifacts_test_runner.yml`

tests/inventory.yml

file added

+14

		`@@ -0,0 +1,14 @@`
		`+ - hosts: localhost`
		`+ tags:`
		`+ - always`
		`+ tasks:`
		`+ - name: Get inventory with empty input`
		`+ command: merge-standard-inventory`
		`+ environment:`
		`+ # Reset vars to default values`
		`+ TEST_SUBJECTS:`
		`+ register: inventory`
		`+ delegate_to: localhost`
		`+ - name: Assert merge-standard-inventory produces correct JSON inventory (empty)`
		`+ fail:`
		`+ when: inventory.stdout != '{}'`

tests/invoke-integration.sh

file added

+13

		`@@ -0,0 +1,13 @@`
		`+ #!/bin/sh`
		`+`
		`+ #`
		`+ # This invokes the tests using the local inventory and roles rather`
		`+ # than what's installed on system. This is aslight deviation from`
		`+ # what's specified in the standard test interface`
		`+ #`
		`+`
		`+ cd $(dirname $0)/..`
		`+`
		`+ rm -rf /tmp/artifacts/ /tmp/local-artifacts/`
		`+ TEST_ARTIFACTS=/tmp PATH=$PWD/scripts:$PATH ANSIBLE_INVENTORY=$PWD/inventory \`
		`+ ansible-playbook --extra-vars=artifacts=/tmp/local-artifacts --tags=classic tests/tests.yml`

tests/prepare.yml

file added

+13

		`@@ -0,0 +1,13 @@`
		`+ # Preparation steps`
		`+ - hosts: localhost`
		`+ vars:`
		`+ # Must be the same as in str-common/defaults/main.yml`
		`+ artifacts: "{{ lookup('env', 'TEST_ARTIFACTS')\|default('./artifacts', true) }}"`
		`+ remote_artifacts: /tmp/artifacts/`
		`+ tasks:`
		`+ - import_tasks: shared-tasks/artifacts.yml`
		`+ tags:`
		`+ - always`
		`+ - import_tasks: shared-tasks/req-pkg.yml`
		`+ tags:`
		`+ - always`

tests/rhts.yml

file added

+33

		`@@ -0,0 +1,33 @@`
		`+ - import_playbook: prepare.yml`
		`+`
		`+ # Tests for rhts role only runs on classic tag`
		`+ - hosts: localhost`
		`+ tags:`
		`+ - classic`
		`+ roles:`
		`+ - role: standard-test-rhts`
		`+ tests:`
		`+ - test-rhts-simple`
		`+ required_packages:`
		`+ # Test if we can install required packages`
		`+ - "{{req_pkg}}"`
		`+ tasks:`
		`+ - import_tasks: shared-tasks/artifacts_test_env.yml`
		`+ - import_tasks: shared-tasks/artifacts_test_runner.yml`
		`+`
		`+ # Make sure the role behaves correctly if test fails`
		`+ - hosts: localhost`
		`+ tags:`
		`+ - classic`
		`+ roles:`
		`+ - role: standard-test-rhts`
		`+ tags:`
		`+ - classic`
		`+ tests:`
		`+ - test-rhts-fail`
		`+ ignore_errors: yes`
		`+ tasks:`
		`+ # 'verify_failed_test' tasks should run after 'testi-rhts-fail'`
		`+ - import_tasks: shared-tasks/verify_failed_test.yml`
		`+ - import_tasks: shared-tasks/artifacts_test_env.yml`
		`+ - import_tasks: shared-tasks/artifacts_test_runner.yml`

tests/roles

file added

		`@@ -0,0 +1,1 @@`
		`+ ../roles`
		`\ No newline at end of file`

tests/shared-tasks/artifacts.yml

file added

		`@@ -0,0 +1,9 @@`
		`+ # STR requires that artifacts dir must be absent`
		`+ - name: "Check if {{artifacts}} is absent at test-environment."`
		`+ stat:`
		`+ path: "{{remote_artifacts}}"`
		`+ register: st_te`
		`+`
		`+ - fail:`
		`+ msg: "Remove {{remote_artifacts}} before test run at test-environment."`
		`+ when: st_te.stat.exists`

tests/shared-tasks/artifacts_test_env.yml

file added

		`@@ -0,0 +1,9 @@`
		`+ # These tasks should run after as test that fails`
		`+ - name: "Check if {{remote_artifacts}} was created properly on test environment"`
		`+ shell: "ls {{remote_artifacts}}/test.log"`
		`+`
		`+ - name: "Clean up {{remote_artifacts}} to not affect other tests on test environment"`
		`+ file:`
		`+ state: absent`
		`+ path: "{{remote_artifacts}}/"`
		`+`

tests/shared-tasks/artifacts_test_runner.yml

file added

+10

		`@@ -0,0 +1,10 @@`
		`+ # These tasks should run after as test that fails`
		`+ - name: "Check if {{artifacts}} was created properly on test runner"`
		`+ shell: "ls {{artifacts}}/test.log"`
		`+ delegate_to: localhost`
		`+`
		`+ - name: "Clean up {{artifacts}} to not affect other tests on test runner"`
		`+ file:`
		`+ state: absent`
		`+ path: "{{artifacts}}/"`
		`+ delegate_to: localhost`

tests/shared-tasks/req-pkg.yml

file added

+12

		`@@ -0,0 +1,12 @@`
		`+ - import_role:`
		`+ name: str-common-pkgs`
		`+ tasks_from: inspect.yml`
		`+`
		`+ - set_fact:`
		`+ req_pkg: zsh`
		`+`
		`+ - name: Remove a test package from test-environment`
		`+ package:`
		`+ name: "{{req_pkg}}"`
		`+ state: absent`
		`+ when: not is_atomic`

tests/shared-tasks/verify_failed_test.yml

file added

+11

		`@@ -0,0 +1,11 @@`
		`+ # These tasks should run after as test that fails`
		`+ - name: Read test.log on test environment`
		`+ shell: "cat {{remote_artifacts}}/test.log"`
		`+ register: test_log`
		`+`
		`+ - name: Check for FAIL on test.log on test environment`
		`+ fail: msg="Could not find FAIL on test log"`
		`+ when: test_log.stdout.find("FAIL") == -1`
		`+ # We should fail when we can not find FAIL string on log`
		`+ # -1 means string not found`
		`+`

tests/test-basic-fail/runtest.sh

file added

		`@@ -0,0 +1,5 @@`
		`+ #!/bin/bash`
		`+ echo "Running a command that should fail"`
		`+`
		`+ false`
		`+`

tests/test-basic-simple/runtest.sh

file added

		`@@ -0,0 +1,5 @@`
		`+ #!/bin/bash`
		`+ PACKAGE="bash"`
		`+`
		`+ echo "Using $PACKAGE version:"`
		`+ rpm -q $PACKAGE`

tests/test-beakerlib-fail/runtest.sh

file added

+14

		`@@ -0,0 +1,14 @@`
		`+ #!/bin/bash`
		`+`
		`+ # Include Beaker environment`
		`+ . /usr/share/beakerlib/beakerlib.sh \|\| exit 1`
		`+`
		`+ PACKAGE="bash"`
		`+`
		`+ rlJournalStart`
		`+ rlPhaseStartTest "Simple beakerlib test"`
		`+ # Running a command that should fail`
		`+ rlRun "false"`
		`+ rlPhaseEnd`
		`+ rlJournalPrintText`
		`+ rlJournalEnd`

tests/test-beakerlib-simple/runtest.sh

file added

+12

		`@@ -0,0 +1,12 @@`
		`+ #!/bin/bash`
		`+ # Include Beaker environment`
		`+ . /usr/share/beakerlib/beakerlib.sh \|\| exit 1`
		`+`
		`+ PACKAGE="bash"`
		`+`
		`+ rlJournalStart`
		`+ rlPhaseStartTest "Simple beakerlib test"`
		`+ rlRun "ls /"`
		`+ rlPhaseEnd`
		`+ rlJournalPrintText`
		`+ rlJournalEnd`

tests/test-rhts-fail/Makefile

file added

+34

		`@@ -0,0 +1,34 @@`
		`+ export TESTVERSION=1.0`
		`+`
		`+ BUILT_FILES=`
		`+`
		`+ FILES=$(METADATA) runtest.sh Makefile`
		`+`
		`+ .PHONY: all install clean`
		`+`
		`+ run: $(FILES) build`
		`+ ./runtest.sh`
		`+`
		`+ build: $(BUILT_FILES)`
		`+ test -x runtest.sh \|\| chmod a+x runtest.sh`
		`+`
		`+ clean:`
		`+ rm -f *~ $(BUILT_FILES)`
		`+`
		`+`
		`+ include /usr/share/rhts/lib/rhts-make.include`
		`+`
		`+ $(METADATA): Makefile`
		`+ @echo "Owner: Bruno Goncalves <bgoncalv@redhat.com>" > $(METADATA)`
		`+ @echo "Name: $(TEST)" >> $(METADATA)`
		`+ @echo "TestVersion: $(TESTVERSION)" >> $(METADATA)`
		`+ @echo "Path: $(TEST_DIR)" >> $(METADATA)`
		`+ @echo "Description: fail rhts test role" >> $(METADATA)`
		`+ @echo "Type: Sanity" >> $(METADATA)`
		`+ @echo "TestTime: 10m" >> $(METADATA)`
		`+ @echo "Priority: Normal" >> $(METADATA)`
		`+ @echo "License: GPLv2" >> $(METADATA)`
		`+ @echo "Confidential: no" >> $(METADATA)`
		`+ @echo "Destructive: no" >> $(METADATA)`
		`+`
		`+ rhts-lint $(METADATA)`

tests/test-rhts-fail/runtest.sh

file added

+14

		`@@ -0,0 +1,14 @@`
		`+ #!/bin/bash`
		`+`
		`+ # Include Beaker environment`
		`+ . /usr/share/beakerlib/beakerlib.sh \|\| exit 1`
		`+`
		`+ PACKAGE="bash"`
		`+`
		`+ rlJournalStart`
		`+ rlPhaseStartTest "Simple beakerlib test"`
		`+ # Running a command that should fail`
		`+ rlRun "false"`
		`+ rlPhaseEnd`
		`+ rlJournalPrintText`
		`+ rlJournalEnd`

tests/test-rhts-simple/Makefile

file added

+34

		`@@ -0,0 +1,34 @@`
		`+ export TESTVERSION=1.0`
		`+`
		`+ BUILT_FILES=`
		`+`
		`+ FILES=$(METADATA) runtest.sh Makefile`
		`+`
		`+ .PHONY: all install clean`
		`+`
		`+ run: $(FILES) build`
		`+ ./runtest.sh`
		`+`
		`+ build: $(BUILT_FILES)`
		`+ test -x runtest.sh \|\| chmod a+x runtest.sh`
		`+`
		`+ clean:`
		`+ rm -f *~ $(BUILT_FILES)`
		`+`
		`+`
		`+ include /usr/share/rhts/lib/rhts-make.include`
		`+`
		`+ $(METADATA): Makefile`
		`+ @echo "Owner: Bruno Goncalves <bgoncalv@redhat.com>" > $(METADATA)`
		`+ @echo "Name: $(TEST)" >> $(METADATA)`
		`+ @echo "TestVersion: $(TESTVERSION)" >> $(METADATA)`
		`+ @echo "Path: $(TEST_DIR)" >> $(METADATA)`
		`+ @echo "Description: simple rhts test role" >> $(METADATA)`
		`+ @echo "Type: Sanity" >> $(METADATA)`
		`+ @echo "TestTime: 10m" >> $(METADATA)`
		`+ @echo "Priority: Normal" >> $(METADATA)`
		`+ @echo "License: GPLv2" >> $(METADATA)`
		`+ @echo "Confidential: no" >> $(METADATA)`
		`+ @echo "Destructive: no" >> $(METADATA)`
		`+`
		`+ rhts-lint $(METADATA)`

tests/test-rhts-simple/runtest.sh

file added

+12

		`@@ -0,0 +1,12 @@`
		`+ #!/bin/bash`
		`+ # Include Beaker environment`
		`+ . /usr/share/beakerlib/beakerlib.sh \|\| exit 1`
		`+`
		`+ PACKAGE="bash"`
		`+`
		`+ rlJournalStart`
		`+ rlPhaseStartTest "Simple beakerlib test"`
		`+ rlRun "ls /"`
		`+ rlPhaseEnd`
		`+ rlJournalPrintText`
		`+ rlJournalEnd`

tests/tests.yml

file added

		`@@ -0,0 +1,5 @@`
		`+ ---`
		`+ - import_playbook: inventory.yml`
		`+ - import_playbook: basic.yml`
		`+ - import_playbook: beakerlib.yml`
		`+ - import_playbook: avocado.yml`

stefw commented 5 years ago

When we open a pull request to this repo we should be testing our own tests. Lets move the tests here, and gate in our own upstream repo ... before we even get to an RPM.

Obviously we can use these tests again when we get to dist-git.

stefw commented 5 years ago

I have a corresponding branch for the rpms/standard-test-roles ... so that the tests are not duplicated... but am unable to make a pull request due to my fork being stuck in the PENDING status.

3 new commits added

tests: Distribute and invoke our own self-tests in Jenkins
scripts: merge-standard-inventory ignore non-existant directory
roles: Fix broken standard-test-avocado role

5 years ago

stefw commented 5 years ago

Depends on https://pagure.io/standard-test-roles/pull-request/275 to be first merged and deployed ... before this one will even be validated.

rebased onto 52bf0c8455f3a5d9891524d284098230605f7869

5 years ago

stefw commented 5 years ago

Rebased now that #275 is merged

stefw commented 5 years ago

This now blocks on https://pagure.io/standard-test-roles/pull-request/276 . We need ansible in the testing container for the tests to pass.

astepano commented 5 years ago

New image is ready for usage. That image has #276.
Image is located at: https://console.apps.ci.centos.org:8443/console/project/fedora-atomic-process/browse/images/str-tester
It has package ansible.noarch 0:2.7.0-1.el7

rebased onto e4d631b88606a04c8232c565d48d4ce0c1f5dd6e

5 years ago

stefw commented 5 years ago

This is now dependent on https://pagure.io/standard-test-roles/pull-request/278

I had to do more work to get ansible running correctly in the container. cIn addition I had to switch the container to Fedora so that FMF is available.

astepano commented 5 years ago

#278 was merged.

rebased onto 390ad3aae5f5dcbde65665aca14b31863e2984a7

5 years ago

stefw commented 5 years ago

This pull request is blocked on asking for a ServiceAccount to be able to create root pods. standard-test-roles needs to run as root.

3 new commits added