| |
@@ -0,0 +1,353 @@
|
| |
+ == Description ==
|
| |
+
|
| |
+ Package `standard-test-roles` provides shared Ansible roles and inventory scripts implementing the xref:standard-test-interface.adoc[Standard Test Interface].
|
| |
+ It has support for multiple testing frameworks (such as BeakerLib or Avocado) and in this way allows to easily enable existing tests in Fedora CI.
|
| |
+
|
| |
+ == Setup ==
|
| |
+
|
| |
+ === Packages ===
|
| |
+
|
| |
+ STR is available for Centos/RHEL from https://fedoraproject.org/wiki/EPEL[EPEL repository].
|
| |
+ As the first step install all necessary packages:
|
| |
+
|
| |
+ sudo dnf install fedpkg libselinux-python standard-test-roles
|
| |
+
|
| |
+ You can also install the latest version from the copr repo:
|
| |
+
|
| |
+ sudo dnf copr -y enable @osci/standard-test-roles
|
| |
+ sudo dnf update standard-test-roles
|
| |
+
|
| |
+ === Artifacts ===
|
| |
+
|
| |
+ Output of the test (such as the stdout/stderr output, log files or screenshots) is by default saved in the `artifacts` directory.
|
| |
+ Use `TEST_ARTIFACTS` environment variable to choose a different location if desired:
|
| |
+
|
| |
+ export TEST_ARTIFACTS=/tmp/artifacts
|
| |
+
|
| |
+ [NOTE]
|
| |
+ ====
|
| |
+ *Artifacts cleanup* +
|
| |
+ Before running tests make sure that all logs `/tmp/artifacts/test.*` are deleted.
|
| |
+ ====
|
| |
+
|
| |
+ === Inventory ===
|
| |
+
|
| |
+ A _test subject_ is what we call the thing to be tested.
|
| |
+ To turn a test subject into a launched, installed system to be tested, we use http://docs.ansible.com/ansible/intro_dynamic_inventory.html[Ansible dynamic inventory].
|
| |
+ Use the following command to enable it:
|
| |
+
|
| |
+ export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory)
|
| |
+
|
| |
+ As you can see from the way how the inventory is set, tests may contain their own inventory, which defines their own instructions for turning a _test subject_ into one or more testable systems.
|
| |
+
|
| |
+ == Testing ==
|
| |
+
|
| |
+ === Classic ===
|
| |
+
|
| |
+ You can always invoke the tests locally.
|
| |
+ Many tests modify or change the system they are run against, so take that into account when looking at how to invoke tests.
|
| |
+ The following examples invoke tests against the same system that the package git repository is checked out on.
|
| |
+ Below there are further options for invoking tests against another fully formed and integrated systems, such as an Atomic Host or container image _test subject_.
|
| |
+
|
| |
+ There may be more than one test present in a package git repository, but the file `tests.yml` is the main entry point.
|
| |
+ To run it use the following command:
|
| |
+
|
| |
+ ansible-playbook tests.yml
|
| |
+
|
| |
+ You can find output artifacts of the tests in an `artifacts/` or specify a specific directory like this:
|
| |
+
|
| |
+ ansible-playbook -e artifacts=/tmp/output tests.yml
|
| |
+
|
| |
+ You can filter which kinds of tests are run by providing a `--tags` argument.
|
| |
+ To only run tests that are suited for classic systems installed by `yum` or `dnf` you can use a command like:
|
| |
+
|
| |
+ ansible-playbook --tags=classic tests.yml
|
| |
+
|
| |
+ When run by a CI System the tests are invoked according to the xref:standard-test-interface.adoc[Standard Test Interface].
|
| |
+ Look here for more details and standard invocation variables.
|
| |
+
|
| |
+ === Package ===
|
| |
+
|
| |
+ When you run the tests as above, the tests assume that the system to be tested is the same as the system invoking the tests.
|
| |
+ In particular, the test assumes that the thing to be tested is already installed.
|
| |
+
|
| |
+ A _test subject_ is what we call the thing to be tested.
|
| |
+ RPMs are a particular kind of _test subject_.
|
| |
+ To turn a test subject into a launched, installed system to be tested, we use http://docs.ansible.com/ansible/intro_dynamic_inventory.html[Ansible dynamic inventory].
|
| |
+ Let's invoke the tests with an inventory and a specific version of gzip:
|
| |
+
|
| |
+ curl -o gzip.rpm https://kojipkgs.fedoraproject.org//packages/gzip/1.8/2.fc26/x86_64/gzip-1.8-2.fc26.x86_64.rpm
|
| |
+ export TEST_SUBJECTS=$PWD/gzip.rpm
|
| |
+ ansible-playbook tests.yml
|
| |
+
|
| |
+ You'll notice that the RPM is installed into the testable system before invoking the tests.
|
| |
+ Some tests contain their own inventory, that is their own instructions for turning a _test subject_ into one or more testable systems.
|
| |
+ But in this case we use the default `standard-test-roles` inventory in `/usr/share/ansible/inventory` to do this.
|
| |
+
|
| |
+ === Container ===
|
| |
+
|
| |
+ Another example is to use a _test subject_ of a container image.
|
| |
+ This is also a fully formed and integrated deliverable.
|
| |
+ The _test subject_ again represents the thing to be tested.
|
| |
+ For testing containers there is an additional dependency needed:
|
| |
+
|
| |
+ sudo dnf install standard-test-roles-inventory-docker
|
| |
+
|
| |
+ The container image is pulled from a registry and launched using docker by an http://docs.ansible.com/ansible/intro_dynamic_inventory.html[Ansible dynamic inventory].
|
| |
+
|
| |
+ export TEST_SUBJECTS=docker:docker.io/library/fedora:27
|
| |
+ ansible-playbook --tags=container tests.yml
|
| |
+
|
| |
+ If you watch closely you'll notice the image is pulled if not already local, launched as a container, and then prepared for the tests to run on.
|
| |
+ The first time this may take a little longer.
|
| |
+ Not all tests are able to function in the somewhat different environment of a container.
|
| |
+ In fact, for certain tests, the software to be tested may not be included in the container.
|
| |
+ But many of the tests for core packages should work here.
|
| |
+
|
| |
+ The `--tags` argument filters out tests that are not suitable for running in a container, either because the system functions differently, or the correct packages are not installable.
|
| |
+
|
| |
+ See the <<_debug>> section for instructions how to log into a running container and diagnose why the tests failed.
|
| |
+
|
| |
+ ==== Additional arguments for Docker ====
|
| |
+
|
| |
+ Tests for containers are run with a help of Docker.
|
| |
+ Containers are run within default security context.
|
| |
+ For more info see https://docs.docker.com/engine/security/seccomp/[Seccomp security profiles for Docker].
|
| |
+ It is possible that some tests require additional privileges.
|
| |
+ In this case specify necessary arguments for Docker using an environment variable `TEST_DOCKER_EXTRA_ARGS`.
|
| |
+ For this create a file `inventory` file in `tests` directory with the following content:
|
| |
+
|
| |
+ #!/bin/bash
|
| |
+ export TEST_DOCKER_EXTRA_ARGS="--security-opt seccomp:unconfined"
|
| |
+ exec merge-standard-inventory "$@"
|
| |
+
|
| |
+ or
|
| |
+
|
| |
+ #!/bin/bash
|
| |
+ export TEST_DOCKER_EXTRA_ARGS="--privileged"
|
| |
+ exec merge-standard-inventory "$@"
|
| |
+
|
| |
+ See https://pagure.io/standard-test-roles/blob/master/f/scripts/README.md[merge-standard-inventory] documentation for details.
|
| |
+
|
| |
+ === Atomic ===
|
| |
+
|
| |
+ The former example may seem a bit contrived, but the concept of a _test subject_ starts to make more sense when you want to test a fully formed and integrated deliverable, such as Atomic Host.
|
| |
+ The _test subject_ again represents the thing to be tested.
|
| |
+ The _test subject_ in this case is a QCow2 image.
|
| |
+ To turn a test subject into a launched system ready to be tested, we use http://docs.ansible.com/ansible/intro_dynamic_inventory.html[Ansible dynamic inventory].
|
| |
+
|
| |
+ curl -Lo /tmp/atomic.qcow2 https://getfedora.org/atomic_qcow2_latest
|
| |
+ export TEST_SUBJECTS=/tmp/atomic.qcow2
|
| |
+ ansible-playbook --tags=atomic tests.yml
|
| |
+
|
| |
+ If you watch closely you'll see that the Atomic Host image is booted, and the tests run against the launched image.
|
| |
+ Not all tests are able to function in the somewhat different environment of Atomic Host, in fact, for certain cases, the software to be tested may not be included in the Atomic Host _test subject_.
|
| |
+ But most of the tests in core packages should work here.
|
| |
+
|
| |
+ Some tests contain their own inventory, that is their own instructions for turning a _test subject_ into one or more testable systems.
|
| |
+ But in this case we use the default `standard-test-roles` inventory to do this.
|
| |
+
|
| |
+ The `--tags` argument filters out tests that are not suitable for running on an Atomic Host, either because the system functions differently, or the correct packages are not available on that system.
|
| |
+
|
| |
+ See the <<_debug>> section to learn how to diagnose why the tests failed, and log into the running Atomic Host.
|
| |
+
|
| |
+
|
| |
+ [NOTE]
|
| |
+ ====
|
| |
+ *Required Packages* +
|
| |
+ are specified in `tests.yml` for Atomic Host, additional packages will be installed using the `rpm-ostree` command which is affecting the test subject (it's similar as rebuilding an rpm package to be tested) so this should be used with caution and only when necessary.
|
| |
+ Also be aware that there are certain limitations for this approach (e.g. it's not possible to install different version of packages that are already part of the tree).
|
| |
+ ====
|
| |
+
|
| |
+ [NOTE]
|
| |
+ ====
|
| |
+ *Required Packages* +
|
| |
+ Atomic Host is shipped as a base ostree image, however you can install additional packages with a help of `rpm-ostree install` command.
|
| |
+ Currently (10.01.2018 ) repo with additional packages is actual only for the latest base-ostree image.
|
| |
+ Consequence: tests that install additional packages for Atomic Host can fail sometimes with: `error: The following base packages would be replaced: xxx` Solution: make sure you have the latest Atomic Host image.
|
| |
+ Additional information you can find https://github.com/projectatomic/rpm-ostree/issues/415[rpm-ostree issue 415] and a possible solution in the feature using `rpm-ostree jigdo` https://github.com/projectatomic/rpm-ostree/issues/1081[rpm-ostree issue 1081].
|
| |
+ ====
|
| |
+
|
| |
+ === Debug ===
|
| |
+
|
| |
+ To increase output verbosity use option `-v` or `-vvv`:
|
| |
+
|
| |
+ ansible-playbook --tags=container tests.yml -v
|
| |
+
|
| |
+ or for full verbosity:
|
| |
+
|
| |
+ ansible-playbook --tags=container tests.yml -vvv
|
| |
+
|
| |
+ To debug tests in a running container or atomic host use the `TEST_DEBUG` environment variable.
|
| |
+ After the playbook runs, you'll see diagnosis information with a helpful command to log in.
|
| |
+
|
| |
+ export TEST_DEBUG=1
|
| |
+
|
| |
+ For container you'll see output like this:
|
| |
+
|
| |
+ DIAGNOSE: docker exec -it 56de801f0ddde36fc9770666f7be2a68f89d7f18f52b7b6fe7df7a12b193bf08 /bin/bash
|
| |
+ DIAGNOSE: kill 18261 # when finished
|
| |
+
|
| |
+ For atomic host the instructions are a bit different:
|
| |
+
|
| |
+ DIAGNOSE: ssh -p 2222 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@127.0.0.3 # password: foobar
|
| |
+ DIAGNOSE: export ANSIBLE_INVENTORY=/tmp/inventory-cloudxyhF2M/inventory
|
| |
+ DIAGNOSE: kill 16611 # when finished
|
| |
+
|
| |
+ Now you can easily connect using these commands.
|
| |
+ Use suggested `kill` command to finish the running instance when done with investigation.
|
| |
+
|
| |
+ == Roles ==
|
| |
+
|
| |
+ Here's the list of currently supported roles for test execution:
|
| |
+
|
| |
+ * standard-test-avocado - role for executing tests written via the Avocado testing framework
|
| |
+ * standard-test-basic - a simple role for executing runtest.sh scripts, or other scripts in given directories
|
| |
+ * standart-test-beakerlib - role for executing tests written via Beakerlib testing framework, supporting all testing environments
|
| |
+ * standard-test-scripts - role for executing arbitrary test scripts
|
| |
+
|
| |
+ Here's list of currently supported helper roles:
|
| |
+
|
| |
+ * standard-test-repo - a role for installing packages for additional yum repository files
|
| |
+ * standard-test-rpm - a role for installing additional rpms
|
| |
+ * standard-test-source - a role for extracting upstream source tarball (with tests)
|
| |
+
|
| |
+ === BeakerLib ===
|
| |
+
|
| |
+ This is the recommended role for running tests written via the https://github.com/beakerlib/beakerlib[Beakerlib Testing Framework] as it supports all currenlty supported testing environments (atomic, classic, container).
|
| |
+ It also supports https://pagure.io/beakerlib-libraries[beakerlib-libraries] which allow easy code reuse among multiple tests.
|
| |
+
|
| |
+ To use this role create `tests.yml` file with contents similar to the following snippet.
|
| |
+ The `tests` parameter should include the list of directories with your beakerlib tests.
|
| |
+ [source,ansible]
|
| |
+ ----
|
| |
+ - hosts: localhost
|
| |
+ tags:
|
| |
+ - atomic
|
| |
+ - classic
|
| |
+ - container
|
| |
+ roles:
|
| |
+ - role: standard-test-beakerlib
|
| |
+ tests:
|
| |
+ - cmd-line-options
|
| |
+ ----
|
| |
+
|
| |
+ The `required_packages` parameter can be used to list additional packages that need to be installed on the system to run tests.
|
| |
+ If you have required packages correctly specified in the beakerlib test metadata (in Makefile `RhtsRequires` stands for hard requirements, `Requires` for soft requirements) it is not necessary to list them again here.
|
| |
+
|
| |
+ [source,ansible]
|
| |
+ ----
|
| |
+ - hosts: localhost
|
| |
+ tags:
|
| |
+ - atomic
|
| |
+ - classic
|
| |
+ - container
|
| |
+ roles:
|
| |
+ - role: standard-test-beakerlib
|
| |
+ tests:
|
| |
+ - cmd-line-options
|
| |
+ required_packages:
|
| |
+ - which # which package required for cmd-line-options
|
| |
+ - rpm-build # upstream-testsuite requires rpmbuild command
|
| |
+ - libtool # upstream-testsuite requires libtool
|
| |
+ - gettext # upstream-testsuite requires gettext
|
| |
+ ----
|
| |
+
|
| |
+ [NOTE]
|
| |
+ ====
|
| |
+ The `required_packages` parameter is ignored when running on Atomic Host--since there is no way to install additional packages in that environment.
|
| |
+ ====
|
| |
+
|
| |
+ Instead of manually listing all tests to be executed it is also possible to provide an fmf filter in the following way:
|
| |
+
|
| |
+ [source,ansible]
|
| |
+ ----
|
| |
+ - hosts: localhost
|
| |
+ roles:
|
| |
+ - role: standard-test-beakerlib
|
| |
+ tags:
|
| |
+ - classic
|
| |
+ repositories:
|
| |
+ - repo: "https://src.fedoraproject.org/tests/shell.git"
|
| |
+ dest: "shell"
|
| |
+ fmf_filter: "tier: 1"
|
| |
+ ----
|
| |
+
|
| |
+ Filter can be used also if tests are stored directly in the git:
|
| |
+
|
| |
+ [source,ansible]
|
| |
+ ----
|
| |
+ - hosts: localhost
|
| |
+ roles:
|
| |
+ - role: standard-test-beakerlib
|
| |
+ tags:
|
| |
+ - classic
|
| |
+ fmf_filter: "tier: 1"
|
| |
+ ----
|
| |
+
|
| |
+ See https://pagure.io/fedora-ci/metadata[Metadata] for more info about filtering tests based on fmf metadata.
|
| |
+
|
| |
+ === Basic ===
|
| |
+
|
| |
+ Basic role can be used for executing scripts or binaries as simple tests.
|
| |
+ For example the following `tests.yml` file will run `binary --help` as a shell command in the current directory and provide pass/fail based on its return code:
|
| |
+
|
| |
+ [source,ansible]
|
| |
+ ----
|
| |
+ - hosts: localhost
|
| |
+ roles:
|
| |
+ - role: standard-test-basic
|
| |
+ tags:
|
| |
+ - classic
|
| |
+ tests:
|
| |
+ - simple:
|
| |
+ dir: .
|
| |
+ run: binary --help
|
| |
+ ----
|
| |
+
|
| |
+ Here's another example `tests.yml` file which fetches a single integration test from a shared repository and uses parametrizing to run it multiple times with different environment variables:
|
| |
+
|
| |
+ [source,ansible]
|
| |
+ ----
|
| |
+ - hosts: localhost
|
| |
+ roles:
|
| |
+ - role: standard-test-basic
|
| |
+ tags:
|
| |
+ - classic
|
| |
+ repositories:
|
| |
+ - repo: "https://src.fedoraproject.org/tests/python.git"
|
| |
+ dest: "python"
|
| |
+ tests:
|
| |
+ - smoke27:
|
| |
+ dir: python/smoke
|
| |
+ run: VERSION=2.7 METHOD=virtualenv ./venv.sh
|
| |
+ - smoke37:
|
| |
+ dir: python/smoke
|
| |
+ run: VERSION=3.7 ./venv.sh
|
| |
+ required_packages:
|
| |
+ - python27
|
| |
+ - python37
|
| |
+ - python2-virtualenv
|
| |
+ - python3-virtualenv
|
| |
+ - python2-devel
|
| |
+ - python3-devel
|
| |
+ ----
|
| |
+
|
| |
+ === RHTS ===
|
| |
+
|
| |
+ This role has been obsoleted by the <<_beakerlib>> role which provides similar functionality.
|
| |
+
|
| |
+ == More ==
|
| |
+
|
| |
+ === Links ===
|
| |
+
|
| |
+ Pagure and Copr repositories:
|
| |
+
|
| |
+ * https://pagure.io/standard-test-roles[pagure.io/standard-test-roles]
|
| |
+ * https://copr.fedorainfracloud.org/coprs/g/osci/standard-test-roles/builds[standard-test-roles copr builds]
|
| |
+
|
| |
+ === Contact ===
|
| |
+
|
| |
+ * Andrei Stepanov (astepano)
|
| |
+ * Miroslav Vadkerti (mvadkert)
|
| |
+
|
| |
ph0zzy
commented 5 years ago
copy data from old wiki main page