| |
@@ -0,0 +1,253 @@
|
| |
+ :toc:
|
| |
+
|
| |
+ = Tests =
|
| |
+
|
| |
+ [cols="1", options="header"]
|
| |
+ |===
|
| |
+ |Quick links to automated test reports
|
| |
+
|
| |
+ |https://fedoraproject.org/wiki/CI/Tests/stat[stat]
|
| |
+
|
| |
+ |https://fedoraproject.org/wiki/CI/Tests/new-stat[new-stat]
|
| |
+
|
| |
+ |https://fedoraproject.org/wiki/CI/Tests/stat_atomic[stat atomic]
|
| |
+
|
| |
+ |https://fedoraproject.org/wiki/CI/Tests/recent_builds[recent builds]
|
| |
+
|
| |
+ |https://fedoraproject.org/wiki/CI/Tests/stat_everything_subset[stat everything subset]
|
| |
+
|
| |
+ |https://fedoraproject.org/wiki/CI/Tests/stat_fedoraserver[stat fedoraserver]
|
| |
+ |===
|
| |
+
|
| |
+ == Enabling ==
|
| |
+
|
| |
+ Tests may be written in different ways, but are exposed and invoked in a standard way as defined by the xref:standard-test-interface.adoc[Standard Test Interface] directly in the package https://src.fedoraproject.org/projects/rpms/%2A[git repository].
|
| |
+ It is also possible to enable pipeline for the tests namespace, see xref:share-test-code.adoc#_testing_tests[Testing Tests] for details.
|
| |
+ To start working on tests you can clone a package repo directly:
|
| |
+
|
| |
+ git clone https://src.fedoraproject.org/rpms/qrencode.git
|
| |
+
|
| |
+ You can also use the `fedpkg` to clone the repo.
|
| |
+ See the xref:package-maintenance-guide.adoc[Package Maintenance Guide] for more info about the tool:
|
| |
+
|
| |
+ fedpkg clone -a qrencode
|
| |
+
|
| |
+ Tests are enabled by including the `tests.yml` file under the `tests` directory:
|
| |
+
|
| |
+ cd qrencode/tests
|
| |
+ cat tests.yml
|
| |
+
|
| |
+ Tests are wrapped or written as http://docs.ansible.com/ansible/playbooks.html[Ansible playbooks].
|
| |
+ Here is an example of a simple playbok which enables a single `smoke` test of the `qrencode` package:
|
| |
+
|
| |
+ [source,ansible]
|
| |
+ ----
|
| |
+ - hosts: localhost
|
| |
+ roles:
|
| |
+ - role: standard-test-beakerlib
|
| |
+ tags:
|
| |
+ - classic
|
| |
+ - container
|
| |
+ - atomic
|
| |
+ tests:
|
| |
+ - smoke
|
| |
+ required_packages:
|
| |
+ - qrencode
|
| |
+ - file
|
| |
+ ----
|
| |
+
|
| |
+ Let's now briefly look at the playbook to see which variables are defined in order to enable the smoke test:
|
| |
+
|
| |
+ * *role* — this test uses role `standard-test-beakerlib` from xref:standard-test-roles.adoc[Standard Test Roles] to run a BeakerLib test
|
| |
+ * *tags* — all three test subjects (xref:standard-test-roles.adoc#_classic[classic] rpm, docker xref:standard-test-roles.adoc#_container[container] and xref:standard-test-roles.adoc#_atomic[atomic] host) are relevant for this test
|
| |
+ * *tests* — list of tests to be executed (here we have just a single smoke test)
|
| |
+ * *required_packages* — list of rpm packages required for test execution
|
| |
+
|
| |
+ There may by multiple files ending in `.yml` in the `tests/` subdirectory and each of them can represent a test or a part of a test.
|
| |
+ All of them need to be included in the main `tests.yml` file.
|
| |
+ Let's have a look at the https://src.fedoraproject.org/rpms/gzip/blob/master/f/tests[gzip] example:
|
| |
+
|
| |
+ > fedpkg clone -a gzip
|
| |
+ Cloning into 'gzip'...
|
| |
+
|
| |
+ > cd gzip/tests/
|
| |
+ > ls
|
| |
+ test-simple test_simple.yml tests.yml
|
| |
+
|
| |
+ > cat tests.yml
|
| |
+ - include: test_simple.yml
|
| |
+
|
| |
+ == Executing ==
|
| |
+
|
| |
+ Before running tests make sure you have the following dependencies installed on your system:
|
| |
+
|
| |
+ dnf install ansible python2-dnf libselinux-python standard-test-roles
|
| |
+
|
| |
+ Although some playbooks may function without sudo, tests are always invoked as root.
|
| |
+ The test itself may set up users and/or drop permissions if a part of that test.
|
| |
+ But in general be sure to be root when invoking tests.
|
| |
+
|
| |
+ [NOTE]
|
| |
+ ====
|
| |
+ *Tests may modify or destroy your environment* +
|
| |
+ It's recommended to use a virtual machine for testing to prevent any unwated changes performed by the test to your system.
|
| |
+ ====
|
| |
+
|
| |
+ Running a test directly on the current system is easy:
|
| |
+
|
| |
+ ansible-playbook tests.yml
|
| |
+
|
| |
+ To only run tests that are suited for classic systems installed by `yum` or `dnf` use the `--tags` argument:
|
| |
+
|
| |
+ ansible-playbook --tags=classic tests.yml
|
| |
+
|
| |
+ See xref:standard-test-roles.adoc[Standard Test Roles] documentation for detailed instructions how to run tests for a specific xref:standard-test-roles.adoc#_package[Rpm Package], xref:standard-test-roles.adoc#_container[Docker Container] or xref:standard-test-roles.adoc#_atomic[Atomic Host].
|
| |
+
|
| |
+ == Writing ==
|
| |
+
|
| |
+ Test code itself can be stored directly in the dist-git (recommended as default) or fetched from another repository hosted in the Fedora infrastructure such as the xref:share-test-code.adoc[Test Namespace].
|
| |
+ The simplest way to add a new test is by using one of the existing xref:standard-test-roles.adoc[Standard Test Roles] which take care of many implementatin details.
|
| |
+ If you want to create a custom test follow instructions below.
|
| |
+
|
| |
+ Once you've identified a dist-git repository you will be adding new tests to (above), you can start to write a new Ansible test.
|
| |
+ Create an http://docs.ansible.com/ansible/latest/playbooks.html[Ansible playbook] with a new name.
|
| |
+ Make sure the extension is `.yml`.
|
| |
+ Lets place the following example in `test_pid_1.yml` file.
|
| |
+
|
| |
+ [source,ansible]
|
| |
+ ----
|
| |
+ ---
|
| |
+ - hosts: localhost
|
| |
+ vars:
|
| |
+ - artifacts: ./artifacts
|
| |
+ tags:
|
| |
+ - atomic
|
| |
+ - classic
|
| |
+ - container
|
| |
+ tasks:
|
| |
+ - name: Test block
|
| |
+ block:
|
| |
+ - name: Test that /proc/1 exists
|
| |
+ shell: ls /proc > /tmp/test.log && grep -qw 1 /tmp/test.log
|
| |
+
|
| |
+ always:
|
| |
+ - name: Pull out the artifacts
|
| |
+ fetch:
|
| |
+ dest: "{{ artifacts }}/"
|
| |
+ src: "/tmp/test.log"
|
| |
+ flat: yes
|
| |
+ ----
|
| |
+
|
| |
+ All tests have an artifacts directory where they place their output.
|
| |
+ The testing or CI system that invokes the test will fill in this variable with a directory that it will archive.
|
| |
+ We ensure this directory exists in the test.
|
| |
+
|
| |
+ By use of `tags` we note what kind of systems this test is suitable to run on.
|
| |
+
|
| |
+ The `block` is the section that runs the actual test.
|
| |
+ In this example, we use a rather convoluted way of checking that PID 1 exists.
|
| |
+ However, by doing so, we place an extra test artifact in the artifacts directory.
|
| |
+
|
| |
+ Lastly, we download the artifacts.
|
| |
+ Remember that the test is not always running on the same system that it was invoked on.
|
| |
+ Try running this example test against an xref:standard-test-roles.adoc#_atomic[Atomic Host] or xref:standard-test-roles.adoc#_container[Docker Container].
|
| |
+ It should pass.
|
| |
+ Try changing the `/proc/1` argument to another value, and the test should fail.
|
| |
+
|
| |
+ You can use most of the Ansible techniques in your playbooks.
|
| |
+ Take a look at the xref:standard-test-roles.adoc[Standard Test Roles] for Ansible roles to make writing your tests easier.
|
| |
+
|
| |
+ *Marking the test to be run*
|
| |
+
|
| |
+ Just having a `.yml` file in the right directory doesn't yet mean it will be invoked.
|
| |
+ Make sure to reference or add it from a `tests.yml` playbook.
|
| |
+ This is the entry point that the testing or CI system will use to invoke all the tests for a given package.
|
| |
+
|
| |
+ If the `tests.yml` file doesn't yet exist, create it.
|
| |
+ Lets continue with our above example and create a `tests.yml` with the following content:
|
| |
+
|
| |
+ [source,ansible]
|
| |
+ ----
|
| |
+ - import_playbook: test_pid_1.yml
|
| |
+ ----
|
| |
+
|
| |
+ You can now run this test with the standard commands above.
|
| |
+
|
| |
+ See the xref:quick-start-guide.adoc#_contributing[Quick Start Guide] to get recommendations for contributing new tests.
|
| |
+
|
| |
+ == Wrapping ==
|
| |
+
|
| |
+ Let's say you have a script that runs a test.
|
| |
+ Its stdout and stderr is the test output, and an exit status of zero indicates success.
|
| |
+ Here's how we would wrap that test to be invoked.
|
| |
+ Lets say we have a simple script like in a file called `test-simple`
|
| |
+
|
| |
+ #!/bin/sh
|
| |
+ set -ex
|
| |
+ # exercise installed gzip/gunzip programs
|
| |
+ echo "Bla" > bla.file
|
| |
+ cp bla.file bla.file.orig
|
| |
+ gzip bla.file
|
| |
+ gunzip bla.file.gz
|
| |
+ cmp bla.file bla.file.orig
|
| |
+ rm bla.file bla.file.orig
|
| |
+
|
| |
+ We can write an Ansible wrapper for this script like this in `test_simple.yml`:
|
| |
+
|
| |
+ [source,ansible]
|
| |
+ ----
|
| |
+ ---
|
| |
+ - hosts: localhost
|
| |
+ vars:
|
| |
+ - artifacts: ./artifacts
|
| |
+ tags:
|
| |
+ - atomic
|
| |
+ - classic
|
| |
+ - container
|
| |
+ remote_user: root
|
| |
+ tasks:
|
| |
+ - name: Install the test files
|
| |
+ copy: src={{ item.file }} dest=/usr/local/bin/{{ item.dest }} mode=0755
|
| |
+ with_items:
|
| |
+ - {file: test-simple, dest: test-simple }
|
| |
+
|
| |
+ - name: Test block
|
| |
+ block:
|
| |
+ - name: Execute the tests
|
| |
+ shell: exec > /tmp/test.log 2>&1 && /usr/local/bin/test-simple
|
| |
+
|
| |
+ always:
|
| |
+ - name: Pull out the logs
|
| |
+ fetch:
|
| |
+ dest: "{{ artifacts }}/"
|
| |
+ src: "/tmp/test.log"
|
| |
+ flat: yes
|
| |
+ ----
|
| |
+
|
| |
+ All tests have an artifacts directory where they place their output.
|
| |
+ The testing or CI system that invokes the test will fill in this variable with a directory that it will archive.
|
| |
+ We create ensure this directory exists in the test.
|
| |
+
|
| |
+ The `block` is the section that runs the actual test.
|
| |
+
|
| |
+ Lastly, we download the artifacts.
|
| |
+ Remember that the test is not always running on the same system that it was invoked on.
|
| |
+
|
| |
+
|
| |
+ If the `tests.yml` file doesn't yet exist, create it.
|
| |
+ Lets continue with our above example and create a `tests.yml` with the following content:
|
| |
+
|
| |
+ [source,ansible]
|
| |
+ ----
|
| |
+ - import_playbook: test_simple.yml
|
| |
+ ----
|
| |
+
|
| |
+ Try running this example test against an xref:standard-test-roles.adoc#_atomic[Atomic Host] or xref:standard-test-roles.adoc#_container[Docker Container].
|
| |
+ It should pass.
|
| |
+
|
| |
+ See xref:standard-test-roles.adoc[Standard Test Roles] documentation for instructions how to wrap a xref:standard-test-roles.adoc#_beakerlib[BeakerLib] and xref:standard-test-roles.adoc#_rhts[RHTS] tests.
|
| |
+
|
| |
+ See the xref:quick-start-guide.adoc#_contributing[Quick Start Guide] to get recommendations for contributing new tests.
|
| |
+
|
| |
+ [[Category:FedoraAtomicCi]]
|
| |
+
|
| |
ph0zzy
commented 5 years ago
add page with test