| |
@@ -0,0 +1,192 @@
|
| |
+ On this page you can find some inspiration from real-life examples of tests already enabled in the Fedora CI.
|
| |
+
|
| |
+ == did ==
|
| |
+
|
| |
+ For each component it makes sense to enable even the most simple test such as running the binary with `--help` or using an internal smoke test.
|
| |
+ Here's an example from the https://src.fedoraproject.org/rpms/did/pull-request/5[did] component:
|
| |
+
|
| |
+ [source,ansible]
|
| |
+ ----
|
| |
+ - hosts: localhost
|
| |
+ roles:
|
| |
+ - role: standard-test-basic
|
| |
+ tags:
|
| |
+ - classic
|
| |
+ tests:
|
| |
+ - smoke:
|
| |
+ dir: .
|
| |
+ run: did this year --test
|
| |
+ required_packages:
|
| |
+ - did
|
| |
+ ----
|
| |
+
|
| |
+ That's it.
|
| |
+ As you see above, executing a single command as a test is very easy with the help of the xref:standard-test-roles.adoc#_basic[Basic] role.
|
| |
+
|
| |
+ == Python ==
|
| |
+
|
| |
+ There are multiple versions of Python programming language available in Fedora and a number of related subpackages.
|
| |
+ As all of them should be tested (including their various combinatios) we https://fedoraproject.org/wiki/CI/Share_Test_Code[share] test coverage for them in the `tests` namespace:
|
| |
+
|
| |
+ * https://src.fedoraproject.org/tests/python[Python tests]
|
| |
+
|
| |
+ The test repo contains basic smoke test for virtualenv together with example test https://pagure.io/fedora-ci/metadata[Metadata] stored in the https://fedoraproject.org/wiki/Flexible_Metadata_Format[Flexible Metadata Format]:
|
| |
+
|
| |
+ * https://src.fedoraproject.org/tests/python/blob/master/f/main.fmf[main.fmf]
|
| |
+ * https://src.fedoraproject.org/tests/python/blob/master/f/smoke/venv.fmf[venv.fmf]
|
| |
+
|
| |
+ Once the test is avaible in the share test repository it can be easily linked from supported Python versions:
|
| |
+
|
| |
+ Python 2::
|
| |
+ * https://src.fedoraproject.org/rpms/python2/blob/master/f/tests/tests.yml[python2]
|
| |
+ * https://src.fedoraproject.org/rpms/python26/blob/master/f/tests/tests.yml[python26]
|
| |
+ Python 3::
|
| |
+ * https://src.fedoraproject.org/rpms/python3/blob/master/f/tests/tests.yml[python3]
|
| |
+ * https://src.fedoraproject.org/rpms/python34/blob/master/f/tests/tests.yml[python34]
|
| |
+ * https://src.fedoraproject.org/rpms/python35/blob/master/f/tests/tests.yml[python35]
|
| |
+ * https://src.fedoraproject.org/rpms/python36/blob/master/f/tests/tests.yml[python36]
|
| |
+
|
| |
+ We test additional Python implementations as well::
|
| |
+
|
| |
+ * https://src.fedoraproject.org/rpms/pypy/blob/master/f/tests/tests.yml[pypy]
|
| |
+ * https://src.fedoraproject.org/rpms/pypy3/blob/master/f/tests/tests.yml[pypy3]
|
| |
+
|
| |
+ Plus we ensure that essential tools for venv and virtualnv, such as `setuptools`, `pip` or `virtualenv` itself correctly work with all supported versions:
|
| |
+
|
| |
+ * https://src.fedoraproject.org/rpms/python-pip/blob/master/f/tests/tests.yml[python-pip]
|
| |
+ * https://src.fedoraproject.org/rpms/python-wheel/blob/master/f/tests/tests.yml[python-wheel]
|
| |
+ * https://src.fedoraproject.org/rpms/python-setuptools/blob/master/f/tests/tests.yml[python-setuptools]
|
| |
+ * https://src.fedoraproject.org/rpms/python-virtualenv/blob/master/f/tests/tests.yml[python-virtualenv]
|
| |
+ * https://src.fedoraproject.org/rpms/python-tox/blob/master/f/tests/tests.yml[python-tox]
|
| |
+
|
| |
+ Note that for the last set of examples we run the same test several times with modified environment.
|
| |
+ For example:
|
| |
+
|
| |
+ [source,ansible]
|
| |
+ ----
|
| |
+ - smoke36:
|
| |
+ dir: python/smoke
|
| |
+ run: VERSION=3.6 ./venv.sh
|
| |
+ - smoke37:
|
| |
+ dir: python/smoke
|
| |
+ run: VERSION=3.7 ./venv.sh
|
| |
+ - smoke26:
|
| |
+ dir: python/smoke
|
| |
+ run: VERSION=2.6 METHOD=virtualenv TOX=false ./venv.sh
|
| |
+ - smoke27:
|
| |
+ dir: python/smoke
|
| |
+ run: VERSION=2.7 METHOD=virtualenv ./venv.sh
|
| |
+ - smoke34_virtualenv:
|
| |
+ dir: python/smoke
|
| |
+ run: VERSION=3.4 METHOD=virtualenv ./venv.sh
|
| |
+ ----
|
| |
+
|
| |
+ In this way we create several virtual test cases from a single test code which prevents duplication and minimizes future maintenance.
|
| |
+
|
| |
+ == Shell ==
|
| |
+
|
| |
+ There are several shells which implement the POSIX specification: bash, ksh, mksh, zsh, dash.
|
| |
+ All of them share a significant amount of test coverage and it does not make sense to commit & maintain identical tests in five different repositories (+ possible branches).
|
| |
+ Thus we store test code in the `tests` namespace:
|
| |
+
|
| |
+ * https://src.fedoraproject.org/tests/shell[Shell tests]
|
| |
+
|
| |
+ These tests are then linked from all relevant `tests.yml` files:
|
| |
+
|
| |
+ * https://src.fedoraproject.org/rpms/bash/blob/master/f/tests/tests.yml[bash]
|
| |
+ * https://src.fedoraproject.org/rpms/ksh/blob/master/f/tests/tests.yml[ksh]
|
| |
+ * https://src.fedoraproject.org/rpms/mksh/blob/master/f/tests/tests.yml[mksh]
|
| |
+ * https://src.fedoraproject.org/rpms/zsh/blob/master/f/tests/tests.yml[zsh]
|
| |
+ * https://src.fedoraproject.org/rpms/dash/blob/master/f/tests/tests.yml[dash]
|
| |
+
|
| |
+ https://fedoraproject.org/wiki/Flexible_Metadata_Format[Flexible Metadata Format] filter is used to select appropriate tests instead of listing individual tests manually.
|
| |
+ Environment variables `PACKAGES` and `SH_BIN` are used to specify which shell implementation is being tested:
|
| |
+
|
| |
+ [source,ansible]
|
| |
+ ----
|
| |
+ - hosts: localhost
|
| |
+ roles:
|
| |
+ - role: standard-test-beakerlib
|
| |
+ tags:
|
| |
+ - classic
|
| |
+ repositories:
|
| |
+ - repo: "https://src.fedoraproject.org/tests/shell.git"
|
| |
+ dest: "shell"
|
| |
+ fmf_filter: "tier: 1, 2 & tags: classic"
|
| |
+ environment:
|
| |
+ PACKAGES: ksh
|
| |
+ SH_BIN: ksh
|
| |
+ required_packages:
|
| |
+ - ksh
|
| |
+ - expect # login requires expect
|
| |
+ - which # smoke requires which
|
| |
+ ----
|
| |
+
|
| |
+ Some of the tests might be relevant only for selected components.
|
| |
+ This can be handled easily by additional `component` condition:
|
| |
+
|
| |
+ [source,ansible]
|
| |
+ ----
|
| |
+ repositories:
|
| |
+ - repo: "https://src.fedoraproject.org/tests/shell.git"
|
| |
+ dest: "shell"
|
| |
+ fmf_filter: "tier: 1, 2 & component: dash"
|
| |
+ ----
|
| |
+
|
| |
+ See the https://pagure.io/fedora-ci/metadata[Metadata] page for the full list of so far drafted attributes.
|
| |
+
|
| |
+ == SELinux ==
|
| |
+
|
| |
+ There are several components related to SELinux.
|
| |
+ They are tightly connected so change in one of them can cause problems in other.
|
| |
+ That's why their tests are shared and executed together:
|
| |
+
|
| |
+ * https://src.fedoraproject.org/tests/selinux[SELinux]
|
| |
+
|
| |
+ Instead of listing relevant tests to be executed manually in each dist git rpms repository https://fedoraproject.org/wiki/Flexible_Metadata_Format[Flexible Metadata Format] is used:
|
| |
+
|
| |
+ [source,ansible]
|
| |
+ ----
|
| |
+ - hosts: localhost
|
| |
+ roles:
|
| |
+ - role: standard-test-beakerlib
|
| |
+ tags:
|
| |
+ - classic
|
| |
+ repositories:
|
| |
+ - repo: "https://src.fedoraproject.org/tests/selinux.git"
|
| |
+ dest: "selinux"
|
| |
+ fmf_filter: "tier: 1 | component: selinux-policy"
|
| |
+ ----
|
| |
+
|
| |
+ Provided `fmf_filter` selects all tests relevant for the `selinux-policy` component plus all Tier 1 selinux tests:
|
| |
+
|
| |
+ tier: 1 | component: selinux-policy
|
| |
+
|
| |
+ The following six components are covered:
|
| |
+
|
| |
+ * https://src.fedoraproject.org/rpms/checkpolicy/blob/master/f/tests/tests.yml[checkpolicy]
|
| |
+ * https://src.fedoraproject.org/rpms/libselinux/blob/master/f/tests/tests.yml[libselinux]
|
| |
+ * https://src.fedoraproject.org/rpms/libsemanage/blob/master/f/tests/tests.yml[libsemanage]
|
| |
+ * https://src.fedoraproject.org/rpms/libsepol/blob/master/f/tests/tests.yml[libsepol]
|
| |
+ * https://src.fedoraproject.org/rpms/policycoreutils/blob/master/f/tests/tests.yml[policycoreutils]
|
| |
+ * https://src.fedoraproject.org/rpms/selinux-policy/blob/master/f/tests/tests.yml[selinux-policy]
|
| |
+
|
| |
+ Use the `fmf` command line tool to quickly check which tests will be scheduled:
|
| |
+
|
| |
+ # dnf install -y fmf
|
| |
+ # fedpkg clone -a tests/selinux
|
| |
+ # cd selinux
|
| |
+ # fmf ls --filter "tier: 1 | component: checkpolicy"
|
| |
+ /selinux-policy/policy-rpm-macros
|
| |
+ /checkpolicy/sedispol
|
| |
+ /checkpolicy/checkmodule
|
| |
+ /checkpolicy/sedismod
|
| |
+ /checkpolicy/checkpolicy
|
| |
+ /checkpolicy/checkpolicy-docs
|
| |
+ /libsepol/sepol_check_context
|
| |
+ /libsemanage/verify-options-in-semanage-conf
|
| |
+ /libselinux/getsebool
|
| |
+ /policycoreutils/booleans
|
| |
+
|
| |
+ See the Flexible Metadata Format documentation for other options how to https://fmf.readthedocs.io/en/latest/overview.html#install[install] fmf.
|
| |
+
|
| |
add page with examples