#11 add page with examples
Merged 5 years ago by psss. Opened 5 years ago by ph0zzy.
fedora-ci/ ph0zzy/docs examples  into  master

file modified
+1
@@ -1,3 +1,4 @@ 

  * xref:standard-test-interface.adoc[Standard Test Interface]

  * xref:standard-test-roles.adoc[Standard Test Roles]

+ * xref:examples.adoc[Examples]

  * xref:faq.adoc[FAQ]

@@ -0,0 +1,192 @@ 

+ On this page you can find some inspiration from real-life examples of tests already enabled in the Fedora CI.

+ 

+ == did ==

+ 

+ For each component it makes sense to enable even the most simple test such as running the binary with `--help` or using an internal smoke test.

+ Here's an example from the https://src.fedoraproject.org/rpms/did/pull-request/5[did] component:

+ 

+ [source,ansible]

+ ----

+ - hosts: localhost

+   roles:

+   - role: standard-test-basic

+     tags:

+     - classic

+     tests:

+     - smoke:

+         dir: .

+         run: did this year --test

+     required_packages:

+     - did

+ ----

+ 

+ That's it.

+ As you see above, executing a single command as a test is very easy with the help of the xref:standard-test-roles.adoc#_basic[Basic] role.

+ 

+ == Python ==

+ 

+ There are multiple versions of Python programming language available in Fedora and a number of related subpackages.

+ As all of them should be tested (including their various combinatios) we https://fedoraproject.org/wiki/CI/Share_Test_Code[share] test coverage for them in the `tests` namespace:

+ 

+ * https://src.fedoraproject.org/tests/python[Python tests]

+ 

+ The test repo contains basic smoke test for virtualenv together with example test https://pagure.io/fedora-ci/metadata[Metadata] stored in the https://fedoraproject.org/wiki/Flexible_Metadata_Format[Flexible Metadata Format]:

+ 

+ * https://src.fedoraproject.org/tests/python/blob/master/f/main.fmf[main.fmf]

+ * https://src.fedoraproject.org/tests/python/blob/master/f/smoke/venv.fmf[venv.fmf]

+ 

+ Once the test is avaible in the share test repository it can be easily linked from supported Python versions:

+ 

+ Python 2::

+ * https://src.fedoraproject.org/rpms/python2/blob/master/f/tests/tests.yml[python2]

+ * https://src.fedoraproject.org/rpms/python26/blob/master/f/tests/tests.yml[python26]

+ Python 3::

+ * https://src.fedoraproject.org/rpms/python3/blob/master/f/tests/tests.yml[python3]

+ * https://src.fedoraproject.org/rpms/python34/blob/master/f/tests/tests.yml[python34]

+ * https://src.fedoraproject.org/rpms/python35/blob/master/f/tests/tests.yml[python35]

+ * https://src.fedoraproject.org/rpms/python36/blob/master/f/tests/tests.yml[python36]

+ 

+ We test additional Python implementations as well::

+ 

+ * https://src.fedoraproject.org/rpms/pypy/blob/master/f/tests/tests.yml[pypy]

+ * https://src.fedoraproject.org/rpms/pypy3/blob/master/f/tests/tests.yml[pypy3]

+ 

+ Plus we ensure that essential tools for venv and virtualnv, such as `setuptools`, `pip` or `virtualenv` itself correctly work with all supported versions:

+ 

+ * https://src.fedoraproject.org/rpms/python-pip/blob/master/f/tests/tests.yml[python-pip]

+ * https://src.fedoraproject.org/rpms/python-wheel/blob/master/f/tests/tests.yml[python-wheel]

+ * https://src.fedoraproject.org/rpms/python-setuptools/blob/master/f/tests/tests.yml[python-setuptools]

+ * https://src.fedoraproject.org/rpms/python-virtualenv/blob/master/f/tests/tests.yml[python-virtualenv]

+ * https://src.fedoraproject.org/rpms/python-tox/blob/master/f/tests/tests.yml[python-tox]

+ 

+ Note that for the last set of examples we run the same test several times with modified environment.

+ For example:

+ 

+ [source,ansible]

+ ----

+ - smoke36:

+     dir: python/smoke

+     run: VERSION=3.6 ./venv.sh

+ - smoke37:

+     dir: python/smoke

+     run: VERSION=3.7 ./venv.sh

+ - smoke26:

+     dir: python/smoke

+     run: VERSION=2.6 METHOD=virtualenv TOX=false ./venv.sh

+ - smoke27:

+     dir: python/smoke

+     run: VERSION=2.7 METHOD=virtualenv ./venv.sh

+ - smoke34_virtualenv:

+     dir: python/smoke

+     run: VERSION=3.4 METHOD=virtualenv ./venv.sh

+ ----

+ 

+ In this way we create several virtual test cases from a single test code which prevents duplication and minimizes future maintenance.

+ 

+ == Shell ==

+ 

+ There are several shells which implement the POSIX specification: bash, ksh, mksh, zsh, dash.

+ All of them share a significant amount of test coverage and it does not make sense to commit & maintain identical tests in five different repositories (+ possible branches).

+ Thus we store test code in the `tests` namespace:

+ 

+ * https://src.fedoraproject.org/tests/shell[Shell tests]

+ 

+ These tests are then linked from all relevant `tests.yml` files:

+ 

+ * https://src.fedoraproject.org/rpms/bash/blob/master/f/tests/tests.yml[bash]

+ * https://src.fedoraproject.org/rpms/ksh/blob/master/f/tests/tests.yml[ksh]

+ * https://src.fedoraproject.org/rpms/mksh/blob/master/f/tests/tests.yml[mksh]

+ * https://src.fedoraproject.org/rpms/zsh/blob/master/f/tests/tests.yml[zsh]

+ * https://src.fedoraproject.org/rpms/dash/blob/master/f/tests/tests.yml[dash]

+ 

+ https://fedoraproject.org/wiki/Flexible_Metadata_Format[Flexible Metadata Format] filter is used to select appropriate tests instead of listing individual tests manually.

+ Environment variables `PACKAGES` and `SH_BIN` are used to specify which shell implementation is being tested:

+ 

+ [source,ansible]

+ ----

+ - hosts: localhost

+   roles:

+   - role: standard-test-beakerlib

+     tags:

+     - classic

+     repositories:

+     - repo: "https://src.fedoraproject.org/tests/shell.git"

+       dest: "shell"

+       fmf_filter: "tier: 1, 2 & tags: classic"

+     environment:

+       PACKAGES: ksh

+       SH_BIN: ksh

+     required_packages:

+     - ksh

+     - expect            # login requires expect

+     - which             # smoke requires which

+ ----

+ 

+ Some of the tests might be relevant only for selected components.

+ This can be handled easily by additional `component` condition:

+ 

+ [source,ansible]

+ ----

+ repositories:

+ - repo: "https://src.fedoraproject.org/tests/shell.git"

+   dest: "shell"

+   fmf_filter: "tier: 1, 2 & component: dash"

+ ----

+ 

+ See the https://pagure.io/fedora-ci/metadata[Metadata] page for the full list of so far drafted attributes.

+ 

+ == SELinux ==

+ 

+ There are several components related to SELinux.

+ They are tightly connected so change in one of them can cause problems in other.

+ That's why their tests are shared and executed together:

+ 

+ * https://src.fedoraproject.org/tests/selinux[SELinux]

+ 

+ Instead of listing relevant tests to be executed manually in each dist git rpms repository https://fedoraproject.org/wiki/Flexible_Metadata_Format[Flexible Metadata Format] is used:

+ 

+ [source,ansible]

+ ----

+ - hosts: localhost

+   roles:

+   - role: standard-test-beakerlib

+     tags:

+     - classic

+     repositories:

+     - repo: "https://src.fedoraproject.org/tests/selinux.git"

+       dest: "selinux"

+       fmf_filter: "tier: 1 | component: selinux-policy"

+ ----

+ 

+ Provided `fmf_filter` selects all tests relevant for the `selinux-policy` component plus all Tier 1 selinux tests:

+ 

+  tier: 1 | component: selinux-policy

+ 

+ The following six components are covered:

+ 

+ * https://src.fedoraproject.org/rpms/checkpolicy/blob/master/f/tests/tests.yml[checkpolicy]

+ * https://src.fedoraproject.org/rpms/libselinux/blob/master/f/tests/tests.yml[libselinux]

+ * https://src.fedoraproject.org/rpms/libsemanage/blob/master/f/tests/tests.yml[libsemanage]

+ * https://src.fedoraproject.org/rpms/libsepol/blob/master/f/tests/tests.yml[libsepol]

+ * https://src.fedoraproject.org/rpms/policycoreutils/blob/master/f/tests/tests.yml[policycoreutils]

+ * https://src.fedoraproject.org/rpms/selinux-policy/blob/master/f/tests/tests.yml[selinux-policy]

+ 

+ Use the `fmf` command line tool to quickly check which tests will be scheduled:

+ 

+  # dnf install -y fmf

+  # fedpkg clone -a tests/selinux

+  # cd selinux

+  # fmf ls --filter "tier: 1 | component: checkpolicy"

+  /selinux-policy/policy-rpm-macros

+  /checkpolicy/sedispol

+  /checkpolicy/checkmodule

+  /checkpolicy/sedismod

+  /checkpolicy/checkpolicy

+  /checkpolicy/checkpolicy-docs

+  /libsepol/sepol_check_context

+  /libsemanage/verify-options-in-semanage-conf

+  /libselinux/getsebool

+  /policycoreutils/booleans

+ 

+ See the Flexible Metadata Format documentation for other options how to https://fmf.readthedocs.io/en/latest/overview.html#install[install] fmf.

+ 

add page with examples

Metadata Update from @psss:
- Request assigned

5 years ago

Pull-Request has been merged by psss

5 years ago

Thanks for the examples, Arthur. I've added main title and updated a few links in ec4397e. Also using definition list for the Python examples looked a bit too much bold font around so I changed that into a plain text.

Also note that the stats page will probably not be moved as it is auto-updated by a script directly on the wiki. I've updated the link accordingly.