| |
@@ -0,0 +1,182 @@
|
| |
+ :toc:
|
| |
+
|
| |
+ = Share Test Code =
|
| |
+
|
| |
+ == Motivation ==
|
| |
+
|
| |
+ In order to make the CI workflow reliable and efficient it is crucial to keep the test coverage in a good shape at all times.
|
| |
+ Sharing test code between several packages (even within multiple branches of the same package) may significantly help to:
|
| |
+
|
| |
+ * Prevent test code duplication
|
| |
+ * Minimize test maintenance
|
| |
+ * Catch incompatibilities early
|
| |
+
|
| |
+ In general, tests define how the software works and the basic functionality of many packages doesn’t change that often.
|
| |
+ We try hard to keep the backward compatibility where possible.
|
| |
+ Thus it seems natural that, for such components, tests guarding the spec could change at a slower pace than the distribution branches.
|
| |
+
|
| |
+ See the whole https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org/thread/55U6V6UHA54MJLD2F6JF46EOLMVRUAE7[ci-list discussion] for some more context.
|
| |
+
|
| |
+ == Implementation ==
|
| |
+
|
| |
+ Store test code in your preferred repository and reference the tests from the dist-git yaml file.
|
| |
+ There is also a special `tests` namespace dedicated for storing Fedora CI integration tests:
|
| |
+
|
| |
+ * https://src.fedoraproject.org/projects/tests
|
| |
+
|
| |
+ Use `fedpkg` to quickly clone repositories from the tests namespace:
|
| |
+
|
| |
+ fedpkg clone tests/shell
|
| |
+
|
| |
+ Some of the xref:standard-test-roles.adoc[Standard Test Roles] (currently basic and beakerlib) support fetching test code from remote repositories directly in their config in this way:
|
| |
+
|
| |
+ [source,ansible]
|
| |
+ ----
|
| |
+ - role: standard-test-beakerlib
|
| |
+ repositories:
|
| |
+ - repo: "https://src.fedoraproject.org/tests/shell.git"
|
| |
+ dest: "shell"
|
| |
+ ----
|
| |
+
|
| |
+ It is also possible to specify version (branch, commit hash) which should be fetched from the remote repository:
|
| |
+
|
| |
+ [source,ansible]
|
| |
+ ----
|
| |
+ - role: standard-test-beakerlib
|
| |
+ repositories:
|
| |
+ - repo: "https://src.fedoraproject.org/tests/shell.git"
|
| |
+ dest: "shell"
|
| |
+ version: "devel"
|
| |
+ ----
|
| |
+
|
| |
+ == Testing Tests ==
|
| |
+
|
| |
+ It is a good idea to ensure that updating tests in the shared repository does not negatively impact packages which they are testing.
|
| |
+ To enable pull request pipeline for tests stored in the Fedora dist git tests namespace simply include `tests.yml` file in the root of the test repository.
|
| |
+
|
| |
+ [source,ansible]
|
| |
+ ----
|
| |
+ - hosts: localhost
|
| |
+ roles:
|
| |
+ - role: standard-test-basic
|
| |
+ tags:
|
| |
+ - classic
|
| |
+ tests:
|
| |
+ - smoke27:
|
| |
+ dir: smoke
|
| |
+ run: VERSION=2.7 METHOD=virtualenv ./venv.sh
|
| |
+ - smoke36:
|
| |
+ dir: smoke
|
| |
+ run: VERSION=3.6 METHOD=virtualenv ./venv.sh
|
| |
+ - smoke37:
|
| |
+ dir: smoke
|
| |
+ run: VERSION=3.7 METHOD=virtualenv ./venv.sh
|
| |
+ required_packages:
|
| |
+ - python27
|
| |
+ - python36
|
| |
+ - python37
|
| |
+ ----
|
| |
+
|
| |
+ The example above is a simplified version of the https://src.fedoraproject.org/tests/python/blob/master/f/tests.yml[tests.yml] file from the Python shared test repo and shows how to enable `smoke` test to be executed against three versions of the Python interpreter.
|
| |
+
|
| |
+ == Examples ==
|
| |
+
|
| |
+ Here are some real-life examples where sharing test code can increase long-term efficiency.
|
| |
+
|
| |
+ === Shell ===
|
| |
+
|
| |
+ There are several shells which implement the POSIX specification:
|
| |
+
|
| |
+ * bash
|
| |
+ * ksh
|
| |
+ * mksh
|
| |
+ * zsh
|
| |
+ * dash
|
| |
+
|
| |
+ All of them share a significant amount of test coverage and it does not make sense to commit & maintain identical tests in five different repositories (+ possible branches).
|
| |
+
|
| |
+ Shell tests repository:
|
| |
+
|
| |
+ * https://src.fedoraproject.org/tests/shell
|
| |
+
|
| |
+ Bash https://src.fedoraproject.org/rpms/bash/blob/master/f/tests/tests.yml[tests.yml]:
|
| |
+
|
| |
+ [source,ansible]
|
| |
+ ----
|
| |
+ - hosts: localhost
|
| |
+ roles:
|
| |
+ - role: standard-test-beakerlib
|
| |
+ tags:
|
| |
+ - classic
|
| |
+ repositories:
|
| |
+ - repo: "https://src.fedoraproject.org/tests/shell.git"
|
| |
+ dest: "shell"
|
| |
+ tests:
|
| |
+ - shell/func
|
| |
+ - shell/login
|
| |
+ - shell/smoke
|
| |
+ required_packages:
|
| |
+ - expect # login requires expect
|
| |
+ - which # smoke requires which
|
| |
+ ----
|
| |
+
|
| |
+ Ksh https://src.fedoraproject.org/rpms/ksh/blob/master/f/tests/tests.yml[tests.yml]:
|
| |
+
|
| |
+ [source,ansible]
|
| |
+ ----
|
| |
+ - hosts: localhost
|
| |
+ roles:
|
| |
+ - role: standard-test-beakerlib
|
| |
+ tags:
|
| |
+ - classic
|
| |
+ repositories:
|
| |
+ - repo: "https://src.fedoraproject.org/tests/shell.git"
|
| |
+ dest: "shell"
|
| |
+ tests:
|
| |
+ - shell/func
|
| |
+ - shell/login
|
| |
+ - shell/smoke
|
| |
+ environment:
|
| |
+ PACKAGES: ksh
|
| |
+ SH_BIN: ksh
|
| |
+ required_packages:
|
| |
+ - ksh
|
| |
+ - expect # login requires expect
|
| |
+ - which # smoke requires which
|
| |
+ ----
|
| |
+
|
| |
+ === Ruby ===
|
| |
+
|
| |
+ Another example is Ruby: With about 80 packages related to Ruby on Rails it would be useful and efficient to have a single place for integration tests which verify that the framework is correctly working after updating any of these packages.
|
| |
+ Conversely, maintaining those tests in 80 repos would be a tedious task.
|
| |
+
|
| |
+ Currently the shared https://src.fedoraproject.org/tests/ruby[tests/ruby] repository hosts these three ruby integration tests:
|
| |
+
|
| |
+ Available integration tests:
|
| |
+
|
| |
+ * systemtap-static-probes-in-ruby - exercising ruby's systemtap api
|
| |
+ * bundler-unit-test - run bundler's unit tests
|
| |
+ * run-basic-rails-application - run a simple rails application
|
| |
+
|
| |
+ === SELinux ===
|
| |
+
|
| |
+ Several SELinux user space components are sharing test coverage in a single https://src.fedoraproject.org/tests/selinux.git[selinux] test repository:
|
| |
+
|
| |
+ * https://src.fedoraproject.org/rpms/libsepol/blob/master/f/tests/tests.yml[libsepol]
|
| |
+ * https://src.fedoraproject.org/rpms/libselinux/blob/master/f/tests/tests.yml[libselinux]
|
| |
+ * https://src.fedoraproject.org/rpms/libsemanage/blob/master/f/tests/tests.yml[libsemanage]
|
| |
+ * https://src.fedoraproject.org/rpms/libsepol/blob/master/f/tests/tests.yml[policycoreutils]
|
| |
+
|
| |
+ == Start ==
|
| |
+
|
| |
+ In order to create a new repository in the tests namespace use the fedpkg's request-tests-repo command.
|
| |
+ For example to create a shared test repository with the name foo, which will be available at https://src.fedoraproject.org/tests/foo.git
|
| |
+
|
| |
+ * Setup authentication to pagure according to the help in request-repo command
|
| |
+
|
| |
+ fedpkg request-repo -h
|
| |
+
|
| |
+ * Request a new repository with a sensible decription
|
| |
+
|
| |
+ fedpkg --module-name foo request-tests-repo "Description of the foo repository"
|
| |
+
|
| |
ph0zzy
commented 5 years ago
add share test code page