fedora-qa / fedora_openqa

Created 7 years ago
Maintained by adamwill
Fedora openQA job scheduler and result forwarders  |  https://openqa.fedoraproject.org
Members 3

Fedora openQA scheduler and report tools

This repository contains a Python library (fedora_openqa) and CLI tool (fedora-openqa) that handle scheduling jobs in Fedora's instances (production and staging) of openQA. The library contains several fedmsg consumers which together form a fully automated pipeline for running tests in Fedora's official openQA instances. Jobs are scheduled in response to fedmsgs from release engineering. When a job completes, a fedmsg is emitted, and the 'reporter' consumers forward results to Wikitcms and ResultsDB. The CLI allows manual triggering of jobs and reporting of results for testing with a local openQA deployment, or in case some issue in the automated pipeline requires manual intervention.

This repository does not contain the actual Fedora openQA tests. Those can be found here.

For general information on openQA in Fedora, including an overview of the system and how to install your own instance for testing, please see the Fedora openQA wiki page.

Issues and contributions

Issues and pull requests are tracked in fedora_openqa Pagure. Pagure uses a Github-like pull request workflow, so if you're familiar with that, you can easily submit Pagure pull requests. If not, you can read up in the Pagure documentation. CI is enabled for the repository, so pull requests are automatically tested. To run the tests locally, just run tox.

Requirements

NOTE: If you have deployed openQA in docker containers, note that this tool should be configured and run (and installed, if desired) on the host, not in any of the containers.

To use the fedora-openqa CLI, you do not have to do a system install, you can run it directly from the source tree. However, you do need to ensure some dependencies are available. Most of them are available as packages on Fedora, and can be installed as follows:

dnf install python3-setuptools python3-six python3-requests python3-fedfind python3-wikitcms python3-fedora-messaging python3-resultsdb_conventions-fedora python3-openqa_client

You can then run ./fedora-openqa.py from /home/someuser/local/fedora_openqa to use the CLI tool.

CLI usage

Simple usages are:

./fedora-openqa.py compose https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20170214.n.0/compose/
./fedora-openqa.py update https://bodhi.fedoraproject.org/updates/FEDORA-2020-424d2b310e
./fedora-openqa.py task 12345678 39
./fedora-openqa.py tag a-tag-name 39
./fedora-openqa.py report --wiki Fedora-Rawhide-20170214.n.0
./fedora-openqa.py report --resultsdb 1 2 3 4 5

The compose subcommand schedules jobs for a particular compose, update for a particular update. task schedules the update jobs for a Koji task instead of an update (this is mainly useful for testing scratch builds). tag schedules the update jobs for a tag; in this case, instead of using an additional repo containing the packages from the update or task, the tests will configure the Koji repo for the specified tag as an additional repo. Note update does not require a release number (it will be deduced from the update's properties), but task and tag require you to specify the release to test on, as we cannot easily deduce it. For testing on Rawhide, pass the currently-corresponding release number. The first report command reports results for all jobs for a given compose to the wiki. The second reports results for the specified jobs to ResultsDB. You can use report without --wiki or --resultsdb to produce a list of passed Wikitcms test cases without reporting them anywhere.

See the command's help (and the help for the subcommands) for more details on usage.

Installation

You can install the library and CLI systemwide if you choose to. After installing all the dependencies, just run pip install ./. You must install systemwide if you wish to use the fedora-messaging consumers to trigger jobs automatically.

Configuration

If the openQA server is deployed directly on the same system, openQA API access credentials should already be in place. If you are interacting with a remote openQA server, or one running in a container, you will need to configure these. First, visit http://openqa.server(:PORT)/api_keys (where http://openqa.server is the correct URL for the server) and authenticate as an administrator if necessary, and you should be able to generate an API key and secret. Then create a file /etc/openqa/client.conf or ~/.config/openqa/client.conf with this content:

[openqa.server(:PORT)]
key = KEY
secret = SECRET

By default the CLI will use the first openQA server specified in this config file. You can pass the --openqa-hostname argument to override this, but you must have a key and secret specified for the chosen server to create jobs on it (no key or secret are needed for result reporting, as this only requires querying of the openQA server).

If you wish to forward results to Wikitcms, you must either authenticate interactively via a browser (which requires a graphical environment) periodically - each time you do this, a token will be kept for around a week, during which time reporting will work non-interactively, until one day you'll be prompted to authenticate again - or request a special non-expiring token from the wiki administrator. Please be careful before doing this, as usually only the official Fedora openQA systems should report results to Wikitcms. Ideally this should be a dedicated account for the purpose of reporting test results.

This tool has its own configuration file which can be installed to /etc/fedora-openqa/schedule.conf or ~/.config/fedora-openqa/schedule.conf. In this config file you can specify the locations of the wiki and ResultsDB instance that will be used when reporting results with fedora-openqa report; by default, results will be reported to the staging wiki and to a ResultsDB instance running on localhost port 5001 (which is what you get if you follow the instructions to do a local deployment of ResultsDB for testing). A sample config file is provided as sample-configs/schedule.conf.sample, which you can copy into place and modify.

You can configure the set of images from each compose which will be downloaded and tested. For more details on this, see the comments in sample-configs/images.json.sample.

To run openQA jobs whenever a compose completes, and to report results to a ResultsDB instance and/or a wiki, you can use the fedora-messaging systemd service pattern.

PLEASE NOTE that there should never be more than one Fedora production Wiki or ResultsDB reporter enabled in the world, and both of these are run in the Fedora infrastructure, so please don't enable these on your own deployments. Reporting to your own wiki or ResultsDB instance for testing is of course fine.

First, you need to install one or more consumer configuration files to /etc/fedora-messaging. There are sample files provided in sample-configs/ for Fedora production and staging configurations for the job scheduler, ResultsDB reporter and Wiki reporter:

fedora_openqa_resultsdb_reporter.stg.toml
fedora_openqa_resultsdb_reporter.toml
fedora_openqa_scheduler.stg.toml
fedora_openqa_scheduler.toml
fedora_openqa_wiki_reporter.stg.toml
fedora_openqa_wiki_reporter.toml

You can copy these into place and modify them as you like. You will at least need to replace the dummy UUID in each file (00000000-0000-0000-0000-000000000000) with a unique one generated by uuidgen, as explained in the comments. You may also change the authentication configuration and the settings in consumer_config. openqa_hostname is the openQA hostname to schedule jobs on (for the scheduler) or to retrieve the full job result details from (for the reporters). openqa_baseurl is the base URL to use for constructing links back to the results (for the reporters). For the Wiki reporter, wiki_hostname is the hostname of the wiki to send results to. For the ResultsDB reporter, resultsdb_url is the URL to send the results to (it should be the top-level API URL). For both reporter plugins, do_report configures whether to actually send the reports (if it is set false, the consumer will just log what it would have reported instead of actually reporting it).

You will also need to install the fedora-messaging package:

sudo dnf install fedora-messaging

Once the configuration file(s) is/are in place, you can enable and start each consumer as a systemd service, for e.g.:

sudo systemctl enable fm-consumer@fedora_openqa_scheduler
sudo systemctl enable fedora_openqa_wiki_reporter.stg
sudo systemctl start fm-consumer@fedora_openqa_scheduler
sudo systemctl start fedora_openqa_wiki_reporter.stg

Once again, please please do NOT enable result reporter consumers unless you're really sure you should be doing it, and they are definitely not pointed at Fedora's production wiki or ResultsDB.

Also be aware that the result reporter consumer configurations are set to respond to jobs run in the official Fedora openQA deployments, not your own deployment. If you want to test the reporting workflow with jobs run on any other openQA instance, you will need to configure that instance such that fedmsgs emitted by openQA are forwarded to a fedora-messaging broker that the reporter consumers can subscribe to, and then configure the reporter consumers to subscribe to that broker instead. This is all somewhat outside the scope of this document. Usually, these consumers will only be used by the official Fedora openQA deployments.

vim: set ts=8 et sw=4: