About

These playbooks reproduce Fedora's OpenQA deployment with minimal dependencies so that OpenQA can be run easily on a local machine or with cloud resources. General info about Fedora's OpenQA is here: https://fedoraproject.org/wiki/OpenQA

There are two main components: a server and workers to run the tests. The playbook openqa-cloud-server.yml will start the server and its related components in containers. The playbook openqa-cloud-worker-container.yml will start workers in containers.

1. Example: a Local Deployment

Start the openQA server containers using the Demo account, "Fake" authentication, and self-signed certificates:

ansible-playbook \
-e web_ip_private=$(hostname -I | awk '{print $1}') \
-e auth=Fake \
-e use_http=true \
-e dir=$HOME/my_local_cloud_server \
--ask-become-pass \
playbooks/hosts/openqa-cloud-server.yml

The first time it runs, it will take a while to build the podman images. Once the ansible playbook is finished, the containers will probably still be initializing; check on the containers with podman ps -a and journalctl -f. Keep checking localhost in your browser or just curl localhost. Make sure your browser will let you load a web page with self-signed certificates.

Once localhost is displaying the openQA user interface, there will be several new systemd services running on the host machine:
- openqa-dispatcher: runs fedora_openqa to listen for new Fedora images and updates to be tested;
- openqa-database: holds user information and test jobs;
- openqa-webserver: runs the web user interface, test scheduler, websockets and live handler for talking with the workers;
- openqa-reverse-proxy: handles outside requests and ssl/tls certificates;
- openqa-test-update: a small service to regularly pull test and needle updates from os-autoinst-distri-fedora.

Once the server is up, click the login button on the user interface. Even though this example is using 'Fake' authentication, the test templates will be loaded using the Demo user's api key/secret. If the Demo user isn't logged in, the test templates won't load. Once the Demo user is logged in, load the test templates with this command:

podman exec -it openqa-webserver /bin/bash -c "cd /var/lib/openqa/share/tests/fedora/;
./fifloader.py --load  templates.fif.json templates-updates.fif.json;"

The successfully loaded templates will appear in the Job Groups drop-down menu.

After the test templates are loaded, schedule a build for testing. If you wait long enough, the openqa-dispatcher will pick up fedora messages with an image to test. Alternatively, to schedule jobs manually, get a BUILDURL from the "Settings" tab of any test by clicking through the test's coloured dot: https://openqa.fedoraproject.org. Fedora Cloud images are the simplest to test with openQA because they don't require any local images or advanced networking. For example:

podman exec -it openqa-dispatcher /bin/bash -c "source /venv/bin/activate;
fedora-openqa compose -f https://kojipkgs.fedoraproject.org/compose/cloud/Fedora-Cloud-40-20240610.0/compose"

A successfully scheduled build will appear on the localhost homepage.

Another way to see the scheduled jobs is to query the openqa database directly. For example:

podman exec -it openqa-database /bin/bash -c "psql -U postgres -d openqa";
openqa=# \dt;
openqa=# \d jobs;
openqa=# SELECT build FROM jobs;

Yet another way to see the scheduled jobs is to use openQA's REST API. For example:

podman exec -it openqa-webserver /bin/bash -c "openqa-cli api --host http://$(hostname -I | awk '{print $1}')  --json jobs | jq -r '.jobs[]'"

Next, start some workers locally to run the tests. This example will start 16 worker containers along with a corresponding systemd service for each worker container. Make sure that the Demo user is still logged in because the workers will also use the Demo user's api key/secret. If the Demo user isn't logged in, the workers will shut themselves down.

ansible-playbook \
-e worker_ip_private=$(hostname -I | awk '{print $1}')  \
-e use_http=true \
-e dir=$HOME/my_cloud \
--ask-become-pass \
playbooks/hosts/openqa-cloud-worker-container.yml

See a list of all the containerized workers:

sudo systemctl list-units --all 'openqa-worker@*.service';

In addition to the workers, there will be a new systemd service running on the host machine:
- openqa-createhdds: a service that runs createhdds to build images that can't be fetched remotely.

If you have limited disk space or are not interested in waiting several hours for the openqa-createhdds service to finish generating local images, then just disable

Click through the web user interface to monitor the test progress. Initially, some of the tests will fail while waiting for the openqa-createhdds service to finish generating local images. This process can take several hours. Also, tests that require tap devices, won't run in containers; use the playbook openqa-cloud-worker.yml instead for tap tests.

Stop the web server and components manually. Make sure to stop the database last or it won't exit gracefully.

for service in openqa-test-update.timer openqa-test-update openqa-createhdds.timer openqa-createhdds openqa-reverse-proxy openqa-dispatcher openqa-webserver openqa-database; do
    sudo systemctl stop $service && sudo systemctl disable $service;
done

To stop containerized workers:

sudo systemctl stop openqa-worker@{0..15} && sudo systemctl disable openqa-worker@{0..15}

2. Example: a Production Deployment

Run the web server containers with a public ip using OpenID authentication and SSL/TLS certificates:

ansible-playbook \
-e web_ip_private=$(hostname -I | awk '{print $1}') \
-e web_ip_public=openqa.fedorainfracloud.org \
-e apikey=<> \
-e apisecret=<> \
-e dir=$HOME/cloud_server \
--ask-become-pass \
playbooks/hosts/openqa-cloud-server.yml

Run workers in containers with a public web server.

ansible-playbook \
-e worker_ip_private=$(hostname -I | awk '{print $1}')  \
-e web_ip_public=openqa.fedorainfracloud.org \
-e worker_ip_public=$(curl ipinfo.io/ip) \
-e apikey=<> \
-e apisecret=<> \
-e dir=$HOME/my_cloud_worker \
--ask-become-pass \
playbooks/hosts/openqa-cloud-worker-container.yml

Run tap workers without containers. Workers of any class can be run without containers, but tap workers must be run outside of containers so that the QEMU machines can make calls to the D-Bus to dynamically add/remove tags to tap devices to allow for VLAN segregation. Run all workers belonging to the same tap class on the same machine because the virtual LANs created using OpenvSwitch will not work across different machines.

This command will start 32 workers of class 'qemu_x86_64,tap'.

ansible-playbook \
-e worker_ip_private=$(hostname -I | awk '{print $1}')  \
-e web_ip_public=openqa.fedorainfracloud.org \
-e worker_ip_public=$(curl ipinfo.io/ip) \
-e apikey=<> \
-e apisecret=<> \
-e eth=$(ls -1 /sys/class/net | grep -E '^(en|eth)[0-9]*') \
-e worker_class=qemu_x86_64,tap \
-e number_of_workers=31 \
--ask-become-pass \
playbooks/hosts/openqa-cloud-worker.yml

On another machine start 32 workers of class 'qemu_x86_64,tap2'.

ansible-playbook \
-e worker_ip_private=$(hostname -I | awk '{print $1}')  \
-e web_ip_public=openqa.fedorainfracloud.org \
-e worker_ip_public=$(curl ipinfo.io/ip) \
-e apikey=<> \
-e apisecret=<> \
-e eth=$(ls -1 /sys/class/net | grep -E '^(en|eth)[0-9]*') \
-e worker_class=qemu_x86_64,tap2 \
-e number_of_workers=31 \
--ask-become-pass \
playbooks/hosts/openqa-cloud-worker.yml

List the workers running outside of containers:

sudo systemctl list-units --all 'openqa-worker-plain@*.service';

Stop the workers running outside of containers:

sudo systemctl stop openqa-worker-plain@{0..31}

Optional variables with their defaults: