Release Engineering Automation

This is the Fedora Release Engineering repository for Automation Workflow Tooling here.

One of the primary goals of the Fedora Release Engineering Team moving forward into the future is to continue to automate more and more components of the process that actually creates Fedora, and to do so in a way that is maintainable and can be collaborated upon. This repository is meant to be the focal point of that effort.

Release Engineering Automation Workflow

This effort has been put together by the collaboration of many involved in Fedora and the original public write up of this effort and future goals was the Release Engineering Automation Workflow Engine. The pertinant details the original proposal that have lived on are were iterated upon are now in this repository.

Currently the goal is to automate or make self-service options for as much as humanly possible within the Fedora RelEng group using Ansible. The technical implementation of how this is being accomplished can be seen below.

Technical Implementation

Everything will be powered by Ansible as this is a toolchain that both Fedora Infrastucture and Fedora Release Engineering is familiar with and has been using heavily for automation tasks. We are simply aiming to solve a new automation problem space with the same tool and a different set of rules/policy.

Tasks or sets of tasks should be in an "Include Playbook" such that they are not meant to stand on their own but should be included by other Playbooks or an Ansible Role.

Workflow Playbooks should effectively be "glue" that supply necessary variables to make the "Include Playbooks" and Roles useful for the Workflow at hand.

Execution

In the past (or still currently, depending on the specific task and state of migration to the new workflow) Fedora Release Engineering Automation tasks are performed by various scripts run on various machines within the Fedora Infrastructure with no real centralized logging. Some of these are automated by chron jobs and some run by hand by request of various members within the Fedora Community, normally around Fedora Test Days. Finding information about old tasks is not always the easiest of things to do and the delegation of tasks is currently not available. The goal here is to provide a solution that removes those barriers.

Workflows will be executed and potentially orchestrate actions between multiple other systems or tools such as bodhi, pungi, and koji. Fedmsgs will be emitted with information about the start and completion of workflows along with metadata about them.

In the event of a compose, certain fedmsg output will be picked up by taskotron and autocloud to perform various levels of testing.

+--------------+                                         +----------------+
|              |               +------------+            |                |
|  AutoCloud   |<--------------+            +----------->+   Taskotron    |
|              |               |   fedmsg   |            |                |
|              +-------------->|            |<-----------+                |
+--------------+               |            |            +----------------+
                               +----+-------+
                                    | ^
                                    | |
                                    | |
                                    | |
                                    | |
                                    | |
                                    V |
                   +------------------+-----------------+
                   |                                    |
                   |      Release Engineering           +-----------+
                   |      Workflow Automation Engine    |           |
                   |            (loopabull)             |           |
                   |                                    |           |
                   +------------------+-----------------+           |
                         |            |                             |
                         |            |                             |
       +-----------------+            |                             |
       |                              |                             |
       |                              |                             |
       V                              V                             |
+-------------+                   +--------------+                  |
|             |                   |              |                  |
|    bodhi    |                   |              |                  |
|             |                   |    pungi     |                  |
+-------------+                   |              |                  |
                                  |              |                  |
                                  +----------+---+                  |
                                       ^     |                      V
                                       |     |            +---------------+
                                       |     |            |               |
                                       |     +----------->|     koji      |
                                       |                  |               |
                                       +------------------+               |
                                                          +---------------+

Master Plan (read: the future)

The ultimate goal is to wire up this workflow where playbooks execute via events coming across fedmsg and are then fed via `loopabull`_ to Ansible AWX. Since the `loopabull`_ ansible-playbook binary is configurable, we can accomplish this with the AWX CLI util that allows us to perform command and control in way that triggers a job in AWX over the REST API via the command line. The side effect here is that we can have full implementation of the Role Based Access Control as that will be enforced at the point the REST API call occurs, we will have centralized logging, a convenient web console, and the ability to delegate access the logs and execution history graphs of the automated tasks by FAS groups. This way we are protecting the possibility of a failed run exposing secrets because the access to the centralized logs is just as it is today, limited by group, but it's easier to visualize, detect, and inspect failures.

The proposed strategy for taking this general approach and focusing on Fedora Release Engineering, we would want fedmsg messages from Fedora CI and `Taskotron`_ to be picked up by `loopabull`_ to trigger a playbook in Ansible AWX which would then conditionally perform various release tasks. This workflow could extend beyond just that, but the initial intent is to target Fedora Atomic Host and Fedora Containers as those release artifacts are released at a much more rapid cadence than anything else in Fedora.

+--------------+                                         +----------------+
|              |               +------------+            |                |
|  Fedora CI   |<--------------+            +----------->+   Taskotron    |
|              |               |   fedmsg   |            |                |
|              +-------------->|            |<-----------+                |
+--------------+               |            |            +----------------+
                               +----+-------+
                                    | ^
                                    | |
                                    | |
                                    | |
                                    | |
                                    | |
                                    V |
                            +-----------------------+
                            |                       |
                            |                       |
                            |                       |
                            +-----------------------+
                                    |
                                    |
                                    |
                                    V
                   +------------------+-----------------+
                   |                                    |
                   |            Ansible AWX             +-----------+
                   |                                    |           |
                   +------------------+-----------------+           |
                         |            |                             |
                         |            |                             |
       +-----------------+            |                             |
       |                              |                             |
       |                              |                             |
       V                              V                             |
+-------------+                   +--------------+                  |
|             |                   |              |                  |
|    bodhi    |                   |              |                  |
|             |                   |    pungi     |                  |
+-------------+                   |              |                  |
                                  |              |                  |
                                  +----------+---+                  |
                                       ^     |                      V
                                       |     |            +---------------+
                                       |     |            |               |
                                       |     +----------->|     koji      |
                                       |                  |               |
                                       +------------------+               |
                                                          +---------------+

Licensing

Everything is copyrighted by the respective authors. You can use and redistribute the code under the terms of version 3 or later of the GNU Public License as published by the Free Software Foundation.

To make licensing easier, license headers in the source files will be a single line reference to Unique License Identifiers as defined by the Linux Foundation's SPDX project. For example, in a source file the full "GPL v3.0 or later" header text will be replaced by a single line:

SPDX-License-Identifier:    GPL-3.0+

The license terms of all files in the source tree should be defined by such License Identifiers; in no case a file can contain more than one such License Identifier list.

If a SPDX-License-Identifier: line references more than one Unique License Identifier, then this means that the respective file can be used under the terms of either of these licenses, i. e. with

SPDX-License-Identifier:    GPL-3.0+    LGPL-3.1+

you can chose between GPL-2.0+ and LGPL-2.1+ licensing.

We use the SPDX Unique License Identifiers here;