| |
@@ -0,0 +1,128 @@
|
| |
+ :toc:
|
| |
+
|
| |
+ = Onboarding of a CI System =
|
| |
+
|
| |
+ == Testing in Fedora ==
|
| |
+
|
| |
+ It's relatively easy to start testing Fedora artifacts (Koji builds, Bodhi updates, etc.) and to contribute test results back so that they can be later used for gating -- i.e. deciding whether the tested artifact should be promoted or not.
|
| |
+
|
| |
+ Here we describe necessary steps to add a new CI system.
|
| |
+
|
| |
+
|
| |
+ == What is a (suitable) CI System? ==
|
| |
+
|
| |
+ In order to provide a good workflow and user experience, here are some aspects of CI systems that have proven to be successful:
|
| |
+
|
| |
+ * Tests are reliable (low false negative and false positive rates) and cover important stories
|
| |
+ * Tests can be contributed to (open source model) and are ideally similar to other tests, e.g. by using established frameworks / languages
|
| |
+ * Results / notifications are easy to understand and help identify errors quickly
|
| |
+ * Tests are reproducible, if necessary artifacts from test runs are stored for users to consume, such as virtual machine images
|
| |
+
|
| |
+ In short, test results should be directly actionable! As a developer, I need to quickly decide whether the test is broken or the code, and then fix the issue.
|
| |
+
|
| |
+ == Testing and Gating Builds ==
|
| |
+
|
| |
+ === Gating Workflow ===
|
| |
+
|
| |
+ On the highest level, the gating workflow consists of the following steps:
|
| |
+
|
| |
+ * Submit a build of a package (Koji)
|
| |
+ * Trigger CI systems to run tests (Fedora CI, Your CI, etc.)
|
| |
+ * Collect results from the CI systems (ResultsDB)
|
| |
+ * Make a decision (Greenwave)
|
| |
+ * If the decision is "pass", then let the build pass the gate to the main repository (Bodhi)
|
| |
+
|
| |
+ === Gating Messages ===
|
| |
+
|
| |
+ Gating process is implemented as a set of services which interact with each other via message bus (FedoraMessaging).
|
| |
+
|
| |
+ Therefore to add a piece (for example a CI system) to the process, you essentially need to start receiving and sending messages via FedoraMessaging.
|
| |
+
|
| |
+ TIP: Find out more about https://docs.fedoraproject.org/en-US/rawhide-gating/[gating in Fedora Rawhide] -- especially about the https://docs.fedoraproject.org/en-US/rawhide-gating/single-builds/#_how_does_gating_single_build_updates_work[single] and https://docs.fedoraproject.org/en-US/rawhide-gating/multi-builds/[multi-build] updates.
|
| |
+
|
| |
+
|
| |
+ == How to add a CI System ==
|
| |
+
|
| |
+ CI systems in Fedora are autonomous entities that typically need to handle following things:
|
| |
+
|
| |
+ * triggering when certain events occur
|
| |
+ * actual testing
|
| |
+ * publishing test results
|
| |
+
|
| |
+
|
| |
+ === Triggering and Testing ===
|
| |
+
|
| |
+ Services in Fedora publish messages when various events occur and thus CI systems can trigger testing when for example new Bodhi update is created.
|
| |
+
|
| |
+ When Bodhi update is created, a new "koji-build-group.build.complete" message is published on the https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.prod.bodhi.update.status.testing.koji-build-group.build.complete[org.fedoraproject.prod.bodhi.update.status.testing.koji-build-group.build.complete] topic.
|
| |
+
|
| |
+ The schema of these "koji-build-group.build.complete" messages is defined in the https://pagure.io/fedora-ci/messages[CI Messages specification].
|
| |
+
|
| |
+ If your CI system is Jenkins, then you can use the https://plugins.jenkins.io/jms-messaging/[jms-messaging plugin] to trigger your tests when defined events occur. Getting the trigger syntax right in Jenkins pipelines can be tricky, but you can take a look at the existing example https://github.com/fedora-ci/rpmdeplint-trigger/blob/b078ae3f7134dbc5a155aa435362cd3a241ab99a/Jenkinsfile#L13-L29[here].
|
| |
+
|
| |
+ It's of course possible to trigger testing on other types of events, not just the Bodhi updates. You can find more Fedora message topics in the older https://fedmsg2.readthedocs.io/en/latest/topics.html[fedmsg2 documentation]. Beware though, the list is incomplete.
|
| |
+
|
| |
+
|
| |
+ === Sharing Test Results ===
|
| |
+
|
| |
+ CI systems should publish standardized CI messages so the progress of the testing can be observed and the results can be acted upon by other services in the Fedora infrastructure.
|
| |
+ Although publishing test results is not mandatory, CI systems which don't publish the standardized CI messages cannot be part of the gating process.
|
| |
+
|
| |
+ There are four types of messages that CI systems should be sending:
|
| |
+
|
| |
+ * test.queued - when there is an artifact (a Bodhi update for example) in a queue waiting to be tested
|
| |
+ * test.running - when testing is in progress
|
| |
+ * test.complete - when testing is finished
|
| |
+ * test.error - when testing couldn't start or couldn't finish due to an outside circumstances (typically an infrastructure error)
|
| |
+
|
| |
+ These messages have well-defined schemas. The schemas are part of https://pagure.io/fedora-ci/messages[the CI Messages specification].
|
| |
+
|
| |
+
|
| |
+ For convenience, here are links to schemas for simple koji-build artifacts and koji-build-group artifacts:
|
| |
+
|
| |
+ * koji-build
|
| |
+ ** https://pagure.io/fedora-ci/messages/blob/master/f/schemas/koji-build.test.queued.yaml[test.queued message schema] (https://pagure.io/fedora-ci/messages/blob/master/f/examples/koji-build.test.queued.json[example])
|
| |
+ ** https://pagure.io/fedora-ci/messages/blob/master/f/schemas/koji-build.test.running.yaml[test.running message schema] (https://pagure.io/fedora-ci/messages/blob/master/f/examples/koji-build.test.running.json[example])
|
| |
+ ** https://pagure.io/fedora-ci/messages/blob/master/f/schemas/koji-build.test.complete.yaml[test.complete message schema] (https://pagure.io/fedora-ci/messages/blob/master/f/examples/koji-build.test.complete.json[example])
|
| |
+ ** https://pagure.io/fedora-ci/messages/blob/master/f/schemas/koji-build.test.error.yaml[test.error message schema] (https://pagure.io/fedora-ci/messages/blob/master/f/examples/koji-build.test.error.json[example])
|
| |
+ * koji-build-group
|
| |
+ ** https://pagure.io/fedora-ci/messages/blob/master/f/schemas/koji-build-group.test.queued.yaml[test.queued message schema] (https://pagure.io/fedora-ci/messages/blob/master/f/examples/koji-build-group.test.queued.json[example])
|
| |
+ ** https://pagure.io/fedora-ci/messages/blob/master/f/schemas/koji-build-group.test.running.yaml[test.running message schema] (https://pagure.io/fedora-ci/messages/blob/master/f/examples/koji-build-group.test.running.json[example])
|
| |
+ ** https://pagure.io/fedora-ci/messages/blob/master/f/schemas/koji-build-group.test.complete.yaml[test.complete message schema] (https://pagure.io/fedora-ci/messages/blob/master/f/examples/koji-build-group.test.complete.json[example])
|
| |
+ ** https://pagure.io/fedora-ci/messages/blob/master/f/schemas/koji-build-group.test.error.yaml[test.error message schema] (https://pagure.io/fedora-ci/messages/blob/master/f/examples/koji-build-group.test.error.json[example])
|
| |
+
|
| |
+
|
| |
+ ==== Test Identifiers ====
|
| |
+
|
| |
+ When you send a "koji-build.test.complete" message to the message bus, the test result gets stored in the ResultsDB. Later on you can refer to the stored test result in a Greenwave policy: "if test XYZ passed, let the build through the gate".
|
| |
+
|
| |
+ Therefore you need unique identifiers for test results.
|
| |
+
|
| |
+ Test identifiers are built from three parts: test namespace, test category and test type.
|
| |
+
|
| |
+ In the "koji-build.test.complete" schema, these variables are represented by namespace, category and type fields respectively.
|
| |
+
|
| |
+ Namespace is always an ID of your CI system with the name of the artifact type, so for example: fedora-ci.koji-build, or osci.pull-request.
|
| |
+
|
| |
+ Category you can choose from a predefined list:
|
| |
+
|
| |
+ * static-analysis
|
| |
+ * functional
|
| |
+ * integration
|
| |
+ * validation
|
| |
+
|
| |
+ Type is an arbitrary string, which you can define based on the specifics of your CI system.
|
| |
+
|
| |
+ It is your responsibility as an owner of the CI system to maintain consistent naming of tests under your namespace.
|
| |
+
|
| |
+ The final test identifier may look like this: fedora-ci.koji-build.tier0.functional
|
| |
+
|
| |
+
|
| |
+ == Useful Links ==
|
| |
+
|
| |
+ * https://docs.fedoraproject.org/en-US/rawhide-gating/[Fedora Rawhide Gating]
|
| |
+ * https://docs.fedoraproject.org/en-US/rawhide-gating/optin/[How to Opt in to Gating?]
|
| |
+ * https://pagure.io/greenwave[Greenwave] - service to evaluate gating policies based on test results
|
| |
+ * https://pagure.io/taskotron/resultsdb[ResultsDB] - results store engine
|
| |
+ * https://pagure.io/waiverdb[WaiverDB] - service for recording waivers against test results
|
| |
+ * Greenwave's https://docs.pagure.org/greenwave/package-specific-policies.html[Package-specific policies]
|
| |
psss commented 3 years ago
Perhaps reference top of the single build page as it is done for multi build?
https://docs.fedoraproject.org/en-US/rawhide-gating/single-builds/