From ae4d30315c6ded98305ace0d66e833659497f442 Mon Sep 17 00:00:00 2001 From: Michal Srb Date: Aug 08 2020 16:57:11 +0000 Subject: Add CI onboarding section --- diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 5ed1fc0..00e8b1d 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -1,6 +1,7 @@ * Guides ** xref:how-to-add-dist-git-test.adoc[How to add simple dist-git test] ** xref:quick-start-guide.adoc[Quick Start Guide] +** xref:onboarding-of-a-ci-system.adoc[Onboarding of a CI System] ** xref:pull-requests.adoc[Pull Requests] ** xref:share-test-code.adoc[Share Test Code] ** xref:examples.adoc[Examples] diff --git a/modules/ROOT/pages/onboarding-of-a-ci-system.adoc b/modules/ROOT/pages/onboarding-of-a-ci-system.adoc new file mode 100644 index 0000000..655f608 --- /dev/null +++ b/modules/ROOT/pages/onboarding-of-a-ci-system.adoc @@ -0,0 +1,128 @@ +:toc: + += Onboarding of a CI System = + +== Testing in Fedora == + +It's relatively easy to start testing Fedora artifacts (Koji builds, Bodhi updates, etc.) and to contribute test results back so that they can be later used for gating -- i.e. deciding whether the tested artifact should be promoted or not. + +Here we describe necessary steps to add a new CI system. + + +== What is a (suitable) CI System? == + +In order to provide a good workflow and user experience, here are some aspects of CI systems that have proven to be successful: + +* Tests are reliable (low false negative and false positive rates) and cover important stories +* Tests can be contributed to (open source model) and are ideally similar to other tests, e.g. by using established frameworks / languages +* Results / notifications are easy to understand and help identify errors quickly +* Tests are reproducible, if necessary artifacts from test runs are stored for users to consume, such as virtual machine images + +In short, test results should be directly actionable! As a developer, I need to quickly decide whether the test is broken or the code, and then fix the issue. + +== Testing and Gating Builds == + +=== Gating Workflow === + +On the highest level, the gating workflow consists of the following steps: + +* Submit a build of a package (Koji) +* Trigger CI systems to run tests (Fedora CI, Your CI, etc.) +* Collect results from the CI systems (ResultsDB) +* Make a decision (Greenwave) +* If the decision is "pass", then let the build pass the gate to the main repository (Bodhi) + +=== Gating Messages === + +Gating process is implemented as a set of services which interact with each other via message bus (FedoraMessaging). + +Therefore to add a piece (for example a CI system) to the process, you essentially need to start receiving and sending messages via FedoraMessaging. + +TIP: Find out more about https://docs.fedoraproject.org/en-US/rawhide-gating/[gating in Fedora Rawhide] -- especially about the https://docs.fedoraproject.org/en-US/rawhide-gating/single-builds/#_how_does_gating_single_build_updates_work[single] and https://docs.fedoraproject.org/en-US/rawhide-gating/multi-builds/[multi-build] updates. + + +== How to add a CI System == + +CI systems in Fedora are autonomous entities that typically need to handle following things: + +* triggering when certain events occur +* actual testing +* publishing test results + + +=== Triggering and Testing === + +Services in Fedora publish messages when various events occur and thus CI systems can trigger testing when for example new Bodhi update is created. + +When Bodhi update is created, a new "koji-build-group.build.complete" message is published on the https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.prod.bodhi.update.status.testing.koji-build-group.build.complete[org.fedoraproject.prod.bodhi.update.status.testing.koji-build-group.build.complete] topic. + +The schema of these "koji-build-group.build.complete" messages is defined in the https://pagure.io/fedora-ci/messages[CI Messages specification]. + +If your CI system is Jenkins, then you can use the https://plugins.jenkins.io/jms-messaging/[jms-messaging plugin] to trigger your tests when defined events occur. Getting the trigger syntax right in Jenkins pipelines can be tricky, but you can take a look at the existing example https://github.com/fedora-ci/rpmdeplint-trigger/blob/b078ae3f7134dbc5a155aa435362cd3a241ab99a/Jenkinsfile#L13-L29[here]. + +It's of course possible to trigger testing on other types of events, not just the Bodhi updates. You can find more Fedora message topics in the older https://fedmsg2.readthedocs.io/en/latest/topics.html[fedmsg2 documentation]. Beware though, the list is incomplete. + + +=== Sharing Test Results === + +CI systems should publish standardized CI messages so the progress of the testing can be observed and the results can be acted upon by other services in the Fedora infrastructure. +Although publishing test results is not mandatory, CI systems which don't publish the standardized CI messages cannot be part of the gating process. + +There are four types of messages that CI systems should be sending: + +* test.queued - when there is an artifact (a Bodhi update for example) in a queue waiting to be tested +* test.running - when testing is in progress +* test.complete - when testing is finished +* test.error - when testing couldn't start or couldn't finish due to an outside circumstances (typically an infrastructure error) + +These messages have well-defined schemas. The schemas are part of https://pagure.io/fedora-ci/messages[the CI Messages specification]. + + +For convenience, here are links to schemas for simple koji-build artifacts and koji-build-group artifacts: + +* koji-build +** https://pagure.io/fedora-ci/messages/blob/master/f/schemas/koji-build.test.queued.yaml[test.queued message schema] (https://pagure.io/fedora-ci/messages/blob/master/f/examples/koji-build.test.queued.json[example]) +** https://pagure.io/fedora-ci/messages/blob/master/f/schemas/koji-build.test.running.yaml[test.running message schema] (https://pagure.io/fedora-ci/messages/blob/master/f/examples/koji-build.test.running.json[example]) +** https://pagure.io/fedora-ci/messages/blob/master/f/schemas/koji-build.test.complete.yaml[test.complete message schema] (https://pagure.io/fedora-ci/messages/blob/master/f/examples/koji-build.test.complete.json[example]) +** https://pagure.io/fedora-ci/messages/blob/master/f/schemas/koji-build.test.error.yaml[test.error message schema] (https://pagure.io/fedora-ci/messages/blob/master/f/examples/koji-build.test.error.json[example]) +* koji-build-group +** https://pagure.io/fedora-ci/messages/blob/master/f/schemas/koji-build-group.test.queued.yaml[test.queued message schema] (https://pagure.io/fedora-ci/messages/blob/master/f/examples/koji-build-group.test.queued.json[example]) +** https://pagure.io/fedora-ci/messages/blob/master/f/schemas/koji-build-group.test.running.yaml[test.running message schema] (https://pagure.io/fedora-ci/messages/blob/master/f/examples/koji-build-group.test.running.json[example]) +** https://pagure.io/fedora-ci/messages/blob/master/f/schemas/koji-build-group.test.complete.yaml[test.complete message schema] (https://pagure.io/fedora-ci/messages/blob/master/f/examples/koji-build-group.test.complete.json[example]) +** https://pagure.io/fedora-ci/messages/blob/master/f/schemas/koji-build-group.test.error.yaml[test.error message schema] (https://pagure.io/fedora-ci/messages/blob/master/f/examples/koji-build-group.test.error.json[example]) + + +==== Test Identifiers ==== + +When you send a "koji-build.test.complete" message to the message bus, the test result gets stored in the ResultsDB. Later on you can refer to the stored test result in a Greenwave policy: "if test XYZ passed, let the build through the gate". + +Therefore you need unique identifiers for test results. + +Test identifiers are built from three parts: test namespace, test category and test type. + +In the "koji-build.test.complete" schema, these variables are represented by namespace, category and type fields respectively. + +Namespace is always an ID of your CI system with the name of the artifact type, so for example: fedora-ci.koji-build, or osci.pull-request. + +Category you can choose from a predefined list: + +* static-analysis +* functional +* integration +* validation + +Type is an arbitrary string, which you can define based on the specifics of your CI system. + +It is your responsibility as an owner of the CI system to maintain consistent naming of tests under your namespace. + +The final test identifier may look like this: fedora-ci.koji-build.tier0.functional + + +== Useful Links == + +* https://docs.fedoraproject.org/en-US/rawhide-gating/[Fedora Rawhide Gating] +* https://docs.fedoraproject.org/en-US/rawhide-gating/optin/[How to Opt in to Gating?] +* https://pagure.io/greenwave[Greenwave] - service to evaluate gating policies based on test results +* https://pagure.io/taskotron/resultsdb[ResultsDB] - results store engine +* https://pagure.io/waiverdb[WaiverDB] - service for recording waivers against test results +* Greenwave's https://docs.pagure.org/greenwave/package-specific-policies.html[Package-specific policies]