CI Messages

This repo specifies details of messages, which should be sent out by Upstream/Downstream CI systems to fedmsg/UMB on events related to artifacts testing. Currently there is low unification of these messages and there are various types of messages sent out by the CI systems. Unification will make it possible to build simpler services which will provide useful features for everyone.

Specification

The message format is described by JSON Schema. In order to allow better readability and review we store the schemas in the YAML format. Use make convert to convert the spec into JSON.

Repository contains also a set of example messages which can be validated against the specification using make test. Use make clean to clean up all temporary files. Run make to convert spec, validate messages and clean up in a single step.

Note: Make sure that python3-jsonschema, python3-anymarkup and python3-rfc3987 are installed on your system when converting specification and validating examples (alternatively, use tox or install packages from PyPI).

Validation

When implementing CI Messages specification in your application it is a good idea to validate generated messages against the schema. Make sure that schemas are converted and run the validate.py script to ensure your message.json file has proper format:

make convert
./scripts/validate.py schemas/brew-build.test.complete.json message.json

Note: Release tag 1.x.x onward all messages will be validated against scheme before updating the ResultsDB, and invalid messges will not be processed.

Examples

Examples should have exactly the same name as the schema against which they should be validated. Use optional suffix starting with the "+" sign to test multiple examples against a single schema, for example like this:

the-schema-name+simple.json
the-schema-name+complex.json

Virtual topic namespace

The messages will use unique UMB virtual topic hierarchies to mitigate collisions with other systems:

UMB: /topic/VirtualTopic.eng.ci

fedmsg:
  org.fedoraproject.prod.ci
  org.centos.prod.ci

For all the tests on all the artifacts the systems will use this topic naming convention:

UMB: /topic/VirtualTopic.eng.ci.<namespace>.<artifact>.<event>.{queued,running,complete,error}
fedmsg: org.fedoraproject.prod.ci.<namespace>.<artifact>.<event>.{queued,running,complete,error}

where namespace is any string that denotes the source of the message, required to be able to use AMQP ACLs to secure publication rights between various publishing teams, for example:

• osci - a message from the "Operating System CI" team's systems.
• cvp - a message from the "Container Verification Pipeline".
• art - a message from the jenkins master belonging to the "Automatic Release Team".

where artifact is a product in the release pipeline, for example:

• brew-build - scratch or non-scratch rpm build from Brew build system
• container-image - generic container image build
• component-version - artifact in Prd Export License Control (PELC)
• fedora-update - Bodhi update artifact
• koji-build - scratch or non-scratch rpm build from Koji build system
• koji-build-group - group of koji builds (sidetag)
• product-build - generic product build artifact
• product-scenario - layered product scenario
• productmd-compose - compose artifact - https://github.com/release-engineering/productmd
• pull-request - dist-git pull request
• redhat-container-image - container image build from OSBS, metadata accessible via Brew build system
• redhat-container-group - group of redhat-container-image artifacts
• redhat-module - Red Hat MBS module build
• uncontrolled-build - build not tied to any specific build system

where event is one of the recognized events, for example:

• build - tracking progress of building artifacts
• test - for testing progression and results
• promote - for promoting events of artifacts, i.e. events representing the progression of the artifact in the release pipeline

Note that an older version of this schema omitted the namespace field. Messages bearing those topics will still appear for some time while teams migrate to the modern topic structure above.

UMB: /topic/VirtualTopic.eng.ci.<artifact>.<event>.{queued,running,complete,error}
fedmsg: org.fedoraproject.prod.ci.<artifact>.<event>.{queued,running,complete,error}

Message Anatomy

All data should be stored in the body of the message. Additional fields can be added to the messages as required by the senders, although it is advised to document them here. Syntax requirements for field names: Use only characters [a-z0-9_] No spaces are allowed in the name of the field * For separation of multi-word strings in field names use the underscore character

Syntax requirements for field values:

• Boolean value must be one of true or false
• If a field contains multiple values a list must be used for it e.g. recipients: ["mvadkert", "optak"]. In case of an empty field, this needs to be and empty list: []

Licensing

The contents of this repo fall under two licenses. Code (e.g. the Makefile and the contents of the scripts directory) is provided under a MIT-style license, included in each file. Non-code content - everything without the MIT-style license, e.g. the schemas - is provided under the Creative Commons Attribution 4.0 International License.

Links

• Fedora Messaging Documentation