+ # WaiverDB on OpenShift

+ 

+ This directory contains documents and OpenShift manifests for WaiverDB OpenShift deployments

+ and CI/CD automation on top of OpenShift.

+ 

+ ## What do you want?

+ 

+ - Build and deploy a WaiverDB container manually

+     - [Build a WaiverDB container locally](docs/build-waiverdb-container.md)

+     - [Deploy a WaiverDB container to OpenShift manually](docs/deploy-waiverdb-container.md)

+ - Build, test, and deploy a WaiverDB container with WaiverDB C3I Pipeline (using CI/CD approach)

+     - [Build and test a pull-request with WaiverDB C3I pipeline](docs/pipeline-build-and-test-pr.md)

+     - [Build and test a Git branch with WaiverDB C3I pipeline](docs/pipeline-build-and-test-branch.md)

+     - [Test a WaiverDB container with WaiverDB C3I pipeline](docs/pipeline-test-image.md)

+     - [Deploy a WaiverDB container to dev environment](docs/pipeline-deploy-to-dev.md)

+     - [Promote a WaiverDB container to stage or prod environment](docs/pipeline-promote-to-stage-or-prod.md)

+     - [Fix broken WaiverDB C3I pipeline](docs/pipeline-fix-broken-build.md)

+ - Advanced topics about WaiverDB C3I pipeline

+     - [WaiverDB C3I Pipeline Introduction](https://docs.google.com/a/redhat.com/document/d/e/2PACX-1vR11OgAmBRF1_Lprs8hpgchNodMb9sh1QKCO6t61JaGYs7UpBcsBJSQW_G7fqrwuRLBQPDrDbwKUtek/pub): See how WaiverDB C3I Pipeline works.

+     - [Deploy WaiverDB C3I pipeline (not WaiverDB app) to OpenShift](docs/pipeline-deployment.md)

+ # Build a WaiverDB container image locally

+ Note: If you are looking to build and test a WaiverDB container image using CI/CD, go to [Test and build an image using WaiverDB C3I pipeline](pipeline-build-and-test-branch.md).

+ 

+ 1. Make sure your local machine has `podman` (or Docker) installed. If not, go to [Basic Setup and Use of Podman][1].

+ 

+ 2. Go to the root directory of WaiverDB repo, build an image. Replace `<image>` with the image registry/repo:tag you want to use, such as `quay.io/yuxzhu/waiverdb:test`.

+ 

+     ```

+     podman build -t <image> -f ./openshift/containers/waiverdb/Dockerfile .

+     ```

+ 

+ 3. Push the image to the registry. (Suppose you have the access the registry repository)

+ 

+     ```

+     podman push <image>

+     ```

+ 

+ [1]: https://github.com/containers/libpod/blob/master/docs/tutorials/podman_tutorial.md 

+ # Deploy WaiverDB to OpenShift manually

+ NOTE: This instruction should only be used to deploy a test environment of WaiverDB. Production deployment shouldn't use the OpenShift manifests in this document.

+ 

+ 1. To create an environment from the template, process the template file `openshift/waiverdb-test-template.yaml` and apply it:

+ 

+     ```

+     oc process -f openshift/waiverdb-test-template.yaml -p TEST_ID=<test-id> -p WAIVERDB_APP_IMAGE=quay.io/factory2/waiverdb:0.1.2.dev24-git.94c0119 | oc apply -f -

+     ```

+ 

+     where `<test-id>` should be a unique string to identify your deployed environment, and `WAIVERDB_APP_IMAGE` should be the WaiverDB image you want to run.

+ 

+ 2. To clean up the environment, use a selector on the environment label:

+ 

+     ```

+     oc delete dc,deploy,pod,configmap,secret,svc,route -l environment=test-<test-id>

+     ```

+ 

+     where `<test-id>` should equal to the string given in step 1.

+ # Build and test a Git branch using WaiverDB C3I pipeline

+ - Note: 

+     1. You should have the `admin` or `edit` permission to `waiverdb-test` project on Red Hat internal OpenShift cloud (PSI UpShift).

+     2. DO NOT push test images to `latest`, `stage`, or `prod` tag, otherwise they would be deployed to real `dev`, `stage`, or `prod` environments!

+ 

+ 1. Log into UpShift, switch to `waiverdb-test` project:

+ 

+     ```

+     oc project waiverdb-test

+     ```

+ 

+ 2. Start a new build of `waiverdb-postmerge` pipeline job, specifying the WaiverDB Git repository and branch that you want to build from and the tag to be applied:

+ 

+     ```

+     oc start-build waiverdb-postmerge [ -e WAIVERDB_GIT_REPO=<git-repository-url> ] [ -e WAIVERDB_GIT_REF=<git-branch-name> ] [ -e FORCE_PUBLISH_IMAGE=true ] [ -e WAIVERDB_DEV_IMAGE_TAG=<tag-name> ]

+     ```

+     

+     For example, to rebuild the latest `master` branch, run

+ 

+     ```

+     oc start-build waiverdb-postmerge

+     ```

+ 

+     To build the `feature-foo` branch and push it to `quay.io/factory2/waiverdb:feature-foo`, run

+ 

+     ```

+     oc start-build waiverdb-postmerge -e WAIVERDB_GIT_REF=feature-foo -e FORCE_PUBLISH_IMAGE=true -e WAIVERDB_DEV_IMAGE_TAG=feature-foo

+     ```

+ 

+     For more information about job parameters, see  [waiverdb-build-template.yaml](../pipelines/templates/waiverdb-build-template.yaml).

+ 

+ 3. Visualize the build process at https://paas.upshift.redhat.com/console/project/waiverdb-test/browse/pipelines/waiverdb-postmerge.

+ 4. Your newly built image will be available at `quay.io/factory2/waiverdb:<tag-name>` if `WAIVERDB_GIT_REF` is `master` or `FORCE_PUBLISH_IMAGE` is `true`. Otherwise the newly built image will be thrown away. 

+ # Build and test a pull-request using WaiverDB C3I pipeline

+ 

+ If you are looking to rebuild a pull-request, the simplest way is to update your pull-request. WaiverDB C3I pipeline should be aware of your pull-request change and start a new pipeline build automatically.

+ This document is for advanced users to manually trigger a `premerge` pipeline build.

+ 

+ - Note: 

+     1. You should have the `admin` or `edit` permission to `waiverdb-test` project on Red Hat internal OpenShift cloud (PSI UpShift).

+     2. DO NOT push test images to `latest`, `stage`, or `prod` tag, otherwise the change will go to the real `dev`, `stage`, or `prod` environments!

+ 

+ 1. Log into UpShift, switch to `waiverdb-test` project:

+ 

+     ```

+     oc project waiverdb-test

+     ```

+ 

+ 2. Start a new build of `waiverdb-premerge` pipeline job with WaiverDB Git repository and pull-request branch (in the form of `pull/<number>/head`) that you want to build from and the tag to be applied:

+ 

+     ```

+     oc start-build waiverdb-premerge [ -e WAIVERDB_GIT_REPO=<git-repository-url> ] -e WAIVERDB_GIT_REF=<git-branch-name> [ [ -e FORCE_PUBLISH_IMAGE=true -e WAIVERDB_DEV_IMAGE_TAG=<tag-name> ]

+     ```

+ 

+     For example, to build and test pull-request #123, run

+ 

+     ```

+     oc start-build waiverdb-premerge -e WAIVERDB_GIT_REF=pull/123/head

+     ```

+ 

+     If you don't specify `FORCE_PUBLISH_IMAGE`, the newly built image will be thrown away after build.

+     `WAIVERDB_DEV_IMAGE_TAG` defaults to `latest`. Change the tag name to avoid accidentally overwriting the `latest` image.

+ 

+     For more information about job parameters, see  [waiverdb-build-template.yaml](../pipelines/templates/waiverdb-build-template.yaml).

+ 

+ 3. Visualize the build process at https://paas.upshift.redhat.com/console/project/waiverdb-test/browse/pipelines/waiverdb-premerge.

+ 

+ 4. Your newly built image will be available at `quay.io/factory2/waiverdb:<tag-name>` if `FORCE_PUBLISH_IMAGE` and `WAIVERDB_DEV_IMAGE_TAG` is specified.

+ # Deploy a WaiverDB Container Image to Dev Environment with WaiverDB C3I Pipeline

+ 

+ ## Steps

+ 

+ 1. [Run waiverdb-postmerge pipeline job](pipeline-build-and-test-branch.md) with `-e -e FORCE_PUBLISH_IMAGE=true` and `-e WAIVERDB_DEV_IMAGE_TAG=latest`.

+ 

+ 2. Verify the image change by logging into your container registry (quay.io). 

+ # OpenShift Manifests for CI/CD Pipelines and Deployment

+ 

+ This directory includes OpenShift manifests and Dockerfiles that can be used

+ to deploy a CI/CD pipeline and a test environment for WaiverDB.

+ 

+ ## WaiverDB Pipeline

+ 

+ WaiverDB Pipeline gives you control over building, testing, deploying, and promoting WaiverDB on OpenShift.

+ 

+ It uses [OpenShift Pipeline][], a technology powered by OpenShift, using a combination of the [Jenkins Pipeline Build Strategy][], [Jenkinsfiles][], and the OpenShift Domain Specific Language (DSL) powered by the [OpenShift Jenkins Pipeline (DSL) Plugin][] (previously called the OpenShift Jenkins Client Plugin).

+ 

+ When starting a new pipeline build, the Jenkins slave and the whole testing environment will be dynamically created as OpenShift pods. And after the build is complete, all of them will be destroyed. Concurrent builds are also allowed (but currently disabled) to increase the overall throughput.

+ 

+ Before using the Pipeline, please ensure your Jenkins master has the following plugins installed and configured:

+ - [Openshift Sync plugin][] [1]

+ - [Openshift Client plugin][OpenShift Jenkins Pipeline (DSL) Plugin] [1]

+ - [Kubernetes plugin][] [1]

+ - [Timestamper plugin][]

+ - [SSH Agent plugin][]

+ - [Email Extension plugin][]

+ - [Ownership plugin][]

+ - [JMS Messaging plugin][]

+ - [HTTP Request plugin][]

+ 

+ Notes:

+ [1]: Those plugins are preinstalled if you are using the Jenkins master shipped with OpenShift.

+ 

+ ### Batch Updating All Pipeline Jobs

+ Running `make install` inside `openshift/pipelines` will perform batch updates on all jobs

+ with parameters defined in `openshift/pipelines/jobs`.

+ Logging into your OpenShift project is required before using the Makefile.

+ Secrets and Images in your OpenShift project are *NOT* touched.

+ 

+ All pipeline job definitions are in `./pipelines/jobs` while templates are in `./pipelines/templates`.

+ For each pipeline job, there are 2 files in `openshift/pipelines/jobs`:

+ `<job-name>.env` and `<job-name>.tmpl`. `<job-name>.env` gives all the parameters

+ passing to the OpenShift template specified in `<job-name>.tmpl`.

+ You can also specify an alternate directories for looking up job definitions and templates

+ by using `JOBS_DIR` and `TEMPLATES_DIR` Makefile variables.

+ For example, if you want to set up a stage version of pipeline jobs

+ with experimental modifications to job definitions,

+ you can run: `make install JOBS_DIR=stage-jobs`. For more information, run `make help`.

+ 

+ ### Dev Pipeline

+ Dev Pipeline is a part of the WaiverDB Pipeline, covers the following steps:

+ 

+ - Run Flake8 and Pylint checks

+ - Run unit tests

+ - Build Docs

+ - Publish Docs

+ - Build SRPM

+ - Build RPM

+ - Invoke Rpmlint

+ - Build container

+ - Run functional tests

+ - Push container

+ 

+ Whenever a new Git commit comes in, the dev pipeline is run against the commit to ensure that this commit can be merged into mainline, and a container is built and pushed to the specified registry.

+ 

+ #### Installation

+ To install the pipeline to a project on OpenShift, just login to your

+ OpenShift cloud, switch to a project (I would suggest using a separated

+ project instead of sharing the real dev/stage/prod environment):

+ 

+ ```bash

+   oc login <your OpenShift cloud>

+   oc project waiverdb-test # your project name

+ ```

+ 

+ Then instantiate the template with default parameters

+ (this will set up a pipeline to build the upstream repo and publish the result to upstream registries):

+ 

+ ```bash

+   oc process --local -f pipelines/tempates/waiverdb-dev-template.yaml | oc apply -f -

+ ```

+ 

+ Or you want to set up a pipeline for your own fork:

+ 

+ ```bash

+   NAME=waiverdb-dev

+   oc process --local -f pipelines/tempates/waiverdb-dev-template.yaml \

+    -p NAME="$NAME" \

+    -p WAIVERDB_GIT_REPO="https://pagure.io/forks/<username>/waiverdb.git" \

+    -p WAIVERDB_GIT_REF="<your devel branch>" \

+    -p WAIVERDB_DEV_IMAGE_DESTINATIONS="docker.io/<username>/waiverdb:latest" \

+    -p PAGURE_DOC_REPO_NAME="forks/<username>/waiverdb" \

+  | oc apply -f -

+ ```

+ 

+ #### Configure Secrets for Push Containers

+ This section is optional if publishing containers is not needed.

+ 

+ To publish the container built by the pipeline, you need to set up a secret for your target container registries.

+ 

+ - Please go to your registry dashboard and create a robot account.

+ - Backup your docker-config-json file (`$HOME/.docker/config.json`) if present.

+ - Run `docker login` with the robot account you just created to produce a new docker-config-json file.

+ - Create a new [OpenShift secret for registries][] named `factory2-pipeline-registry-credentials` from your docker-config-json file:

+ ```bash

+   oc create secret generic factory2-pipeline-registry-credentials \

+     --from-file=.dockerconfigjson="$HOME/.docker/config.json" \

+     --type=kubernetes.io/dockerconfigjson

+ ```

+ 

+ #### Configure Secrets for Publishing Pagure Docs

+ This section is optional if publishing Pagure Docs is not needed.

+ 

+ - Please go to your Pagure repository settings and activate the project documentation.

+ - Create an SSH key pair for publishing Pagure Docs and show the public key:

+ ```bash

+ ssh-keygen -f "$HOME/.ssh/id_rsa_pagure"

+ cat "$HOME/.ssh/id_rsa_pagure.pub"

+ ```

+ - Go to your Pagure repository settings, and add the public key as the deploy key.

+ - Add the private key to your OpenShift project by creating a new [OpenShift secret for SSH key authentication][]:

+ ```bash

+   oc create secret generic pagure-doc-secret \

+     --from-file=ssh-privatekey="$HOME/.ssh/id_rsa_pagure" \

+     --type=kubernetes.io/ssh-auth

+   # This label is used by Jenkins OpenShift Sync Plugin for synchonizing OpenShift secrets with Jenkins Credentials.

+   oc label secret pagure-doc-secret credential.sync.jenkins.openshift.io=true

+ ```

+ 

+ #### Configure a Pagure API Key

+ This section is not required if updating Pagure PR status is not needed.

+ 

+ - Go to your Pagure repository settings, and locate to the 'API Keys' section.

+ - Click on the `Create new key` button to add new API key with the `Flag a pull-request` permission.

+ - Add your newly-created API key to OpenShift:

+ ```bash

+   oc create secret generic pagure-api-key --from-literal=secrettext=<your-api-key>

+   oc label secret pagure-api-key credential.sync.jenkins.openshift.io=true

+ ```

+ 

+ #### Build Jenkins slave container image

+ Before running the pipeline, you need to build a container image for Jenkins slave pods.

+ This step should be repeated every time you change

+ the Dockerfile (`openshift/containers/jenkins-slave/Dockerfile`) for the Jenkins slave pods.

+ 

+ ```bash

+  oc start-build waiverdb-premerge-jenkins-slave

+ ```

+ 

+ By default the Dockerfile is retrieved from the master branch of WaiverDB Git repository.

+ To build from an alternative branch, use `--commit` option. For more information, run `oc start-build -h`.

+ 

+ #### Trigger A Pipeline Build

+ To trigger a pipeline build, start the `waiverdb-dev-pipeline` BuildConfig with the Git commit ID or branch that you want to

+ test against:

+ ```bash

+   oc start-build waiverdb-dev -e WAIVERDB_GIT_REF=<branch_name_or_commit_id>

+ ```

+ You can go to the OpenShift Web console to check the details of the pipeline build.

+ 

+ ### Integration Test Pipeline

+ Typically, integration tests should be rerun

+ - when a new container image of WaiverDB is built

+ - when any service that WaiverDB depends on is updated to a new version

+ - to ensure an image is mature enough for promotion

+ 

+ Hence, splitting the functional test stage makes it possible to run integration

+ tests individually. It can also be a step of a larger pipeline.

+ 

+ #### Installation

+ To install this OpenShift pipeline:

+ ```bash

+ oc process --local -f pipelines/templates/waiverdb-integration-test-template.yaml \

+    -p NAME=waiverdb-integration-test \

+   | oc apply -f -

+ ```

+ 

+ Additional installations with default parameters for dev, stage, and prod environments:

+ ```bash

+ # for dev

+ oc process --local -f pipelines/templates/waiverdb-integration-test-template.yaml \

+    -p NAME=waiverdb-dev-integration-test \

+    -p IMAGE="quay.io/factory2/waiverdb:latest" \

+   | oc apply -f -

+ # for stage

+ oc process --local -f pipelines/templates/waiverdb-integration-test-template.yaml \

+    -p NAME=waiverdb-stage-integration-test \

+    -p IMAGE="quay.io/factory2/waiverdb:stage" \

+   | oc apply -f -

+ # for prod

+ oc process --local -f pipelines/templates/waiverdb-integration-test-template.yaml \

+    -p NAME=waiverdb-prod-integration-test \

+    -p IMAGE="quay.io/factory2/waiverdb:prod" \

+   | oc apply -f -

+ ```

+ 

+ #### Usage

+ To trigger a pipeline build for each environment, run:

+ ```bash

+ # for dev

+ oc start-build waiverdb-dev-integration-test

+ # for stage

+ oc start-build waiverdb-stage-integration-test

+ # for prod

+ oc start-build waiverdb-prod-integration-test

+ ```

+ 

+ To trigger a custom integration test, start a new pipeline build with the image reference you want to test against and the Git repository and commit ID/branch name

+ where the functional test suite is used:

+ ```bash

+ oc start-build waiverdb-integration-test \

+   -e IMAGE="quay.io/factory2/waiverdb:test" \

+   -e WAIVERDB_GIT_REPO=https://pagure.io/forks/<username>/waiverdb.git \

+   -e WAIVERDB_GIT_REF=my-branch # master branch is default

+ ```

+ 

+ #### NOTE

+ The stage of reporting test results to ResultsDB has not been implemented.

+ 

+ ### Image Promotion Pipeline

+ Image Promotion Pipeline is a kind of pipeline that promoting an existing image to an upgraded tag.

+ The pipeline pulls the image to be promoted, then pushes it to destinations with promoted tags.

+ Optionally, it tags the promoted image into an image stream after pushes.

+ 

+ NOTE:

+ 1. This pipeline *DOES NOT* cover the step of running tests before actually promoting the image.

+ 2. It's designed to be a callback pipeline triggered by a microservice that handles Greenwave messages.

+ 3. It can be triggered manually (as describe below) to force promoting an image without any tests.

+ 

+ #### Installation

+ An OpenShift Template is provided to produce pipelines for promotions between different environments.

+ You need to instantiate the Template with corresponding parameters for each promotion, like `dev to stage` and `stage to prod`.

+ 

+ Examples:

+ - Installing a pipeline for promoting dev image to stage:

+ ```bash

+ oc process --local -f pipelines/templates/waiverdb-image-promotion-template.yaml \

+     -p NAME=waiverdb-promoting-to-stage \

+     -p IMAGE="quay.io/factory2/waiverdb:latest" \

+     -p PROMOTING_DESTINATIONS="quay.io/factory2/waiverdb:stage,docker-registry.engineering.redhat.com/factory2/waiverdb:stage" \

+   | oc apply -f -

+ ```

+ 

+ - Installing a pipeline for promoting stage image to prod:

+ ```bash

+ oc process --local -f pipelines/templates/waiverdb-image-promotion-template.yaml \

+     -p NAME=waiverdb-promoting-to-prod \

+     -p IMAGE="quay.io/factory2/waiverdb:stage" \

+     -p PROMOTING_DESTINATIONS="quay.io/factory2/waiverdb:prod,docker-registry.engineering.redhat.com/factory2/waiverdb:prod" \

+   | oc apply -f -

+ ```

+ 

+ - Installing a pipeline for promoting dev image to stage with the additional behavior

+ that automatically tags the image into image stream `waiverdb-stage/waiverdb:stage`:

+ ```bash

+ oc process --local -f pipelines/templates/waiverdb-image-promotion-template.yaml \

+     -p NAME=waiverdb-promoting-to-stage \

+     -p IMAGE="quay.io/factory2/waiverdb:latest" \

+     -p PROMOTING_DESTINATIONS="quay.io/factory2/waiverdb:stage,docker-registry.engineering.redhat.com/factory2/waiverdb:stage" \

+     -p TAG_INTO_IMAGESTREAM=true \

+     -p DEST_IMAGESTREAM_NAME=waiverdb \

+     -p DEST_IMAGESTREAM_TAG=stage \

+     -p DEST_IMAGESTREAM_NAMESPACE=waiverdb-stage \

+   | oc apply -f -

+ ```

+ 

+ - Installing a pipeline for promoting stage image to prod with the additional behavior

+ that automatically tags the image into image stream `waiverdb-prod/waiverdb:prod`:

+ ```bash

+ oc process --local -f ./waiverdb-image-promotion-pipeline-template.yml \

+     -p NAME=waiverdb-promoting-to-prod \

+     -p IMAGE="quay.io/factory2/waiverdb:stage" \

+     -p PROMOTING_DESTINATIONS="quay.io/factory2/waiverdb:prod,docker-registry.engineering.redhat.com/factory2/waiverdb:prod" \

+     -p TAG_INTO_IMAGESTREAM=true \

+     -p DEST_IMAGESTREAM_NAME=waiverdb \

+     -p DEST_IMAGESTREAM_TAG=prod \

+     -p DEST_IMAGESTREAM_NAMESPACE=waiverdb-prod \

+   | oc apply -f -

+ ```

+ 

+ #### Manually Trigger A Promotion

+ To trigger a promotion, start the corresponding BuildConfig with `oc start-build $PIPELINE_NAME`:

+ 

+ ```bash

+   # Promoting to stage

+   oc start-build waiverdb-promoting-to-stage

+   # Promoting to prod

+   oc start-build waiverdb-promoting-to-prod

+ ```

+ You can go to the OpenShift Web console for more details of the pipeline build.

+ 

+ [OpenShift Pipeline]: https://docs.okd.io/3.9/dev_guide/openshift_pipeline.html

+ [Jenkins Pipeline Build Strategy]: https://docs.openshift.com/container-platform/3.9/dev_guide/dev_tutorials/openshift_pipeline.html

+ [Jenkinsfiles]: https://jenkins.io/doc/book/pipeline/jenkinsfile/

+ [OpenShift Jenkins Pipeline (DSL) Plugin]: https://github.com/openshift/jenkins-client-plugin

+ [Openshift Sync plugin]: https://github.com/openshift/jenkins-sync-plugin

+ [Kubernetes plugin]: https://github.com/jenkinsci/kubernetes-plugin

+ [Timestamper plugin]: https://github.com/jenkinsci/timestamper-plugin

+ [SSH Agent plugin]: https://github.com/jenkinsci/ssh-agent-plugin

+ [Email Extension plugin]: https://github.com/jenkinsci/email-ext-plugin

+ [Ownership plugin]: https://github.com/jenkinsci/ownership-plugin

+ [JMS Messaging plugin]: https://github.com/jenkinsci/jms-messaging-plugin

+ [OpenShift secret for registries]:https://docs.openshift.com/container-platform/3.9/dev_guide/builds/build_inputs.html#using-docker-credentials-for-private-registries

+ [OpenShift secret for SSH key authentication]: https://docs.openshift.com/container-platform/3.9/dev_guide/builds/build_inputs.html#source-secrets-ssh-key-authentication

+ [HTTP Request plugin]: https://github.com/jenkinsci/http-request-plugin 

+ # Instructions to Fix a Broken Pipeline Build

+ 

+ 1. `waiverdb-premerge` and `waiverdb-postmerge` builds could fail if new package dependencies are introduced. Because we don't root access to the Jenkins agent pod, dependency changes can't be applied automatically. To make changes to the dependencies, you have to rebuild the Jenkins agent image. Follow the `Build Jenkins slave container image` section in [OpenShift Manifests for CI/CD Pipelines and Deployment](pipeline-deployment.md) document.

+ 

+ 2. If you see no updates on the pull-request after build, probably it is because the Pagure API key is expired.

+ Pagure API key is used by the pipeline jobs to interact with pagure.io, however it expires every 3 months.

+ To renew the Pagure API key, just generate a new key and reconfigure it. Refer to `Configure a Pagure API Key` section in OpenShift Manifests for CI/CD Pipelines and Deployment](pipeline-deployment.md) for more information. 

+ # Promoting a WaiverDB image to Stage or Prod with WaiverDB C3I Pipeline

+ 

+ ## Prerequisites

+ 1. The WaiverDB image to be promoted must be available from a container registry (like `quay.io`). If you want to build an image from a Git branch or a Pagure pull-request, see [Build and test a pull-request with WaiverDB C3I pipeline](./pipeline-build-and-test-pr.md) or [Build and test a Git branch with WaiverDB C3I pipeline](./pipeline-build-and-test-branch.md).

+ 

+ ## Steps

+ 

+ 1. Log into UpShift, switch to `waiverdb-test` project:

+ 

+     ```

+     oc project waiverdb-test

+     ```

+ 

+ 2. To promote an image to stage, use pipeline job `waiverdb-promoting-to-stage`. To promote an image to prod, use pipeline job `waiverdb-promoting-to-prod`.

+ 

+ 

+ 3. Trigger your chosen pipeline job:

+ 

+     ```

+     oc start-build <job-name> -e IMAGE=<image-to-promote> [ -e CONTAINER_REGISTRY_CREDENTIALS=<registry-credentials>]

+     ```

+ 

+     For example, to promote `quay.io/factory2/waiverdb:foo` to stage, run:

+ 

+     ```

+     oc start-build waiverdb-promoting-to-stage -e IMAGE=quay.io/factory2/waiverdb:foo

+     ```

+ 

+     If your container registry requires authentication, use the `CONTAINER_REGISTRY_CREDENTIALS` parameter.

+     For more information about parameters of integration test pipeline jobs, see  [waiverdb-image-promotion-template.yaml](../pipelines/templates/waiverdb-image-promotion-template.yaml).

+ 

+ 4. Visualize the promotion process at https://paas.upshift.redhat.com/console/project/waiverdb-test/browse/pipelines/$job-name.

+ 

+ 5. Verify the image change by logging into your container registry (quay.io). 

+ # Test a WaiverDB Container Image with WaiverDB C3I Pipeline

+ 

+ ## Prerequisites

+ 1. The WaiverDB image to be tested must be available from a container registry (like `quay.io`). If you want to build an image from a Git branch or a Pagure pull-request, see [Build and test a pull-request with WaiverDB C3I pipeline](./pipeline-build-and-test-pr.md) or [Build and test a Git branch with WaiverDB C3I pipeline](./pipeline-build-and-test-branch.md).

+ 

+ ## Steps

+ 

+ 1. Log into UpShift, switch to `waiverdb-test` project:

+ 

+     ```

+     oc project waiverdb-test

+     ```

+ 

+ 2. Choose the pipeline job according to type of integration you want to run:

+     - waiverdb-dev-integration-test: test the image in a dev preview environment

+     - waiverdb-stage-integration-test: test the image in a stage preview environment

+     - waiverdb-prod-integration-test: test the image in a prod preview environment

+ 

+ 3. Test the image by triggering a new pipeline build from your chosen pipeline job:

+ 

+     ```

+     oc start-build <job-name> -e IMAGE=<image-to-test> [ -e CONTAINER_REGISTRY_CREDENTIALS=<registry-credentials>]

+     ```

+ 

+     For example, to test `quay.io/factory2/waiverdb:foo` in a stage preview environment. run:

+ 

+     ```

+     oc start-build waiverdb-stage-integration-test -e IMAGE=quay.io/factory2/waiverdb:foo

+     ```

+ 

+     If your container registry requires authentication, use the `CONTAINER_REGISTRY_CREDENTIALS` parameter.

+     For more information about parameters of integration test pipeline jobs, see  [waiverdb-integration-test-template.yaml](../pipelines/templates/waiverdb-integration-test-template.yaml).

+ 

+ 4. Visualize the build process at https://paas.upshift.redhat.com/console/project/waiverdb-test/browse/pipelines/$job-name.

+ 

+ 5. After the test finishes, a UMB message will be sent to `VirtualTopic.eng.ci.container-image.test.complete` topic. The test result will also be available at ResultsDB with the artifact type `container-image`.