#235 Sunset the rcm-tools-jenkins waiverdb job and replace it with C3I pipeline jobs?
Closed: Fixed 5 years ago Opened 5 years ago by rayson.

Hi all,

It's time to discuss the sunset of rcm-tools-jenkins waiverdb job and the roadmap to replace it with the new C3I pipeline jobs. To keep the information open, I decided to move the discussion here.

Once we reach to an agreement on this roadmap, I will convert it to JIRA cards under FACTORY-3097.


Following is my draft plan:

  1. Create and configure all WaiverDB pipeline jobs on UpShift's waiverdb-test project. Polling triggers are temporarily deactivated to avoid competition with the current rcm-tools-jenkins waiverdb job.

  2. Because images built from the pipeline jobs will not have an internal version with Red Hat internal CA, we should make some changes to the internal WaiverDB OpenShift manifests to make sure the public version of image will continue to work on the Red Hat internal infrastructure.

  3. Wait for Pagure polling triggers to be merged and Pagure PR status updater is ported to pipeline jobs.

  4. Deactivate the polling triggers of rcm-tools-jenkins waiverdb job. Activate the pipeline job triggers.

  5. Archive the rcm-tools-jenkins waiverdb job and delete its Jenkinsfile from SCM.

Polling triggers are temporarily deactivated to avoid competition with the current rcm-tools-jenkins waiverdb job.

I think you can set different REFERENCE_JOB_NAME so that there can be multiple test jobs for a PR.

This means the Jenkins job can keep running for a while longer in parallel to the new pipeline.

@lholecek I see. Another problem is that both jobs will try to push images with the latest tag. Should I remove the pushing step fromthe rcm-tools-jenkins job?

@rayson what do you think a good timeline for when we should do this work is? I think step number 2 in your plan above needs to happen first, right?

@acorvin This is a general process and some steps can be done in parallel. I'll investigate the step 2 soon to make sure we can safely change to the public image.

@rayson Ok sounds good. Feel free to create any new cards internally in JIRA to track the work we need to do.

Status Update
Done:
1. The Dockerfile has been changed to use CA_URL env var for downloading CA certificate at runtime.
2. Updated OpenShift manifests have been applied to Red Hat internal WaiverDB dev/stage/prod instances.
Next:
1. Switch to the new images for :latest tag and watch for status of the dev environment.
2. Deactivate the push stage in rcm-tools waiverdb Jenkins jobs.

Update: I've added a new deploy key to this Pagure repo for publishing docs with C3I pipelines. The private key can be found from pagure-doc-secret secret in UpShift waiverdb-test project.

The new deploy key doesn't seem to have the push permission to Pagure even if I checked the Push access box in the project settings, not sure if it is a problem of the recent Pagure upgrade.

@lholecek @csomh Any ideas?

Well, finally I've exported the SSH private key from rcm-tools Jenkins and use that key instead.

It seems OpenShift build has issues to fetch a PR from Pagure. Currently building PRs is not working correctly. I'll try to find a way to fix it after my local QECamp.

@rayson We know there was an outage affecting this yesterday. Are you still seeing this same problem?

@acorvin Thanks for your information. The current status is building master branch works fine, but building a PR before it getting merged always fails. I will take a look if it is caused by the outage tomorrow.

Building PRs should be fixed by https://pagure.io/waiverdb/pull-request/244.
Now we are watching the pipeline builds to see if they are stable enough.
Next, we will disable doc and container push steps in rcm-tools jenkins (https://pagure.io/waiverdb/pull-request/243)

Jenkins jobs now can't clone or push to doc repo of WaiverDB. It seems to be a problem of Pagure, which refuses deploy keys for accessing doc repos.
An issue was created https://pagure.io/pagure/issue/3919 to address this problem, not sure when to recover.

Now we're running both c3i pipelines and rcm-tools jobs for waiverdb in parallel, but only c3i pipelines push docs and containers on master changes.

Closing as resolved.

Metadata Update from @rayson:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

5 years ago

Login to comment on this ticket.

Metadata