#1409 What's the process to update a CentOS CI Tenant / Jenkins?
Closed: Fixed with Explanation a month ago by wombelix. Opened a month ago by wombelix.

Hi,

the jenkins deployment (https://jenkins-pagure.apps.ocp.cloud.ci.centos.org/) in the pagure OCP namespace is a bit dated: Jenkins 2.401.1, latest available 2.440.3.

Also the plugin manager complains about a lot outdated ones that need attention.
What's the process to get it updated? And does that only include Jenkins itself or the Plugins too? Or can / should I manage all Plugin related stuff (updates, install new plugins) through the Jenkins Web UI directly?

I checked out https://docs.infra.centos.org/operations/ci/adding_cico_tenant/adding_cico_tenants/, https://centosci.github.io/ocp4-docs/ and the ansible playbooks in https://github.com/CentOS/ansible-infra-playbooks but who does what when it comes to an update isn't explained :-/

Dom


The jenkins ImageStream in openshift is managed/tagged by Openshift/Red Hat ...
So it's updated whenever they have a new release, and that's how it was updated in the past.

I wanted to search for the link and it's the image : https://catalog.redhat.com/software/containers/openshift4/ose-jenkins/5cdd918ad70cc57c44b2d279

But I see also that they have a new image, which is different, based on information from https://github.com/openshift/jenkins?tab=readme-ov-file#411-and-higher :/
=> https://catalog.redhat.com/software/containers/ocp-tools-4/jenkins-rhel8/5fe1f38288e9c2f788526306

So different name and I guess different ImageStream in openshift now.
IT would need some investigation to see if that's possible to just modify existing deploymentconfig (themselves now deprecated and replaced by deployments) to use that different ImageStream

As you have admin rights in your project that's something that can be tested.
We can eventually sync on a .stg. cluster or in a different namespace to see how that can be done ?
My time is currently very limited to due to plenty of infra work related to c8s/c7 EOL plan and also onboarding c10s in infra (including CI, etc)

Metadata Update from @arrfab:
- Issue tagged with: centos-ci-infra, feature-request, high-gain, high-trouble, investigation

a month ago

The jenkins ImageStream in openshift is managed/tagged by Openshift/Red Hat ...
So it's updated whenever they have a new release, and that's how it was updated in the past.

I wanted to search for the link and it's the image : https://catalog.redhat.com/software/containers/openshift4/ose-jenkins/5cdd918ad70cc57c44b2d279

But I see also that they have a new image, which is different, based on information from https://github.com/openshift/jenkins?tab=readme-ov-file#411-and-higher :/
=> https://catalog.redhat.com/software/containers/ocp-tools-4/jenkins-rhel8/5fe1f38288e9c2f788526306

So different name and I guess different ImageStream in openshift now.

OK that would explain why it's not updated anymore, makes sense.

IT would need some investigation to see if that's possible to just modify existing deploymentconfig (themselves now deprecated and replaced by deployments) to use that different ImageStream

As you have admin rights in your project that's something that can be tested.

I'm totally fine doing that migration and update myself. I just wanted to double check first if there is any process in place. Or if I would interfere with any playbooks or automation you guys have. But as I understand, this isn't the case so I will not disturb anyone from the infra team with whatever I do :)

We can eventually sync on a .stg. cluster or in a different namespace to see how that can be done ?

If you can free up a little time, a sync to stg would be nice to have. But a slight outage wouldn't be much of a problem for the PR Pipeline we have and I can even deploy something side by side in the current namespace.

My time is currently very limited to due to plenty of infra work related to c8s/c7 EOL plan and also onboarding c10s in infra (including CI, etc)

Totally understand that and really appreciate that you took the time to respond to the ticket so fast!

So let's keep that ticket open for now but I'll try to get the move from DeploymentConfig to Deployment and the new ImageStream done myself.

I'll update the ticket with my progress and steps, maybe that helps if there are other similar deployments that need an update.

I did some research and things make much more sense now.
A draft Deployment Manifest is ready.
I have to do some tests and plan the update steps.

Following all my notes if it's needed for later reference or helpful for someone else.

Why was there no auto-update of the jenkins image to a recent version?

TL;DR
tag 2 only receive an updated when ocp is updated. the new tag scheduled-upgrade-redeploy will trigger as soon a new image is published. currently deployed and latest tag 2 image is from 15/06/2023, latest tag scheduled-upgrade-redeploy image is from 29/01/2024. Not bleeding edge but still much more up to date.

Long version:

https://console-openshift-console.apps.ocp.cloud.ci.centos.org/ runs OpenShift version 4.15 (k8s 1.28)
The current ImageStream openshift/jenkins config should be therefore this one: https://github.com/openshift/jenkins/blob/master/openshift/imagestreams/jenkins-rhel.yaml
Tag 2 points to registry.redhat.io/ocp-tools-4/jenkins-rhel8:v4.13.0-1686682222, which is this container image: https://catalog.redhat.com/software/containers/ocp-tools-4/jenkins-rhel8/5fe1f38288e9c2f788526306?image=648adf0d185f308ebdc41cbd&architecture=amd64 (published 15/06/2023)
The SHA256 matches the currently running image if the jenkins DeploymentConfig.
Based on https://access.redhat.com/documentation/en-us/openshift_container_platform/4.15/html-single/jenkins/index#relocation-of-openshift-jenkins-images_important-changes-to-openshift-jenkins-images_important

By default, this image is referenced by the jenkins:2 image stream tag of Jenkins image stream in the openshift namespace in the ImageStream YAML file in the Samples Operator payload.

This this tag (and latest, ocp-upgrade-redeploy) doesn't change before the next OpenShift release and therefore doesn't trigger an update of the jenkins deployment.
To update jenkins independent of OCP, the tag scheduled-upgrade-redeploy has to be used:

To automatically redeploy the latest version of the Jenkins image when it is released, use this image stream tag in your Jenkins deployment configuration. This image stream tag uses the periodic importing of image stream tags feature of the OpenShift Container Platform image stream controller, which checks for changes in the backing image. If the image changes, for example, due to a recent Jenkins security advisory, OpenShift Container Platform triggers a redeployment of your Jenkins deployment configuration.

This tag points to registry.redhat.io/ocp-tools-4/jenkins-rhel8:v4.13.0, current tagged image: https://catalog.redhat.com/software/containers/ocp-tools-4/jenkins-rhel8/5fe1f38288e9c2f788526306?architecture=amd64&image=65b76ad991230cbe794cadc3 (published 29/01/2024)

Migrate from DeploymentConfig > Deployment

TL;DR
capabilities liger trigger are not available but to update an image automatically, an annotation can be used instead.
Adjust kind and some other fields. Clean up and drop generated things like managedFields, not relevant when the manifest is used to create a new deployment.

Long version:

Setup trigger that updates deployment on ImageStream event, annotation:
https://docs.openshift.com/container-platform/4.14/openshift_images/triggering-updates-on-imagestream-changes.html#images-triggering-updates-imagestream-changes-kubernetes-cli_triggering-updates-on-imagestream-changes

Current Trigger:

  triggers:
    - type: ImageChange
      imageChangeParams:
        automatic: true
        containerNames:
          - jenkins
        from:
          kind: ImageStreamTag
          namespace: openshift
          name: 'jenkins:2'
        lastTriggeredImage: 'image-registry.openshift-image-registry.svc:5000/openshift/jenkins@sha256:eab456afb39ed4607b2ee61c8c7635ab1c5ff8f8bddf7640c557e792504d545f'
    - type: ConfigChange

Annotation for Deployment with auto updates to latest jenkins version available:

image.openshift.io/triggers: '[{"from":{"kind":"ImageStreamTag","namespace":"openshift","name":"jenkins:scheduled-upgrade-redeploy"},"fieldPath":"spec.template.spec.containers[?(@.name==\"jenkins\")].image"}]'

Changes that are required in the Manifest file:
- replace apiVersion: apps.openshift.io/v1 with apiVersion: apps/v1
- change kind: DeploymentConfig to kind: Deployment
- change spec.selectors from:

    selector:
      name: ...

to

    selector:
      matchLabels:
        name: ...
  • set spec.template.spec.containers.image to a valid image or use the trigger annotation
  • remove spec.triggers
  • remove spec.test
  • remove this fields from spec.strategy (if they exist)
  • activeDeadlineSeconds
  • resources
  • rollingParams.intervalSeconds
  • rollingParams.timeoutSeconds
  • rollingParams.updatePeriodSeconds
  • rename spec.strategy.rollingParams to spec.strategy.rollingUpdate (if it exist)
  • if spec.strategy.type is set to Rolling replace the value with RollingUpdate

Draft Manifest

That's the current, untested, draft to replace DeploymentConfig with and enforce regular jenkins updates:

kind: Deployment
apiVersion: apps/v1
metadata:
  annotations:
    image.openshift.io/triggers: '[{"from":{"kind":"ImageStreamTag","namespace":"openshift","name":"jenkins:scheduled-upgrade-redeploy"},"fieldPath":"spec.template.spec.containers[?(@.name==\"jenkins\")].image"}]'
  name: jenkins
  namespace: pagure
  labels:
    app: cico-workspace
    template: cico-workspace-template
spec:
  strategy:
    type: Recreate
    recreateParams:
      timeoutSeconds: 1200
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      name: jenkins
  template:
    metadata:
      creationTimestamp: null
      labels:
        name: jenkins
    spec:
      restartPolicy: Always
      serviceAccountName: jenkins
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 30
      securityContext: {}
      containers:
        - resources:
            limits:
              memory: 3Gi
          readinessProbe:
            httpGet:
              path: /login
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 3
            timeoutSeconds: 240
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          terminationMessagePath: /dev/termination-log
          name: jenkins
          livenessProbe:
            httpGet:
              path: /login
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 420
            timeoutSeconds: 240
            periodSeconds: 360
            successThreshold: 1
            failureThreshold: 2
          env:
            - name: OPENSHIFT_ENABLE_OAUTH
              value: 'true'
            - name: OPENSHIFT_ENABLE_REDIRECT_PROMPT
              value: 'true'
            - name: DISABLE_ADMINISTRATIVE_MONITORS
              value: 'True'
            - name: KUBERNETES_MASTER
              value: 'https://kubernetes.default:443'
            - name: KUBERNETES_TRUST_CERTIFICATES
              value: 'true'
            - name: JENKINS_SERVICE_NAME
              value: jenkins
            - name: JNLP_SERVICE_NAME
              value: jenkins-jnlp
          securityContext:
            capabilities: {}
            privileged: false
          imagePullPolicy: Always
          volumeMounts:
            - name: jenkins-data
              mountPath: /var/lib/jenkins
          terminationMessagePolicy: File
          image: "" # to be updated by openshift trigger configured in annotation
      serviceAccount: jenkins
      volumes:
        - name: jenkins-data
          persistentVolumeClaim:
            claimName: jenkins
      dnsPolicy: ClusterFirst

I updated the pagure jenkins instance from version 2.401.1 to version 2.426.3 successful last night.
Part of the update was the migration from the deprecated DeploymentConfig to Deployment.
Here is the complete write up of the steps that worked for me and might help others too.


Step 1 - Migrate from DeploymentConfig to Deployment

Pre-Req: Current oc cli tool installed and configured to connect to the ocp project. Assumption that the resources are named jenkins, adjust if necessary for your environment.

Export the current DeploymentConfig for two reasons: As Backup and to create your Deployment manifest from it.

oc get deploymentconfig jenkins -o yaml > deploymentconfig_jenkins.yaml

cp deploymentconfig_jenkins.yaml deployment_jenkins.yaml

Changes that required in deployment_jenkins.yaml:
- replace apiVersion: apps.openshift.io/v1 with apiVersion: apps/v1
- change kind: DeploymentConfig to kind: Deployment
- change spec.selectors from:

    selector:
      name: ...

to

    selector:
      matchLabels:
        name: ...
  • set spec.template.spec.containers.image to a valid image or use the trigger annotation
  • remove spec.triggers
  • add annotation image.openshift.io/triggers: '[{"from":{"kind":"ImageStreamTag","namespace":"openshift","name":"jenkins:2"},"fieldPath":"spec.template.spec.containers[?(@.name==\"jenkins\")].image"}]' as replacement for the ImageStream trigger functionality
  • set image: "openshift/jenkins:2" as replacement to the current configured image. Together with the annotation it will behave identical to the current DeploymentConfig that way.
  • remove spec.test
  • remove this fields from spec.strategy (if they exist)
  • activeDeadlineSeconds
  • resources
  • rollingParams.intervalSeconds
  • rollingParams.timeoutSeconds
  • rollingParams.updatePeriodSeconds
  • rename spec.strategy.rollingParams to spec.strategy.rollingUpdate (if it exist)
  • if spec.strategy.type is set to Rolling replace the value with RollingUpdate

After the changes, the manifest should look similar to this example:

kind: Deployment
apiVersion: apps/v1
metadata:
  annotations:
    image.openshift.io/triggers: '[{"from":{"kind":"ImageStreamTag","namespace":"openshift","name":"2"},"fieldPath":"spec.template.spec.containers[?(@.name==\"jenkins\")].image"}]'
  name: jenkins
  namespace: pagure
  labels:
    app: cico-workspace
    template: cico-workspace-template
spec:
  strategy:
    type: Recreate
    recreateParams:
      timeoutSeconds: 1200
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      name: jenkins
  template:
    metadata:
      creationTimestamp: null
      labels:
        name: jenkins
    spec:
      restartPolicy: Always
      serviceAccountName: jenkins
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 30
      securityContext: {}
      containers:
        - resources:
            limits:
              memory: 3Gi
          readinessProbe:
            httpGet:
              path: /login
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 3
            timeoutSeconds: 240
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          terminationMessagePath: /dev/termination-log
          name: jenkins
          livenessProbe:
            httpGet:
              path: /login
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 420
            timeoutSeconds: 240
            periodSeconds: 360
            successThreshold: 1
            failureThreshold: 2
          env:
            - name: OPENSHIFT_ENABLE_OAUTH
              value: 'true'
            - name: OPENSHIFT_ENABLE_REDIRECT_PROMPT
              value: 'true'
            - name: DISABLE_ADMINISTRATIVE_MONITORS
              value: 'True'
            - name: KUBERNETES_MASTER
              value: 'https://kubernetes.default:443'
            - name: KUBERNETES_TRUST_CERTIFICATES
              value: 'true'
            - name: JENKINS_SERVICE_NAME
              value: jenkins
            - name: JNLP_SERVICE_NAME
              value: jenkins-jnlp
          securityContext:
            capabilities: {}
            privileged: false
          imagePullPolicy: Always
          volumeMounts:
            - name: jenkins-data
              mountPath: /var/lib/jenkins
          terminationMessagePolicy: File
          image: "openshift/jenkins:2" # to be updated by openshift trigger configured in annotation
      serviceAccount: jenkins
      volumes:
        - name: jenkins-data
          persistentVolumeClaim:
            claimName: jenkins
      dnsPolicy: ClusterFirst

Now go ahead and delete the jenkins DeploymentConfig. Check afterwards that the related pod and all ReplicationController are gone:

oc delete deploymentconfig jenkins

oc get all

If there are still old ReplicationController left, delete them too, example of one that was called jenkins-2:

oc delete replicationcontroller jenkins-2

(Re)Deploy jenkins based on the new Deployment manifest:

oc apply -f deployment_jenkins.yaml

After a couple of minutes your instance should be available and in the exact same state as before.

Step 2 - Update Jenkins

In my case the installation came from Jenkins 2.401.1 and the latest version in the ImageStream is 2.426.3.
I hit issues with Plugin dependencies, the following order worked for me but can be different for other installations and configurations.

Disclaimer: I did all this in prod. We don't have a stg environment right now and it was ok to take the risk of a longer downtime. Depending on your use-case and criticality of jenkins for your project, consider to test the update in stg first. It's possible to break the Jenkins installation, plugin dependencies can become tricky. There are ways to fix things manually through the cli but this will take extra time and effort.

Login to Jenkins and update all installed Plugins with the Plugin Manager. Jenkins will restart and fail, very likely because of an issue like:

2024-05-15 20:37:22 SEVERE  hudson.PluginManager$1$3$2$1 reactOnCycle found cycle in plugin dependencies: (root=Plugin:sshd, deactivating all involved) Plugin:sshd -> Plugin:mina-sshd-api-core -> Plugin:ssh-credentials -> Plugin:credentials -> Plugin:configuration-as-code -> Plugin:prism-api -> Plugin:antisamy-markup-formatter -> Plugin:sshd

Now it's time to change the Deployment and use the image tag scheduled-upgrade-redeploy instead of 2.
You can either adjust your manifest file and re-apply it or use oc edit deployment jenkins.
In both cases, change the tag in the annotation and down in the container spec.
jenkins:2 becomes jenkins:scheduled-upgrade-redeploy.

You can use oc logs pods/jenkins-* -f to check the log output of the Jenkins pod.
If you need to fix something inside the container, use oc exec pods/jenkins-<podid> -i -t -- /bin/bash to access it. The Jenkins installation has it's data in /var/lib/jenkins/.

When your Jenkins installation is up again, check for deprecated plugins and uninstall them if you don't use them.
In my case:

- JavaScript GUI Lib: ACE Editor bundle plugin, Version 1.1
- JavaScript GUI Lib: Moment.js bundle plugin, Version 1.1.1
- Pipeline: Deprecated Groovy Libraries, Version 609.vd95673f149b_b

Also check for plugins with know vulnerabilities and available updates.
In my case:

Matrix Project Plugin 818.v7eb_e657db_924
Path traversal vulnerability
A fix for this issue is available. Go to the plugin manager to update the plugin.

Bitbucket Branch Source Plugin 856.v04c46c86f911
Incorrect trust policy behavior for pull requests from forks
A fix for this issue is available. Go to the plugin manager to update the plugin.

You will get a message that a newer Jenkins version is available. Ignore that for now, it's not available in ImageStream yet and with the tag scheduled-upgrade-redeploy, new version will be installed automatically in future.
There will also be a warning that Java 11 is EOL end of 2024. I'm sure until then a new image will be released that use a more recent version :)

Metadata Update from @wombelix:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)

a month ago

Log in to comment on this ticket.

Metadata