Hi,
the jenkins deployment (https://jenkins-pagure.apps.ocp.cloud.ci.centos.org/) in the pagure OCP namespace is a bit dated: Jenkins 2.401.1, latest available 2.440.3.
pagure
Also the plugin manager complains about a lot outdated ones that need attention. What's the process to get it updated? And does that only include Jenkins itself or the Plugins too? Or can / should I manage all Plugin related stuff (updates, install new plugins) through the Jenkins Web UI directly?
I checked out https://docs.infra.centos.org/operations/ci/adding_cico_tenant/adding_cico_tenants/, https://centosci.github.io/ocp4-docs/ and the ansible playbooks in https://github.com/CentOS/ansible-infra-playbooks but who does what when it comes to an update isn't explained :-/
Dom
The jenkins ImageStream in openshift is managed/tagged by Openshift/Red Hat ... So it's updated whenever they have a new release, and that's how it was updated in the past.
I wanted to search for the link and it's the image : https://catalog.redhat.com/software/containers/openshift4/ose-jenkins/5cdd918ad70cc57c44b2d279
But I see also that they have a new image, which is different, based on information from https://github.com/openshift/jenkins?tab=readme-ov-file#411-and-higher :/ => https://catalog.redhat.com/software/containers/ocp-tools-4/jenkins-rhel8/5fe1f38288e9c2f788526306
So different name and I guess different ImageStream in openshift now. IT would need some investigation to see if that's possible to just modify existing deploymentconfig (themselves now deprecated and replaced by deployments) to use that different ImageStream
As you have admin rights in your project that's something that can be tested. We can eventually sync on a .stg. cluster or in a different namespace to see how that can be done ? My time is currently very limited to due to plenty of infra work related to c8s/c7 EOL plan and also onboarding c10s in infra (including CI, etc)
Metadata Update from @arrfab: - Issue tagged with: centos-ci-infra, feature-request, high-gain, high-trouble, investigation
The jenkins ImageStream in openshift is managed/tagged by Openshift/Red Hat ... So it's updated whenever they have a new release, and that's how it was updated in the past. I wanted to search for the link and it's the image : https://catalog.redhat.com/software/containers/openshift4/ose-jenkins/5cdd918ad70cc57c44b2d279 But I see also that they have a new image, which is different, based on information from https://github.com/openshift/jenkins?tab=readme-ov-file#411-and-higher :/ => https://catalog.redhat.com/software/containers/ocp-tools-4/jenkins-rhel8/5fe1f38288e9c2f788526306 So different name and I guess different ImageStream in openshift now.
So different name and I guess different ImageStream in openshift now.
OK that would explain why it's not updated anymore, makes sense.
IT would need some investigation to see if that's possible to just modify existing deploymentconfig (themselves now deprecated and replaced by deployments) to use that different ImageStream As you have admin rights in your project that's something that can be tested.
IT would need some investigation to see if that's possible to just modify existing deploymentconfig (themselves now deprecated and replaced by deployments) to use that different ImageStream
As you have admin rights in your project that's something that can be tested.
I'm totally fine doing that migration and update myself. I just wanted to double check first if there is any process in place. Or if I would interfere with any playbooks or automation you guys have. But as I understand, this isn't the case so I will not disturb anyone from the infra team with whatever I do :)
We can eventually sync on a .stg. cluster or in a different namespace to see how that can be done ?
If you can free up a little time, a sync to stg would be nice to have. But a slight outage wouldn't be much of a problem for the PR Pipeline we have and I can even deploy something side by side in the current namespace.
My time is currently very limited to due to plenty of infra work related to c8s/c7 EOL plan and also onboarding c10s in infra (including CI, etc)
Totally understand that and really appreciate that you took the time to respond to the ticket so fast!
So let's keep that ticket open for now but I'll try to get the move from DeploymentConfig to Deployment and the new ImageStream done myself.
I'll update the ticket with my progress and steps, maybe that helps if there are other similar deployments that need an update.
I did some research and things make much more sense now. A draft Deployment Manifest is ready. I have to do some tests and plan the update steps.
Deployment
Following all my notes if it's needed for later reference or helpful for someone else.
Why was there no auto-update of the jenkins image to a recent version?
TL;DR tag 2 only receive an updated when ocp is updated. the new tag scheduled-upgrade-redeploy will trigger as soon a new image is published. currently deployed and latest tag 2 image is from 15/06/2023, latest tag scheduled-upgrade-redeploy image is from 29/01/2024. Not bleeding edge but still much more up to date.
2
scheduled-upgrade-redeploy
15/06/2023
29/01/2024
Long version:
https://console-openshift-console.apps.ocp.cloud.ci.centos.org/ runs OpenShift version 4.15 (k8s 1.28) The current ImageStream openshift/jenkins config should be therefore this one: https://github.com/openshift/jenkins/blob/master/openshift/imagestreams/jenkins-rhel.yaml Tag 2 points to registry.redhat.io/ocp-tools-4/jenkins-rhel8:v4.13.0-1686682222, which is this container image: https://catalog.redhat.com/software/containers/ocp-tools-4/jenkins-rhel8/5fe1f38288e9c2f788526306?image=648adf0d185f308ebdc41cbd&architecture=amd64 (published 15/06/2023) The SHA256 matches the currently running image if the jenkins DeploymentConfig. Based on https://access.redhat.com/documentation/en-us/openshift_container_platform/4.15/html-single/jenkins/index#relocation-of-openshift-jenkins-images_important-changes-to-openshift-jenkins-images_important
4.15
1.28
openshift/jenkins
registry.redhat.io/ocp-tools-4/jenkins-rhel8:v4.13.0-1686682222
DeploymentConfig
By default, this image is referenced by the jenkins:2 image stream tag of Jenkins image stream in the openshift namespace in the ImageStream YAML file in the Samples Operator payload.
This this tag (and latest, ocp-upgrade-redeploy) doesn't change before the next OpenShift release and therefore doesn't trigger an update of the jenkins deployment. To update jenkins independent of OCP, the tag scheduled-upgrade-redeploy has to be used:
latest
ocp-upgrade-redeploy
To automatically redeploy the latest version of the Jenkins image when it is released, use this image stream tag in your Jenkins deployment configuration. This image stream tag uses the periodic importing of image stream tags feature of the OpenShift Container Platform image stream controller, which checks for changes in the backing image. If the image changes, for example, due to a recent Jenkins security advisory, OpenShift Container Platform triggers a redeployment of your Jenkins deployment configuration.
This tag points to registry.redhat.io/ocp-tools-4/jenkins-rhel8:v4.13.0, current tagged image: https://catalog.redhat.com/software/containers/ocp-tools-4/jenkins-rhel8/5fe1f38288e9c2f788526306?architecture=amd64&image=65b76ad991230cbe794cadc3 (published 29/01/2024)
registry.redhat.io/ocp-tools-4/jenkins-rhel8:v4.13.0
Migrate from DeploymentConfig > Deployment
TL;DR capabilities liger trigger are not available but to update an image automatically, an annotation can be used instead. Adjust kind and some other fields. Clean up and drop generated things like managedFields, not relevant when the manifest is used to create a new deployment.
trigger
kind
managedFields
Setup trigger that updates deployment on ImageStream event, annotation: https://docs.openshift.com/container-platform/4.14/openshift_images/triggering-updates-on-imagestream-changes.html#images-triggering-updates-imagestream-changes-kubernetes-cli_triggering-updates-on-imagestream-changes
ImageStream
Current Trigger:
triggers: - type: ImageChange imageChangeParams: automatic: true containerNames: - jenkins from: kind: ImageStreamTag namespace: openshift name: 'jenkins:2' lastTriggeredImage: 'image-registry.openshift-image-registry.svc:5000/openshift/jenkins@sha256:eab456afb39ed4607b2ee61c8c7635ab1c5ff8f8bddf7640c557e792504d545f' - type: ConfigChange
Annotation for Deployment with auto updates to latest jenkins version available:
image.openshift.io/triggers: '[{"from":{"kind":"ImageStreamTag","namespace":"openshift","name":"jenkins:scheduled-upgrade-redeploy"},"fieldPath":"spec.template.spec.containers[?(@.name==\"jenkins\")].image"}]'
Changes that are required in the Manifest file: - replace apiVersion: apps.openshift.io/v1 with apiVersion: apps/v1 - change kind: DeploymentConfig to kind: Deployment - change spec.selectors from:
apiVersion: apps.openshift.io/v1
apiVersion: apps/v1
kind: DeploymentConfig
kind: Deployment
spec.selectors
selector: name: ...
to
selector: matchLabels: name: ...
spec.template.spec.containers.image
spec.triggers
spec.test
spec.strategy
activeDeadlineSeconds
resources
rollingParams.intervalSeconds
rollingParams.timeoutSeconds
rollingParams.updatePeriodSeconds
spec.strategy.rollingParams
spec.strategy.rollingUpdate
spec.strategy.type
Rolling
RollingUpdate
Draft Manifest
That's the current, untested, draft to replace DeploymentConfig with and enforce regular jenkins updates:
kind: Deployment apiVersion: apps/v1 metadata: annotations: image.openshift.io/triggers: '[{"from":{"kind":"ImageStreamTag","namespace":"openshift","name":"jenkins:scheduled-upgrade-redeploy"},"fieldPath":"spec.template.spec.containers[?(@.name==\"jenkins\")].image"}]' name: jenkins namespace: pagure labels: app: cico-workspace template: cico-workspace-template spec: strategy: type: Recreate recreateParams: timeoutSeconds: 1200 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: name: jenkins template: metadata: creationTimestamp: null labels: name: jenkins spec: restartPolicy: Always serviceAccountName: jenkins schedulerName: default-scheduler terminationGracePeriodSeconds: 30 securityContext: {} containers: - resources: limits: memory: 3Gi readinessProbe: httpGet: path: /login port: 8080 scheme: HTTP initialDelaySeconds: 3 timeoutSeconds: 240 periodSeconds: 10 successThreshold: 1 failureThreshold: 3 terminationMessagePath: /dev/termination-log name: jenkins livenessProbe: httpGet: path: /login port: 8080 scheme: HTTP initialDelaySeconds: 420 timeoutSeconds: 240 periodSeconds: 360 successThreshold: 1 failureThreshold: 2 env: - name: OPENSHIFT_ENABLE_OAUTH value: 'true' - name: OPENSHIFT_ENABLE_REDIRECT_PROMPT value: 'true' - name: DISABLE_ADMINISTRATIVE_MONITORS value: 'True' - name: KUBERNETES_MASTER value: 'https://kubernetes.default:443' - name: KUBERNETES_TRUST_CERTIFICATES value: 'true' - name: JENKINS_SERVICE_NAME value: jenkins - name: JNLP_SERVICE_NAME value: jenkins-jnlp securityContext: capabilities: {} privileged: false imagePullPolicy: Always volumeMounts: - name: jenkins-data mountPath: /var/lib/jenkins terminationMessagePolicy: File image: "" # to be updated by openshift trigger configured in annotation serviceAccount: jenkins volumes: - name: jenkins-data persistentVolumeClaim: claimName: jenkins dnsPolicy: ClusterFirst
I updated the pagure jenkins instance from version 2.401.1 to version 2.426.3 successful last night. Part of the update was the migration from the deprecated DeploymentConfig to Deployment. Here is the complete write up of the steps that worked for me and might help others too.
Step 1 - Migrate from DeploymentConfig to Deployment
Pre-Req: Current oc cli tool installed and configured to connect to the ocp project. Assumption that the resources are named jenkins, adjust if necessary for your environment.
jenkins
Export the current DeploymentConfig for two reasons: As Backup and to create your Deployment manifest from it.
oc get deploymentconfig jenkins -o yaml > deploymentconfig_jenkins.yaml cp deploymentconfig_jenkins.yaml deployment_jenkins.yaml
Changes that required in deployment_jenkins.yaml: - replace apiVersion: apps.openshift.io/v1 with apiVersion: apps/v1 - change kind: DeploymentConfig to kind: Deployment - change spec.selectors from:
deployment_jenkins.yaml
image.openshift.io/triggers: '[{"from":{"kind":"ImageStreamTag","namespace":"openshift","name":"jenkins:2"},"fieldPath":"spec.template.spec.containers[?(@.name==\"jenkins\")].image"}]'
image: "openshift/jenkins:2"
After the changes, the manifest should look similar to this example:
kind: Deployment apiVersion: apps/v1 metadata: annotations: image.openshift.io/triggers: '[{"from":{"kind":"ImageStreamTag","namespace":"openshift","name":"2"},"fieldPath":"spec.template.spec.containers[?(@.name==\"jenkins\")].image"}]' name: jenkins namespace: pagure labels: app: cico-workspace template: cico-workspace-template spec: strategy: type: Recreate recreateParams: timeoutSeconds: 1200 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: name: jenkins template: metadata: creationTimestamp: null labels: name: jenkins spec: restartPolicy: Always serviceAccountName: jenkins schedulerName: default-scheduler terminationGracePeriodSeconds: 30 securityContext: {} containers: - resources: limits: memory: 3Gi readinessProbe: httpGet: path: /login port: 8080 scheme: HTTP initialDelaySeconds: 3 timeoutSeconds: 240 periodSeconds: 10 successThreshold: 1 failureThreshold: 3 terminationMessagePath: /dev/termination-log name: jenkins livenessProbe: httpGet: path: /login port: 8080 scheme: HTTP initialDelaySeconds: 420 timeoutSeconds: 240 periodSeconds: 360 successThreshold: 1 failureThreshold: 2 env: - name: OPENSHIFT_ENABLE_OAUTH value: 'true' - name: OPENSHIFT_ENABLE_REDIRECT_PROMPT value: 'true' - name: DISABLE_ADMINISTRATIVE_MONITORS value: 'True' - name: KUBERNETES_MASTER value: 'https://kubernetes.default:443' - name: KUBERNETES_TRUST_CERTIFICATES value: 'true' - name: JENKINS_SERVICE_NAME value: jenkins - name: JNLP_SERVICE_NAME value: jenkins-jnlp securityContext: capabilities: {} privileged: false imagePullPolicy: Always volumeMounts: - name: jenkins-data mountPath: /var/lib/jenkins terminationMessagePolicy: File image: "openshift/jenkins:2" # to be updated by openshift trigger configured in annotation serviceAccount: jenkins volumes: - name: jenkins-data persistentVolumeClaim: claimName: jenkins dnsPolicy: ClusterFirst
Now go ahead and delete the jenkins DeploymentConfig. Check afterwards that the related pod and all ReplicationController are gone:
oc delete deploymentconfig jenkins oc get all
If there are still old ReplicationController left, delete them too, example of one that was called jenkins-2:
jenkins-2
oc delete replicationcontroller jenkins-2
(Re)Deploy jenkins based on the new Deployment manifest:
oc apply -f deployment_jenkins.yaml
After a couple of minutes your instance should be available and in the exact same state as before.
Step 2 - Update Jenkins
In my case the installation came from Jenkins 2.401.1 and the latest version in the ImageStream is 2.426.3. I hit issues with Plugin dependencies, the following order worked for me but can be different for other installations and configurations.
Disclaimer: I did all this in prod. We don't have a stg environment right now and it was ok to take the risk of a longer downtime. Depending on your use-case and criticality of jenkins for your project, consider to test the update in stg first. It's possible to break the Jenkins installation, plugin dependencies can become tricky. There are ways to fix things manually through the cli but this will take extra time and effort.
Login to Jenkins and update all installed Plugins with the Plugin Manager. Jenkins will restart and fail, very likely because of an issue like:
2024-05-15 20:37:22 SEVERE hudson.PluginManager$1$3$2$1 reactOnCycle found cycle in plugin dependencies: (root=Plugin:sshd, deactivating all involved) Plugin:sshd -> Plugin:mina-sshd-api-core -> Plugin:ssh-credentials -> Plugin:credentials -> Plugin:configuration-as-code -> Plugin:prism-api -> Plugin:antisamy-markup-formatter -> Plugin:sshd
Now it's time to change the Deployment and use the image tag scheduled-upgrade-redeploy instead of 2. You can either adjust your manifest file and re-apply it or use oc edit deployment jenkins. In both cases, change the tag in the annotation and down in the container spec. jenkins:2 becomes jenkins:scheduled-upgrade-redeploy.
oc edit deployment jenkins
jenkins:2
jenkins:scheduled-upgrade-redeploy
You can use oc logs pods/jenkins-* -f to check the log output of the Jenkins pod. If you need to fix something inside the container, use oc exec pods/jenkins-<podid> -i -t -- /bin/bash to access it. The Jenkins installation has it's data in /var/lib/jenkins/.
oc logs pods/jenkins-* -f
oc exec pods/jenkins-<podid> -i -t -- /bin/bash
/var/lib/jenkins/
When your Jenkins installation is up again, check for deprecated plugins and uninstall them if you don't use them. In my case:
- JavaScript GUI Lib: ACE Editor bundle plugin, Version 1.1 - JavaScript GUI Lib: Moment.js bundle plugin, Version 1.1.1 - Pipeline: Deprecated Groovy Libraries, Version 609.vd95673f149b_b
Also check for plugins with know vulnerabilities and available updates. In my case:
Matrix Project Plugin 818.v7eb_e657db_924 Path traversal vulnerability A fix for this issue is available. Go to the plugin manager to update the plugin. Bitbucket Branch Source Plugin 856.v04c46c86f911 Incorrect trust policy behavior for pull requests from forks A fix for this issue is available. Go to the plugin manager to update the plugin.
You will get a message that a newer Jenkins version is available. Ignore that for now, it's not available in ImageStream yet and with the tag scheduled-upgrade-redeploy, new version will be installed automatically in future. There will also be a warning that Java 11 is EOL end of 2024. I'm sure until then a new image will be released that use a more recent version :)
Metadata Update from @wombelix: - Issue close_status updated to: Fixed with Explanation - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.