#728 OCP Jenkins lost cloud integration
Closed: Fixed 23 days ago by mobrien. Opened 2 months ago by evgeni.

Ohai,

our Jenkins (https://jenkins-foreman.apps.ocp.ci.centos.org/) seems to have lost the connection to the underlying OpenShift, as there are no more clouds defined in "Manage Jenkins → Manage Nodes and Clouds → Configure Clouds".

If I understood @arrfab correctly on IRC last week, this is most probably due to a version mismatch between the running OpenShift and the Jenkins OpenShift/Kubernetes plugins.

There are several similar reports of such behaviour on the internet:

However, judging by those, the combination of plugins we have should work?

Kubernetes 1.31.0
Kubernetes Client API 5.10.1-171.vaa0774fb8c20
OpenShift Sync 1.0.53

Could you please double-check against a working Jenkins which versions we should stick to?


The logs look fine to me too?

2022-04-04 11:12:14 INFO    io.fabric8.jenkins.openshiftsync.GlobalPluginConfiguration configChange OpenShift Sync Plugin processing a newly supplied configuration
2022-04-04 11:12:14 INFO    io.fabric8.jenkins.openshiftsync.OpenShiftUtils shutdownOpenShiftClient Stopping openshift client: null
2022-04-04 11:12:14 INFO    io.fabric8.jenkins.openshiftsync.OpenShiftUtils initializeOpenShiftClient Current OpenShift Client Configuration: io.fabric8.openshift.client.OpenShiftConfig@7e2ba938[oapiVersion=v1,openShiftUrl=https://kubernetes.default:443/oapi/v1/,buildTimeout=300000,openshiftApiGroupsEnabled=false,disableApiGroupCheck=false,trustCerts=true,disableHostnameVerification=false,masterUrl=https://kubernetes.default:443/,apiVersion=v1,namespace=foreman,caCertFile=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt,caCertData=<null>,clientCertFile=<null>,clientCertData=<null>,clientKeyFile=<null>,clientKeyData=<null>,clientKeyAlgo=<null>,clientKeyPassphrase=changeit,trustStoreFile=<null>,trustStorePassphrase=<null>,keyStoreFile=<null>,keyStorePassphrase=<null>,authProvider=<null>,requestConfig=io.fabric8.kubernetes.client.RequestConfig@3a462db2,contexts=[],currentContext=<null>,username=<null>,password=<null>,oauthToken=<null>,watchReconnectInterval=1000,watchReconnectLimit=-1,connectionTimeout=10000,uploadConnectionTimeout=10000,uploadRequestTimeout=120000,requestRetryBackoffLimit=0,requestRetryBackoffInterval=0,requestTimeout=10000,rollingTimeout=900000,scaleTimeout=600000,loggingInterval=20000,websocketTimeout=5000,websocketPingInterval=30000,maxConcurrentRequests=64,maxConcurrentRequestsPerHost=5,impersonateUsername=<null>,oauthTokenProvider=<null>,impersonateGroup=<null>,impersonateGroups=<null>,impersonateExtras=<null>,http2Disable=false,httpProxy=<null>,httpsProxy=<null>,proxyUsername=<null>,proxyPassword=<null>,noProxy={},userAgent=<null>,tlsVersions={TLS_1_2},errorMessages={401=Unauthorized! Configured service account doesn't have access. Service account may have been revoked., 403=Forbidden!Configured service account doesn't have access. Service account may have been revoked.},customHeaders=<null>,autoConfigure=false,file=<null>]
2022-04-04 11:12:15 INFO    io.fabric8.jenkins.openshiftsync.OpenShiftUtils initializeOpenShiftClient New OpenShift client initialized: io.fabric8.openshift.client.DefaultOpenShiftClient@3a0386be
2022-04-04 11:12:16 INFO    io.fabric8.jenkins.openshiftsync.GlobalPluginConfigurationTimerTask doRun Confirming Jenkins is started
2022-04-04 11:12:16 INFO    io.fabric8.jenkins.openshiftsync.GlobalPluginConfigurationTimerTask waitForJenkinsStartup Jenkins not ready...
2022-04-04 11:12:16 INFO    jenkins.InitReactorRunner$1 onAttained System config adapted
2022-04-04 11:12:16 INFO    jenkins.InitReactorRunner$1 onAttained Loaded all jobs
2022-04-04 11:12:16 INFO    jenkins.InitReactorRunner$1 onAttained Configuration for all jobs updated
2022-04-04 11:12:16 INFO    hudson.model.AsyncPeriodicWork lambda$doRun$1 Started Download metadata
2022-04-04 11:12:16 INFO    hudson.model.AsyncPeriodicWork lambda$doRun$1 Finished Download metadata. 3 ms
2022-04-04 11:12:16 INFO    jenkins.util.groovy.GroovyHookScript execute Executing /var/lib/jenkins/init.groovy.d/update-center-init.groovy
2022-04-04 11:12:17 INFO    io.fabric8.jenkins.openshiftsync.GlobalPluginConfigurationTimerTask waitForJenkinsStartup Jenkins not ready...
2022-04-04 11:12:17 INFO    jenkins.InitReactorRunner$1 onAttained Completed initialization
2022-04-04 11:12:17 INFO    io.fabric8.jenkins.openshiftsync.GlobalPluginConfigurationTimerTask stop Stopping all informers ...
2022-04-04 11:12:17 INFO    io.fabric8.jenkins.openshiftsync.GlobalPluginConfigurationTimerTask stop Stopped all informers
2022-04-04 11:12:17 INFO    io.fabric8.jenkins.openshiftsync.BuildConfigInformer start Starting BuildConfig informer for {} !!foreman
2022-04-04 11:12:17 INFO    io.fabric8.jenkins.openshiftsync.BuildConfigManager reconcileJobsAndBuildConfigs Reconciling jobs and build configs
2022-04-04 11:12:17 INFO    io.fabric8.jenkins.openshiftsync.BuildConfigManager reconcileJobsAndBuildConfigs Reconciling jobs and build configs
2022-04-04 11:12:17 INFO    io.fabric8.jenkins.openshiftsync.BuildConfigInformer start BuildConfig informer started for namespace: {}foreman
2022-04-04 11:12:17 INFO    io.fabric8.jenkins.openshiftsync.BuildInformer start Starting Build informer for {} !!foreman
2022-04-04 11:12:17 INFO    io.fabric8.jenkins.openshiftsync.BuildManager reconcileRunsAndBuilds Reconciling job runs and builds
2022-04-04 11:12:17 INFO    io.fabric8.jenkins.openshiftsync.BuildManager reconcileRunsAndBuilds Reconciling job runs and builds
2022-04-04 11:12:17 INFO    io.fabric8.jenkins.openshiftsync.BuildInformer start Build informer started for namespace: {}foreman
2022-04-04 11:12:17 INFO    io.fabric8.jenkins.openshiftsync.ConfigMapInformer start Starting configMap informer for {} !!foreman
2022-04-04 11:12:17 INFO    io.fabric8.jenkins.openshiftsync.ConfigMapInformer start ConfigMap informer started for namespace: {}foreman
2022-04-04 11:12:17 INFO    io.fabric8.jenkins.openshiftsync.ImageStreamInformer start Starting ImageStream informer for {} !!foreman
2022-04-04 11:12:17 INFO    io.fabric8.jenkins.openshiftsync.ImageStreamInformer start ImageStream informer started for namespace: {}foreman
2022-04-04 11:12:17 INFO    io.fabric8.jenkins.openshiftsync.SecretInformer start Starting secret informer {} !!foreman
2022-04-04 11:12:17 INFO    io.fabric8.jenkins.openshiftsync.SecretInformer start Secret informer started for namespace: {}foreman
2022-04-04 11:12:17 INFO    io.fabric8.jenkins.openshiftsync.GlobalPluginConfigurationTimerTask start All the namespaced informers have been registered!! ... starting all registered informers
2022-04-04 11:12:17 INFO    io.fabric8.jenkins.openshiftsync.GlobalPluginConfigurationTimerTask start All registered informers have been started
2022-04-04 11:12:18 INFO    io.fabric8.jenkins.openshiftsync.ConfigMapInformer onAdd ConfigMap informer received add event for: {}cico-workspace
2022-04-04 11:12:18 INFO    io.fabric8.jenkins.openshiftsync.PodTemplateUtils addAgents Adding PodTemplate(s) for foreman

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: centos-ci-infra, high-gain, high-trouble

a month ago

I have tried various combinations of plugins but as you have mentioned the ones there look to be the correct version. I also tried to rollout the pod to see if that would pick up the correct configuration but that also made no difference.

My next plan would be to recreate the project from scratch to bring it back to a "default" state and see if the issue still persists. Is this something you would be ok with me doing?

Please do. All configs are in JJB.

Metadata Update from @mobrien:
- Issue assigned to mobrien

a month ago

Metadata Update from @zlopez:
- Issue priority set to: Medium Priority (was: Waiting on Assignee)

a month ago

@evgeni Sorry for the delay on this. We rolled out a cluster upgrade on Friday which had a new Jenkins image along with it. This appears to have fixed the issue.

Could you test it when you get a chance and let us know if all is working ok?

It seems to work fine again, thanks!

I didn't dare to perform a plugin upgrade round yet.

One thing that I noticed, the user created in https://pagure.io/centos-infra/issue/569 is missing now :(

From irc chat this looks to be resolved so I will close out the ticket

Metadata Update from @mobrien:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

23 days ago

Login to comment on this ticket.

Metadata
Boards 1
CentOS CI Infra Status: Backlog