Since the weekend it seems that our Jenkins at https://jenkins-ceph-csi.apps.ocp.ci.centos.org/ fails to start cico-workspace pods as seen in the console logs of all pending jobs.
cico-workspace
The Jenkins container logs contain failures like OCP Console:
2022-01-31 14:37:52 INFO io.fabric8.jenkins.openshiftsync.OpenShiftUtils initializeOpenShiftClient Current OpenShift Client Configuration: io.fabric8.openshift.client.OpenShiftConfig@743b4717[oapiVersion=v1,openShiftUrl=https://kubernetes.default:443/oapi/v1/,buildTimeout=300000,openshiftApiGroupsEnabled=false,disableApiGroupCheck=false,trustCerts=true,disableHostnameVerification=false,masterUrl=https://kubernetes.default:443/,apiVersion=v1,namespace=ceph-csi,caCertFile=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt,caCertData=<null>,clientCertFile=<null>,clientCertData=<null>,clientKeyFile=<null>,clientKeyData=<null>,clientKeyAlgo=<null>,clientKeyPassphrase=changeit,trustStoreFile=<null>,trustStorePassphrase=<null>,keyStoreFile=<null>,keyStorePassphrase=<null>,authProvider=<null>,requestConfig=io.fabric8.kubernetes.client.RequestConfig@36fd697d,contexts=[],currentContext=<null>,username=<null>,password=<null>,oauthToken=<null>,watchReconnectInterval=1000,watchReconnectLimit=-1,connectionTimeout=1... 2022-01-31 14:37:52 INFO io.fabric8.jenkins.openshiftsync.OpenShiftUtils initializeOpenShiftClient New OpenShift client initialized: io.fabric8.openshift.client.DefaultOpenShiftClient@1857de7c 2022-01-31 14:37:52 WARNING hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1 error Failed to instantiate Key[type=io.fabric8.jenkins.openshiftsync.GlobalPluginConfiguration, annotation=[none]]; skipping this component com.google.inject.ProvisionException: Unable to provision, see the following errors: 1) Error injecting constructor, java.lang.NoSuchMethodError: 'okhttp3.OkHttpClient io.fabric8.openshift.client.DefaultOpenShiftClient.getHttpClient()' at io.fabric8.jenkins.openshiftsync.GlobalPluginConfiguration.<init>(GlobalPluginConfiguration.java:109) 1 error at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:52) at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145) at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:441) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012) at hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:401) at hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:392) at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:349) at hudson.ExtensionList.load(ExtensionList.java:382) at hudson.ExtensionList.ensureLoaded(ExtensionList.java:318) at hudson.ExtensionList.iterator(ExtensionList.java:170) at hudson.ExtensionList.get(ExtensionList.java:147) at org.jenkinsci.main.modules.sshd.SSHD.get(SSHD.java:263) at org.jenkinsci.main.modules.sshd.SSHD.init(SSHD.java:268) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104) at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:180) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1158) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.NoSuchMethodError: 'okhttp3.OkHttpClient io.fabric8.openshift.client.DefaultOpenShiftClient.getHttpClient()' at io.fabric8.jenkins.openshiftsync.OpenShiftUtils.initializeOpenShiftClient(OpenShiftUtils.java:153) at io.fabric8.jenkins.openshiftsync.GlobalPluginConfiguration.start(GlobalPluginConfiguration.java:128) at io.fabric8.jenkins.openshiftsync.GlobalPluginConfiguration.configChange(GlobalPluginConfiguration.java:123) at io.fabric8.jenkins.openshiftsync.GlobalPluginConfiguration.<init>(GlobalPluginConfiguration.java:111) at io.fabric8.jenkins.openshiftsync.GlobalPluginConfiguration$$FastClassByGuice$$7bc306ea.newInstance(<generated>) at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61) at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105) at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:32) at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:89) at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115) at hudson.ExtensionFinder$GuiceFinder$SezpozModule.onProvision(ExtensionFinder.java:568) at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126) at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:87) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ... 29 more
There have been no changes on our side, so this seems to have been caused by some infrastructure change. The Jenkins pod has been restarted several times now, but the error copied above keeps recurring.
Node was restarted. Let us know if it's working again
Metadata Update from @zlopez: - Issue priority set to: Waiting on Reporter (was: Needs Review)
@mobrien I do not see a change in our Jenkins. I have now cleaned-up the failed cico-workspace pods, in the hope any quotas/limits are cleared. That also does not seem to help. A restart of Jenkins after that, still gave the same error as posted earlier.
With help from @anoopcs I was able to reconstruct the "configured cloud" and related "pod template" through the Jenkins webui. It seems the OpenShift Sync plugin has some issues, and the cico-workspace ConfigMap in the OCP project is not being used.
Currently jobs are running again, but there may be some missing configuration somewhere...
[backlog refinement] Verify if the configuration is correctly synced up with OpenShift in newer Jenkins image from Red Hat catalog.
Metadata Update from @zlopez: - Issue tagged with: centos-ci-infra, medium-gain, medium-trouble
@mobrien : is that what was also solved in recent #728 ? and if so maybe this one can be closed too (as it seems same cico-workspace pod template issue, which normally was solved by your openshift/ocp update)
Metadata Update from @mobrien: - Issue close_status updated to: Fixed with Explanation - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.