It seems Jenkins is experiencing some network or storage performance issue.
For example https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-build-pipeline-trigger/454476/console - it took about 15mins to clone the repo branch and send message to FedoraMessaging.
12:45:18 Running on Jenkins in /var/lib/jenkins/jobs/fedora-build-pipeline-trigger/workspace [Pipeline] { [Pipeline] ansiColor [Pipeline] { [Pipeline] deleteDir [Pipeline] stage [Pipeline] { (upstream-fedora-pipeline-build-trigger) [Pipeline] echo 12:48:11 Starting upstream-fedora-pipeline-build-trigger <snip> 12:51:23 + git clone -b master --single-branch --depth 1 https://src.fedoraproject.org/rpms/python3.9/ 12:51:50 Cloning into 'python3.9'... 12:54:14 Checking out files: 11% (2/17) <snip> 1st attempt of sending message to FedoraMessaging failed 12:55:18 Topic is org.centos.prod.ci.pipeline.allpackages-build.package.queued [Pipeline] retry [Pipeline] { [Pipeline] timeout 12:55:39 Timeout set to expire in 1 min 0 sec [Pipeline] { [Pipeline] sendCIMessage 12:56:39 Cancelling nested steps due to timeout <snip> after retry message was sent 12:59:08 Retrying [Pipeline] { [Pipeline] timeout 12:59:25 Timeout set to expire in 1 min 0 sec [Pipeline] { [Pipeline] sendCIMessage 12:59:39 Sending message for job 'fedora-build-pipeline-trigger'. 12:59:39 Message id: 2020-6ec74f11-b34b-4a1e-a373-fe8c70b13a5d 12:59:39 Message topic: org.centos.prod.ci.pipeline.allpackages-build.package.queued 12:59:39 JSON message body: 12:59:39 {"rev":"kojitask-44680924","ci_topic":"org.centos.prod.ci.pipeline.allpackages-build.package.queued","repo":"python3.9","scratch":false,"nvr":"python3.9-3.9.0~b1-1.fc33","comment_id":"","branch":"master","test_guidance":"''","ref":"x86_64","build_id":"1508670","namespace":"rpms","original_spec_nvr":"","build_url":"https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-build-pipeline-trigger/detail/fedora-build-pipeline-trigger/454476/pipeline/","status":"SUCCESS","username":"churchyard"}
I've currently disabled the trigger for scratch builds to reduce the load in the pipeline
Metadata Update from @pingou: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: ci
I disabled the sturivny-test project as well to help with load for the moment.
Can you re-roll out your Jenkins deployment when you get a chance?
This will migrate to a new node while we figure out what's up with n21.kempty
Preparing for shutdown now - will scale down/up once the queue clears
Scaled down and up; let's see if it behaves :)
It seems to be working (as discussed in IRC); I'm walking through failed triggers and trying to restart what I can.
the pod was moved to a new node and it seems working fine, for now! @jimbair, can this be closed? :)
@siddharthvipul1 so I am trying to load it now:
https://jenkins-continuous-infra.apps.ci.centos.org/
504 Gateway Time-out The server didn't respond in time.
I know @msrb saw this behavior this morning - any ideas? :)
Hmm - if I go to /manage and login, it works fine, but it's the main page having issues.
If I go to a pipeline from my history, like so:
https://jenkins-continuous-infra.apps.ci.centos.org/view/Fedora%20All%20Packages%20Pipeline/job/fedora-pr-new-trigger/
It works as well. I am guessing it's to do with the config changes @msrb was working on so I think we're good.
I assume we can close this out, but we'll open a fresh one if needed. :)
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.