fedora-ci / general

Clone
Source Code
GIT

#76 [ERROR] koji: ServerOffline: database outage
Closed 4 years ago by churchyard. Opened 4 years ago by churchyard.

Closed
Reopen Issue

I cannot use the CI because it fails during the koji-build with:

[2019-10-14T20:24:17.739Z] + /tmp/koji_build_pr.sh
[2019-10-14T20:24:17.779Z] + '[' -z python3 ']'
[2019-10-14T20:24:17.779Z] + '[' -z 140 ']'
[2019-10-14T20:24:17.779Z] + '[' -z fa345e8baa5e47e0b8bfed05da2b569e ']'
[2019-10-14T20:24:17.779Z] + '[' -z bpeck/jenkins-continuous-infra.apps.ci.centos.org@FEDORAPROJECT.ORG ']'
[2019-10-14T20:24:17.779Z] + '[' -z https://src.fedoraproject.org ']'
[2019-10-14T20:24:17.779Z] ++ pwd
[2019-10-14T20:24:17.779Z] + CURRENTDIR=/workDir/workspace/fedora-rawhide-pr-pipeline
[2019-10-14T20:24:17.779Z] + '[' /workDir/workspace/fedora-rawhide-pr-pipeline == / ']'
[2019-10-14T20:24:17.779Z] + KOJI_PARAMS=
[2019-10-14T20:24:17.779Z] + RPMDIR=/workDir/workspace/fedora-rawhide-pr-pipeline/python3_repo
[2019-10-14T20:24:17.779Z] + LOGDIR=/workDir/workspace/fedora-rawhide-pr-pipeline/logs
[2019-10-14T20:24:17.779Z] + rm -rf '/workDir/workspace/fedora-rawhide-pr-pipeline/logs/*'
[2019-10-14T20:24:17.779Z] + mkdir -p /workDir/workspace/fedora-rawhide-pr-pipeline/logs
[2019-10-14T20:24:17.779Z] + rm -rf python3
[2019-10-14T20:24:17.779Z] + git clone https://src.fedoraproject.org/rpms/python3.git
[2019-10-14T20:24:17.779Z] Cloning into 'python3'...
[2019-10-14T20:24:19.665Z] + '[' 0 '!=' 0 ']'
[2019-10-14T20:24:19.665Z] + pushd python3
[2019-10-14T20:24:19.665Z] /workDir/workspace/fedora-rawhide-pr-pipeline/python3 /workDir/workspace/fedora-rawhide-pr-pipeline
[2019-10-14T20:24:19.665Z] + git checkout master
[2019-10-14T20:24:19.665Z] Already on 'master'
[2019-10-14T20:24:19.665Z] Your branch is up to date with 'origin/master'.
[2019-10-14T20:24:19.665Z] + git fetch -fu origin refs/pull/140/head:pr
[2019-10-14T20:24:20.590Z] From https://src.fedoraproject.org/rpms/python3
[2019-10-14T20:24:20.590Z]  * [new ref]         refs/pull/140/head -> pr
[2019-10-14T20:24:20.590Z] + git -c 'user.name=Fedora CI' -c user.email=ci@lists.fedoraproject.org merge pr -m 'Fedora CI pipeline'
[2019-10-14T20:24:20.590Z] Updating 4332c4f..a5f6c89
[2019-10-14T20:24:20.590Z] Fast-forward (no commit created; -m option ignored)
[2019-10-14T20:24:20.590Z]  00001-rpath.patch                           | 2 +-
[2019-10-14T20:24:20.590Z]  00102-lib64.patch                           | 2 +-
[2019-10-14T20:24:20.590Z]  00111-no-static-lib.patch                   | 2 +-
[2019-10-14T20:24:20.590Z]  00189-use-rpm-wheels.patch                  | 2 +-
[2019-10-14T20:24:20.590Z]  00251-change-user-install-location.patch    | 2 +-
[2019-10-14T20:24:20.590Z]  00274-fix-arch-names.patch                  | 2 +-
[2019-10-14T20:24:20.590Z]  00328-pyc-timestamp-invalidation-mode.patch | 2 +-
[2019-10-14T20:24:20.590Z]  python3.spec                                | 5 ++++-
[2019-10-14T20:24:20.590Z]  sources                                     | 2 +-
[2019-10-14T20:24:20.590Z]  9 files changed, 12 insertions(+), 9 deletions(-)
[2019-10-14T20:24:20.590Z] ++ rpm -q --define 'dist .fc32' --queryformat '%{name}-%{version}-%{release}\n' --specfile python3.spec
[2019-10-14T20:24:20.590Z] ++ head -n 1
[2019-10-14T20:24:20.590Z] + truenvr=python3-3.8.0-1.fc32
[2019-10-14T20:24:20.590Z] + echo original_spec_nvr=python3-3.8.0-1.fc32
[2019-10-14T20:24:20.590Z] ++ git log --pretty=format:
[2019-10-14T20:24:20.590Z] ++ wc -l
[2019-10-14T20:24:20.590Z] + commits=657
[2019-10-14T20:24:20.590Z] + sed -i 's/%{?dist}/%{?dist}.pr.fa345e8baa5e47e0b8bfed05da2b569e/' python3.spec
[2019-10-14T20:24:20.590Z] + fedpkg --release master srpm
[2019-10-14T20:24:21.515Z] Format argument module is deprecated in anongiturl. Please use "repo" instead.
[2019-10-14T20:24:21.515Z] Downloading Python-3.8.0.tar.xz
[2019-10-14T20:24:22.877Z] 
[2019-10-14T20:24:23.437Z] 
[2019-10-14T20:24:23.437Z] 
[2019-10-14T20:24:23.437Z] Wrote: /workDir/workspace/fedora-rawhide-pr-pipeline/python3/python3-3.8.0-1.fc32.pr.fa345e8baa5e47e0b8bfed05da2b569e.src.rpm
[2019-10-14T20:24:23.437Z] ++ rpmspec --queryformat '%{VERSION}\n' -q python3.spec
[2019-10-14T20:24:23.437Z] ++ head -n 1
[2019-10-14T20:24:23.437Z] + VERSION=3.8.0
[2019-10-14T20:24:23.437Z] + kinit -k -t /workDir/workspace/fedora-rawhide-pr-pipeline/fedora.keytab bpeck/jenkins-continuous-infra.apps.ci.centos.org@FEDORAPROJECT.ORG
[2019-10-14T20:24:24.799Z] + export FORCE_UNSAFE_CONFIGURE=1
[2019-10-14T20:24:24.799Z] + FORCE_UNSAFE_CONFIGURE=1
[2019-10-14T20:24:24.799Z] + koji build --wait --arch-override=x86_64 --scratch rawhide python3-3.8.0-1.fc32.pr.fa345e8baa5e47e0b8bfed05da2b569e.src.rpm
[2019-10-14T20:24:24.799Z] + tee /workDir/workspace/fedora-rawhide-pr-pipeline/logs/kojioutput.txt
[2019-10-14T20:24:32.894Z] Uploading srpm: python3-3.8.0-1.fc32.pr.fa345e8baa5e47e0b8bfed05da2b569e.src.rpm
[2019-10-14T20:24:32.894Z] 
[2019-10-14T20:24:32.894Z] Created task: 38295713
[2019-10-14T20:24:32.894Z] Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=38295713
[2019-10-14T20:24:32.894Z] Watching tasks (this may be safely interrupted)...
[2019-10-14T20:24:32.894Z] 38295713 build (rawhide, python3-3.8.0-1.fc32.pr.fa345e8baa5e47e0b8bfed05da2b569e.src.rpm): free
[2019-10-14T20:24:39.420Z] 38295713 build (rawhide, python3-3.8.0-1.fc32.pr.fa345e8baa5e47e0b8bfed05da2b569e.src.rpm): free -> open (buildvm-armv7-06.arm.fedoraproject.org)
[2019-10-14T20:24:39.420Z]   38295714 rebuildSRPM (noarch): open (buildvm-armv7-10.arm.fedoraproject.org)
[2019-10-14T20:28:45.828Z]   38295733 buildArch (python3-3.8.0-1.fc32.pr.fa345e8baa5e47e0b8bfed05da2b569e.src.rpm, x86_64): free
[2019-10-14T20:28:45.828Z]   38295714 rebuildSRPM (noarch): open (buildvm-armv7-10.arm.fedoraproject.org) -> closed
[2019-10-14T20:28:45.828Z]   1 free  1 open  1 done  0 failed
[2019-10-14T20:28:45.828Z]   38295733 buildArch (python3-3.8.0-1.fc32.pr.fa345e8baa5e47e0b8bfed05da2b569e.src.rpm, x86_64): free -> open (buildhw-07.phx2.fedoraproject.org)
[2019-10-14T20:31:07.206Z] 2019-10-14 20:30:52,431 [ERROR] koji: ServerOffline: database outage
[2019-10-14T20:31:07.206Z] + RPMBUILD_RC=0
[2019-10-14T20:31:07.206Z] + '[' 0 '!=' 0 ']'
[2019-10-14T20:31:07.206Z] + echo status=SUCCESS
[2019-10-14T20:31:07.206Z] + popd
[2019-10-14T20:31:07.206Z] /workDir/workspace/fedora-rawhide-pr-pipeline
[2019-10-14T20:31:07.206Z] ++ cat /workDir/workspace/fedora-rawhide-pr-pipeline/logs/kojioutput.txt
[2019-10-14T20:31:07.206Z] ++ awk '/Created task:/ { print $3 }'
[2019-10-14T20:31:07.206Z] + SCRATCHID=38295713
[2019-10-14T20:31:07.206Z] + echo koji_task_id=38295713
[2019-10-14T20:31:07.206Z] + rm -rf /workDir/workspace/fedora-rawhide-pr-pipeline/python3_repo
[2019-10-14T20:31:07.206Z] + mkdir -p /workDir/workspace/fedora-rawhide-pr-pipeline/python3_repo
[2019-10-14T20:31:07.206Z] + pushd /workDir/workspace/fedora-rawhide-pr-pipeline/python3_repo
[2019-10-14T20:31:07.206Z] /workDir/workspace/fedora-rawhide-pr-pipeline/python3_repo /workDir/workspace/fedora-rawhide-pr-pipeline
[2019-10-14T20:31:07.206Z] + for i in {1..5}
[2019-10-14T20:31:07.206Z] + koji download-build --arch=x86_64 --arch=src --arch=noarch --debuginfo --task-id 38295713
[2019-10-14T20:31:07.206Z] No associated builds for task 38295713
[2019-10-14T20:31:07.206Z] + koji download-task --arch=x86_64 --arch=src --arch=noarch --logs 38295713
[2019-10-14T20:31:07.206Z] Child task 38295733 has not finished yet.
[2019-10-14T20:31:07.206Z] + echo 'koji build download failed, attempt: 1/5'
[2019-10-14T20:31:07.206Z] koji build download failed, attempt: 1/5
[2019-10-14T20:31:07.206Z] + [[ 1 -lt 5 ]]
[2019-10-14T20:31:07.206Z] + sleep 10
[2019-10-14T20:31:07.206Z] + for i in {1..5}
[2019-10-14T20:31:07.206Z] + koji download-build --arch=x86_64 --arch=src --arch=noarch --debuginfo --task-id 38295713
[2019-10-14T20:31:07.206Z] No associated builds for task 38295713
[2019-10-14T20:31:07.206Z] + koji download-task --arch=x86_64 --arch=src --arch=noarch --logs 38295713
[2019-10-14T20:31:07.206Z] Child task 38295733 has not finished yet.
[2019-10-14T20:31:07.206Z] + echo 'koji build download failed, attempt: 2/5'
[2019-10-14T20:31:07.206Z] koji build download failed, attempt: 2/5
[2019-10-14T20:31:07.206Z] + [[ 2 -lt 5 ]]
[2019-10-14T20:31:07.206Z] + sleep 10
[2019-10-14T20:31:17.139Z] + for i in {1..5}
[2019-10-14T20:31:17.139Z] + koji download-build --arch=x86_64 --arch=src --arch=noarch --debuginfo --task-id 38295713
[2019-10-14T20:31:17.699Z] No associated builds for task 38295713
[2019-10-14T20:31:17.699Z] + koji download-task --arch=x86_64 --arch=src --arch=noarch --logs 38295713
[2019-10-14T20:31:18.623Z] Child task 38295733 has not finished yet.
[2019-10-14T20:31:18.624Z] + echo 'koji build download failed, attempt: 3/5'
[2019-10-14T20:31:18.624Z] koji build download failed, attempt: 3/5
[2019-10-14T20:31:18.624Z] + [[ 3 -lt 5 ]]
[2019-10-14T20:31:18.624Z] + sleep 10
[2019-10-14T20:31:28.547Z] + for i in {1..5}
[2019-10-14T20:31:28.547Z] + koji download-build --arch=x86_64 --arch=src --arch=noarch --debuginfo --task-id 38295713
[2019-10-14T20:31:29.470Z] No associated builds for task 38295713
[2019-10-14T20:31:29.470Z] + koji download-task --arch=x86_64 --arch=src --arch=noarch --logs 38295713
[2019-10-14T20:31:30.395Z] Child task 38295733 has not finished yet.
[2019-10-14T20:31:30.395Z] + echo 'koji build download failed, attempt: 4/5'
[2019-10-14T20:31:30.395Z] koji build download failed, attempt: 4/5
[2019-10-14T20:31:30.395Z] + [[ 4 -lt 5 ]]
[2019-10-14T20:31:30.395Z] + sleep 10
[2019-10-14T20:31:40.320Z] + for i in {1..5}
[2019-10-14T20:31:40.320Z] + koji download-build --arch=x86_64 --arch=src --arch=noarch --debuginfo --task-id 38295713
[2019-10-14T20:31:42.829Z] No associated builds for task 38295713
[2019-10-14T20:31:42.829Z] + koji download-task --arch=x86_64 --arch=src --arch=noarch --logs 38295713
[2019-10-14T20:31:43.103Z] Child task 38295733 has not finished yet.
[2019-10-14T20:31:43.103Z] + echo 'koji build download failed, attempt: 5/5'
[2019-10-14T20:31:43.103Z] koji build download failed, attempt: 5/5
[2019-10-14T20:31:43.103Z] + [[ 5 -lt 5 ]]
[2019-10-14T20:31:43.103Z] + exit 1
script returned exit code 1

yet the Koji task is still running.

From the log it appears that:

  1. something bad happens when watching the task (that is expected, stuff happens)
  2. the CI proceeds directly with downloading the results, without waiting for the build to finish

Examples of this:

https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-rawhide-pr-pipeline/detail/fedora-rawhide-pr-pipeline/2140/pipeline/
https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-rawhide-pr-pipeline/detail/fedora-rawhide-pr-pipeline/2139/pipeline/

Yes, we have restarted the database several times trying to isolate performance problems.

Perhaps you should sleep a few and retry when you see db outage? restarts are usually less than a minute and closer to 15seconds...

Perhaps the script should try to resume with watch-task before attempting download?

Also, perhaps that's a koji bug? shouldn't database off line be a exit 1? not a exit 0?

Totally should.

I don't know about the koji exit code, I assume in this case it only exits with 1 if the build fails...

Assuming it will continue to exit with 0 in case like this issue I created a PR for the pipeline to handle it.

https://github.com/CentOS-PaaS-SIG/upstream-fedora-pipeline/pull/193

I'm getting this now:

[2019-10-15T17:12:18.655Z] Starting koji-build
[2019-10-15T17:12:19.699Z] Topic is org.centos.prod.ci.pipeline.allpackages-pr.package.running
[2019-10-15T17:12:22.201Z] Sending message for job 'fedora-rawhide-pr-pipeline'.
[2019-10-15T17:12:27.224Z] JSON message body:
[2019-10-15T17:12:27.224Z] {"msg":{"branch":"master","build_id":"2167","build_url":"https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-rawhide-pr-pipeline/detail/fedora-rawhide-pr-pipeline/2167/pipeline/","ci_topic":"org.centos.prod.ci.pipeline.allpackages-pr.package.running","comment_id":32236,"commit_hash":"1098e8f3e93c4af1006cf42a18d47f78d69ad765","namespace":"rpms","nvr":"","original_spec_nvr":"","ref":"x86_64","repo":"python3","rev":"PR-141","scratch":true,"status":"SUCCESS","test_guidance":"''","username":"churchyard"},"msg_id":"2019-95f04116-b8de-421a-9707-542253ef1899","timestamp":0,"topic":"org.centos.prod.ci.pipeline.allpackages-pr.package.running"}
[2019-10-15T17:12:28.765Z] Currently in setDistBranch for rawhide
[2019-10-15T17:12:29.242Z] ++ curl --retry 10 --retry-delay 60 -s https://src.fedoraproject.org/rpms/fedora-release/raw/master/f/fedora-release.spec
[2019-10-15T17:12:29.248Z] ++ awk '/%define dist_version/ {print $3}'
[2019-10-15T17:12:30.614Z] + echo 32
[2019-10-15T17:12:31.476Z] + mkdir -p /workDir/workspace/fedora-rawhide-pr-pipeline/koji-build
[2019-10-15T17:12:40.576Z] + /tmp/koji_build_pr.sh
[2019-10-15T17:12:40.576Z] + '[' -z python3 ']'
[2019-10-15T17:12:40.576Z] + '[' -z 141 ']'
[2019-10-15T17:12:40.576Z] + '[' -z b18deb9c390a4d1ca4f5338c3ae5751c ']'
[2019-10-15T17:12:40.576Z] + '[' -z bpeck/jenkins-continuous-infra.apps.ci.centos.org@FEDORAPROJECT.ORG ']'
[2019-10-15T17:12:40.576Z] + '[' -z https://src.fedoraproject.org ']'
[2019-10-15T17:12:40.576Z] ++ pwd
[2019-10-15T17:12:40.576Z] + CURRENTDIR=/workDir/workspace/fedora-rawhide-pr-pipeline
[2019-10-15T17:12:40.576Z] + '[' /workDir/workspace/fedora-rawhide-pr-pipeline == / ']'
[2019-10-15T17:12:40.576Z] + KOJI_PARAMS=
[2019-10-15T17:12:40.576Z] + RPMDIR=/workDir/workspace/fedora-rawhide-pr-pipeline/python3_repo
[2019-10-15T17:12:40.576Z] + LOGDIR=/workDir/workspace/fedora-rawhide-pr-pipeline/logs
[2019-10-15T17:12:40.576Z] + rm -rf '/workDir/workspace/fedora-rawhide-pr-pipeline/logs/*'
[2019-10-15T17:12:40.576Z] + mkdir -p /workDir/workspace/fedora-rawhide-pr-pipeline/logs
[2019-10-15T17:12:40.576Z] + rm -rf python3
[2019-10-15T17:12:40.576Z] + git clone https://src.fedoraproject.org/rpms/python3.git
[2019-10-15T17:12:40.576Z] Cloning into 'python3'...
[2019-10-15T17:12:42.484Z] + '[' 0 '!=' 0 ']'
[2019-10-15T17:12:42.484Z] + pushd python3
[2019-10-15T17:12:42.484Z] /workDir/workspace/fedora-rawhide-pr-pipeline/python3 /workDir/workspace/fedora-rawhide-pr-pipeline
[2019-10-15T17:12:42.484Z] + git checkout master
[2019-10-15T17:12:42.484Z] Already on 'master'
[2019-10-15T17:12:42.484Z] Your branch is up to date with 'origin/master'.
[2019-10-15T17:12:42.484Z] + git fetch -fu origin refs/pull/141/head:pr
[2019-10-15T17:12:43.407Z] From https://src.fedoraproject.org/rpms/python3
[2019-10-15T17:12:43.407Z]  * [new ref]         refs/pull/141/head -> pr
[2019-10-15T17:12:43.407Z] + git -c 'user.name=Fedora CI' -c user.email=ci@lists.fedoraproject.org merge pr -m 'Fedora CI pipeline'
[2019-10-15T17:12:43.407Z] Updating 577a903..1098e8f
[2019-10-15T17:12:43.407Z] Fast-forward (no commit created; -m option ignored)
[2019-10-15T17:12:43.407Z]  python3.spec    | 7 ++++---
[2019-10-15T17:12:43.407Z]  tests/tests.yml | 3 +++
[2019-10-15T17:12:43.407Z]  2 files changed, 7 insertions(+), 3 deletions(-)
[2019-10-15T17:12:43.407Z] ++ rpm -q --define 'dist .fc32' --queryformat '%{name}-%{version}-%{release}\n' --specfile python3.spec
[2019-10-15T17:12:43.407Z] ++ head -n 1
[2019-10-15T17:12:43.407Z] + truenvr=python3-3.8.0~rc1-1.fc32
script returned exit code 141[2019-10-15T17:12:46.205Z] + mv logs /workDir/workspace/fedora-rawhide-pr-pipeline/koji-build/logs
[2019-10-15T17:12:47.208Z] script returned exit code 141
[2019-10-15T17:12:47.412Z] end of koji-build
script returned exit code 141[2019-10-15T17:12:48.978Z] Topic is org.centos.prod.ci.pipeline.allpackages-pr.package.complete
[2019-10-15T17:12:51.439Z] Sending message for job 'fedora-rawhide-pr-pipeline'.
[2019-10-15T17:12:56.482Z] JSON message body:
[2019-10-15T17:12:56.482Z] {"msg":{"branch":"master","build_id":"2167","build_url":"https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-rawhide-pr-pipeline/detail/fedora-rawhide-pr-pipeline/2167/pipeline/","ci_topic":"org.centos.prod.ci.pipeline.allpackages-pr.package.complete","comment_id":32236,"commit_hash":"1098e8f3e93c4af1006cf42a18d47f78d69ad765","namespace":"rpms","nvr":"","original_spec_nvr":"","ref":"x86_64","repo":"python3","rev":"PR-141","scratch":true,"status":"FAILURE","test_guidance":"''","username":"churchyard"},"msg_id":"2019-04d354eb-7f55-4ca8-ac3b-7a00633d4ce4","timestamp":0,"topic":"org.centos.prod.ci.pipeline.allpackages-pr.package.complete"}
[2019-10-15T17:12:57.641Z] Topic is org.centos.prod.ci.dist-git-pr.test.error
[2019-10-15T17:13:02.764Z] Sending message for job 'fedora-rawhide-pr-pipeline'.
[2019-10-15T17:13:07.783Z] JSON message body:
[2019-10-15T17:13:07.784Z] {"msg":{"artifact":{"comment_id":32236,"commit_hash":"1098e8f3e93c4af1006cf42a18d47f78d69ad765","id":141,"issuer":"churchyard","repository":"https://src.fedoraproject.org/rpms/python3","type":"pull-request","uid":"b18deb9c390a4d1ca4f5338c3ae5751c"},"contact":{"docs":"https://pagure.io/standard-test-roles","email":"ci@lists.fedoraproject.org","irc":"#fedora-ci","name":"fedora-ci","team":"fedora-ci","url":"https://jenkins-continuous-infra.apps.ci.centos.org/"},"error":{"reason":"Unknown execution error"},"generated_at":"2019-10-15T17:13:01.976Z","pipeline":{"build":"2167","id":"5ba3a446-920f-43c3-b806-2b71505b5bcf","name":"fedora-rawhide-pr-pipeline","stage":{"name":"koji-build"}},"run":{"log":"https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-pr-pipeline/2167/console","rebuild":"https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-pr-pipeline/2167/rebuild/parameterized","url":"https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-rawhide-pr-pipeline/detail/fedora-rawhide-pr-pipeline/2167/pipeline/"},"test":{"category":"static-analysis","namespace":"fedora-ci.dist-git-pr","type":"build"},"version":"0.2.1"},"msg_id":"2019-e4cf12f3-ca65-432f-97e7-3ec1f576b39c","timestamp":0,"topic":"org.centos.prod.ci.dist-git-pr.test.error"}
[2019-10-15T17:13:10.142Z] Error: Exception from koji-build:
[2019-10-15T17:13:10.515Z] script returned exit code 141
[2019-10-15T17:13:11.968Z] + mkdir -p podInfo
[2019-10-15T17:13:14.438Z] Archiving artifacts
[2019-10-15T17:13:14.876Z] Archiving artifacts
[2019-10-15T17:13:15.152Z] Topic is org.centos.prod.ci.pipeline.allpackages-pr.complete
[2019-10-15T17:13:16.116Z] Sending message for job 'fedora-rawhide-pr-pipeline'.
[2019-10-15T17:13:21.126Z] JSON message body:
[2019-10-15T17:13:21.126Z] {"msg":{"branch":"master","build_id":"2167","build_url":"https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-rawhide-pr-pipeline/detail/fedora-rawhide-pr-pipeline/2167/pipeline/","ci_topic":"org.centos.prod.ci.pipeline.allpackages-pr.complete","comment_id":32236,"commit_hash":"1098e8f3e93c4af1006cf42a18d47f78d69ad765","namespace":"rpms","nvr":"","original_spec_nvr":"","ref":"x86_64","repo":"python3","rev":"PR-141","scratch":true,"status":"FAILURE","test_guidance":"''","username":"churchyard"},"msg_id":"2019-5fabcbbc-5092-455c-8b6d-f6b50c56afb4","timestamp":0,"topic":"org.centos.prod.ci.pipeline.allpackages-pr.complete"}

https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-rawhide-pr-pipeline/detail/fedora-rawhide-pr-pipeline/2167/pipeline

That stopped for a while yesterday, but is back now.

hmm, we are running the script with pipefail to catch error when running a command and | tee, but this might cause some problem when we have pipefail set and use | head.

Proposed fix with https://github.com/CentOS-PaaS-SIG/upstream-fedora-pipeline/pull/195

I think this issue is fixed now.

Metadata Update from @bgoncalv:
- Issue status updated to: Closed (was: Open)
4 years ago

Thanks!

The original problem keeps happening:

[2019-10-17T18:33:26.196Z] + koji build --wait --arch-override=x86_64 --scratch rawhide pypy3-7.2.0-1.fc32.pr.5e3c1d2569854a3c8caaafba1a3891c9.src.rpm
[2019-10-17T18:33:30.352Z] Uploading srpm: pypy3-7.2.0-1.fc32.pr.5e3c1d2569854a3c8caaafba1a3891c9.src.rpm
[2019-10-17T18:33:30.352Z] 
[2019-10-17T18:33:30.352Z] Created task: 38351595
[2019-10-17T18:33:30.352Z] Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=38351595
[2019-10-17T18:33:30.352Z] Watching tasks (this may be safely interrupted)...
[2019-10-17T18:33:30.353Z] 38351595 build (rawhide, pypy3-7.2.0-1.fc32.pr.5e3c1d2569854a3c8caaafba1a3891c9.src.rpm): free
[2019-10-17T18:33:36.876Z] 38351595 build (rawhide, pypy3-7.2.0-1.fc32.pr.5e3c1d2569854a3c8caaafba1a3891c9.src.rpm): free -> open (buildvm-03.phx2.fedoraproject.org)
[2019-10-17T18:33:36.876Z]   38351596 rebuildSRPM (noarch): open (buildvm-ppc64le-09.ppc.fedoraproject.org)
[2019-10-17T18:34:15.516Z]   38351596 rebuildSRPM (noarch): open (buildvm-ppc64le-09.ppc.fedoraproject.org) -> closed
[2019-10-17T18:34:15.516Z]   0 free  1 open  1 done  0 failed
[2019-10-17T18:34:37.387Z]   38351597 buildArch (pypy3-7.2.0-1.fc32.pr.5e3c1d2569854a3c8caaafba1a3891c9.src.rpm, x86_64): open (buildvm-20.phx2.fedoraproject.org)
[2019-10-17T19:54:00.477Z] 2019-10-17 19:53:52,143 [ERROR] koji: ServerOffline: database outage
script returned exit code 1[2019-10-17T19:54:02.405Z] + mv logs /workDir/workspace/fedora-rawhide-pr-pipeline/koji-build/logs
[2019-10-17T19:54:02.777Z] script returned exit code 1
[2019-10-17T19:54:02.888Z] end of koji-build
script returned exit code 1[2019-10-17T19:54:03.757Z] Topic is org.centos.prod.ci.pipeline.allpackages-pr.package.complete

https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-rawhide-pr-pipeline/detail/fedora-rawhide-pr-pipeline/2190/pipeline/

Metadata Update from @churchyard:
- Issue status updated to: Open (was: Closed)
4 years ago

FYI, I did restart the database today. It was down for about 5 seconds. :(

hmmm, in this case koji actually exited with error so the pipeline assumed this koji build failed.
@mvadkert do you think we should update the pipeline code to not fail in this case or as the database was restarted it is fair for the build to fail?

I believe it is a valid failure

The Koji build kept running. The koji build command with watching may fail for various reasons (even on connection trouble). I've learned trough experience that relying on its exit code is not a good idea.

IMHO the robust thing to do is to ignore koji's return code entirely and proceed with asking for the build status.
Edited 4 years ago by churchyard

proposed PR to not check the koji exit status in this case...
https://github.com/CentOS-PaaS-SIG/upstream-fedora-pipeline/pull/196

I assume this is fixed. Would reopen if it ever hits me again. Thank you!

Metadata Update from @churchyard:
- Issue status updated to: Closed (was: Open)
4 years ago

Login to comment on this ticket.

Metadata
None
None
None
None
Medium
Powered by Pagure 5.13.3
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.