#350 When starting a build, recover from server startup error on server side
Opened 4 years ago by mmalina. Modified 4 years ago

This is something we experienced a few times with rhpkg. When starting a build using "rhpkg build", the command would fail after a few minutes with this error message:
Could not execute build: server startup error

But if I checked the Brew build afterwards, it was fine and Closed.

I discussed this with Tomas Kopecek and here's what he told me:
It's a restart of the hub. The builders try again later and if the build is already running, they will let it finish.

But apparently the client cannot recover from this. It would be nice if rhpkg could retry after some delay instead of failing immediately.

This is the relevant log snippet of the recent instance of this:
+ ./rebuild_package.py -e stage-rhel8 -v
2019-08-26T18:32:23+00:00 - Starting the building. This usually takes around 5 minutes.
2019-08-26T18:32:23+00:00 - ['rhpkg-stage', '--path', 'release-e2e-test', 'build']
2019-08-26T18:32:23+00:00 - rhpkg-stage --path release-e2e-test build
2019-08-26T18:32:27+00:00 - stdout:Building release-e2e-test-1.0.11216-1.el8 for release-e2e-test-1.0-rhel-8-candidate
2019-08-26T18:38:30+00:00 - stderr:Could not execute build: server startup error
2019-08-26T18:38:30+00:00 - stdout:Created task: 40866490
2019-08-26T18:38:31+00:00 - stdout:Task info: https://brewweb.stage.engineering.redhat.com/brew/taskinfo?taskID=40866490
2019-08-26T18:38:31+00:00 - stdout:Watching tasks (this may be safely interrupted)...
2019-08-26T18:38:31+00:00 - stdout:40866490 build (release-e2e-test-1.0-rhel-8-candidate, /rpms/release-e2e-test:9b62e12e8c0e15bfb2e583047b2fdb9ebca2328e): free
2019-08-26T18:38:31+00:00 - stdout:40866490 build (release-e2e-test-1.0-rhel-8-candidate, /rpms/release-e2e-test:9b62e12e8c0e15bfb2e583047b2fdb9ebca2328e): free -> open (x86-stage-vm-07.build.eng.bos.redhat.com)
2019-08-26T18:38:31+00:00 - stdout:40866491 buildSRPMFromSCM (/rpms/release-e2e-test:9b62e12e8c0e15bfb2e583047b2fdb9ebca2328e): free
2019-08-26T18:38:31+00:00 - stdout:40866491 buildSRPMFromSCM (/rpms/release-e2e-test:9b62e12e8c0e15bfb2e583047b2fdb9ebca2328e): free -> closed
2019-08-26T18:38:31+00:00 - stdout:0 free 1 open 1 done 0 failed
2019-08-26T18:38:31+00:00 - stdout:40866492 buildArch (release-e2e-test-1.0.11216-1.el8.src.rpm, x86_64): free
2019-08-26T18:38:31+00:00 - stdout:40866492 buildArch (release-e2e-test-1.0.11216-1.el8.src.rpm, x86_64): free -> open (x86-stage-vm-07.build.eng.bos.redhat.com)
2019-08-26T18:38:31+00:00 - stdout:40866493 tagBuild (noarch): free
2019-08-26T18:38:31+00:00 - stdout:40866492 buildArch (release-e2e-test-1.0.11216-1.el8.src.rpm, x86_64): open (x86-stage-vm-07.build.eng.bos.redhat.com) -> closed
2019-08-26T18:38:31+00:00 - stdout:1 free 1 open 2 done 0 failed
2019-08-26T18:38:31+00:00 - Return code: 1
2019-08-26T18:38:31+00:00 - Buid task ID: 40866490
Traceback (most recent call last):
File "./rebuild_package.py", line 144, in <module>
main()
File "./rebuild_package.py", line 139, in main
builder.build()
File "./rebuild_package.py", line 95, in build
raise RuntimeError(stderr)
RuntimeError: Could not execute build: server startup error


Login to comment on this ticket.

Metadata