PR#1412: backend: indefinitely retry workers' talk to frontend - copr/copr

copr / copr

#1412 backend: indefinitely retry workers' talk to frontend

Merged 3 years ago by msuchy. Opened 3 years ago by praiskup.

Unknown source try-frontend-indefinitely-from-background-job into master

backend: indefinitely retry workers' talk to frontend

Pavel Raiskup • 3 years ago

eb663eb

backend/copr_backend/background_worker.py

file modified

+2 -1

		`@@ -164,7 +164,8 @@`
		`# setup logging early, to have as complete logs as possible`
		`self._switch_logger_to_redis()`

		`- self.frontend_client = FrontendClient(self.opts, self.log)`
		`+ self.frontend_client = FrontendClient(self.opts, self.log,`
		`+ try_indefinitely=True)`

		`try:`
		`self.handle_task()`

praiskup commented 3 years ago

... again. I accidentally dropped the FrontendClient flag when moving
things around in b7f71e1.

The weird effect of this bug was when we had a small FE downtime today.
The backend workers finished several builds for iucar/cran project - and
then failed to mark the jobs as finished on FE due to the FE downtime.
The background worker ended, the corresponding process doesn't exist
anymore -- but the builds still look like "running" on FE.

Since the project is still receiving many other build requests (at the
time of writing this patch), and all those need source builds first
(srpm builds are prioritized over normal RPM builds) those already
finished builds are still around in "running" state (several hours
now). They are waiting till they bubble up in the queue again.

rebased onto f814ee214dc508871d09c00f34093754e2fb6d41

3 years ago

Metadata Update from @praiskup:
- Pull-request tagged with: release-blocker

3 years ago