PR#1172: backend: separate frontend_client for each Worker process - copr/copr

copr / copr

#1172 backend: separate frontend_client for each Worker process

Merged 4 years ago by praiskup. Opened 4 years ago by praiskup.

Unknown source separate-fc-for-worker into master

backend: separate frontend_client for each Worker process

Pavel Raiskup • 4 years ago

57aed1d

backend/backend/daemons/build_dispatcher.py

file modified

-1

		`@@ -131,7 +131,6 @@`
		`def start_worker(self, vm, job, reattach=False):`
		`worker = Worker(`
		`opts=self.opts,`
		`- frontend_client=self.frontend_client,`
		`vm_manager=self.vm_manager,`
		`worker_id=self.next_worker_id,`
		`vm=vm, job=job, reattach=reattach`

backend/backend/daemons/worker.py

file modified

+9 -4

		`@@ -12,6 +12,7 @@`
		`from ..constants import BuildStatus, build_log_format`
		`from ..helpers import register_build_result, get_redis_logger, \`
		`local_file_logger, run_cmd, pkg_name_evr`
		`+ from ..frontend import FrontendClient`

		`from ..msgbus import (`
		`MsgBusStomp,`
		`@@ -26,18 +27,20 @@`
		`class Worker(multiprocessing.Process):`
		`msg_buses = []`

		`- def __init__(self, opts, frontend_client, vm_manager, worker_id, vm, job, reattach=False):`
		`+ def __init__(self, opts, vm_manager, worker_id, vm, job, reattach=False):`
		`multiprocessing.Process.__init__(self, name="worker-{}".format(worker_id))`

		`+ # safe variables to be inherited through fork() call`
		`self.opts = opts`
		`- self.frontend_client = frontend_client`
		`self.vm_manager = vm_manager`
		`self.worker_id = worker_id`
		`self.vm = vm`
		`self.job = job`
		`self.reattach = reattach`

		`- self.log = get_redis_logger(self.opts, self.name, "worker")`
		`+ # for the sake of readability, those are defined in run()`
		`+ self.log = None`
		`+ self.frontend_client = None`

		`@property`
		`def name(self):`
		`@@ -357,9 +360,11 @@`
		`setproctitle(title)`

		`def run(self):`
		`+ self.log = get_redis_logger(self.opts, self.name, "worker")`
		`+ self.frontend_client = FrontendClient(self.opts, self.log,`
		`+ try_indefinitely=True)`
		`self.log.info("Starting worker")`
		`self.init_buses()`
		`- self.frontend_client.try_indefinitely = True`

		`try:`
		`self.do_job(self.job)`

backend/backend/frontend.py

file modified

+2 -2

		`@@ -22,10 +22,10 @@`
		`# do we block the main daemon process?`
		`try_indefinitely = False`

		`- def __init__(self, opts, logger=None):`
		`- super(FrontendClient, self).__init__()`
		`+ def __init__(self, opts, logger=None, try_indefinitely=False):`
		`self.frontend_url = "{}/backend".format(opts.frontend_base_url)`
		`self.frontend_auth = opts.frontend_auth`
		`+ self.try_indefinitely = try_indefinitely`

		`self.msg = None`
		`self.logger = logger`

backend/tests/daemons/test_dispatcher.py

file modified

-1

		`@@ -171,7 +171,6 @@`
		`def init_worker(self):`
		`self.worker = Worker(`
		`opts=self.opts,`
		`- frontend_client=self.frontend_client,`
		`vm_manager=None,`
		`worker_id=None,`
		`vm=VmDescriptor("1.1.1.1", "vm_name", 3, "ready"),`

praiskup commented 4 years ago

Because we shared FrontendClient object with BuildDispatcher, we for
some reason weren't able to communicate with RedisLogHandler and some
log entries were lost.

praiskup commented on line 5 of backend/backend/frontend.py 4 years ago

this is just irrelevant cleanup, sorry, lemme know I you don't want to have this in one commit

praiskup commented 4 years ago

[copr-build]

praiskup commented 4 years ago

[copr-build]

praiskup commented 4 years ago

This isn't correct I think. We should create FrontendClient and log in run() method I guess.

Metadata Update from @praiskup:
- Pull-request tagged with: needs-work

4 years ago

praiskup commented 4 years ago

[copr-build]