PR#474: Use a single thread pool while watching module builds - rpkg

rpkg

#474 Use a single thread pool while watching module builds

Merged 4 years ago by onosek. Opened 4 years ago by csomh.

csomh/rpkg reuse-threads into master

Use a single thread pool while watching module builds

Hunor Csomortáni • 4 years ago

a99a9a4

pyrpkg/__init__.py

file modified

+4 -1

		`@@ -3826,8 +3826,11 @@`
		`return stats`

		`def get_watched_module_builds(self, build_ids):`
		`+ # Limit the number of threads to an arbitrary 8,`
		`+ # to avoid starting too many threads when running in a`
		`+ # container in the cloud.`
		`+ pool = ThreadPool(min(8, len(build_ids) or 1))`
		`while True:`
		`- pool = ThreadPool()`
		`module_builds = pool.map(self.module_get_build, build_ids)`

		`for module_build in module_builds:`

csomh commented 4 years ago

Calling the Pool() constructor on each iteration leaks the threads,
leading to a "can't start new thread" error during long waits.

This can be aggravated by the constructor being called without an
argument, which by default uses the systems CPU count to create the
pool. When running in a container in a cloud environment this number can
be unexpectedly large (for example, 64).

Reuse the pool and limit the number of threads in the pool.

Signed-off-by: Hunor Csomortáni csomh@redhat.com

csomh commented 4 years ago

@cqi @mprahl ptal

Currently this is a blocker to set up a Jenkins job to run tests for MBS, as the Jenkins instance I'm using creates the node in OpenShift. Under these circumstances, the number of threads created by the "watch" command/option reaches the system limit within 5 minutes or so.