PR#1318: Allow configuring the number of parallel submissions to Koji - fm-orchestrator

fm-orchestrator

#1318 Allow configuring the number of parallel submissions to Koji

Merged 4 years ago by mprahl. Opened 4 years ago by mprahl.

parallel-submissions into master

Allow configuring the number of parallel submissions to Koji

mprahl • 4 years ago

50f0a60

module_build_service/config.py

file modified

+14 -1

		`@@ -653,7 +653,13 @@`
		`"x, y, and z (represents the version). The third value is an optional template "`
		`"string that represent the Product Pages release for major releases "`
		`"(e.g. 8.0.0). After the first match, the rest will be ignored."`
		`- }`
		`+ },`
		`+ "num_threads_for_build_submissions": {`
		`+ "type": int,`
		`+ "default": 5,`
		`+ "desc": "The number of threads when submitting component builds to an external build "`
		`+ "system.",`
		`+ },`
		`}`

		`def __init__(self, conf_section_obj):`
		`@@ -907,3 +913,10 @@`
		`)`

		`self._product_pages_module_streams = d`
		`+`
		`+ def _setifok_num_threads_for_build_submissions(self, i):`
		`+ if not isinstance(i, int):`
		`+ raise TypeError("NUM_THREADS_FOR_BUILD_SUBMISSIONS needs to be an int")`
		`+ if i < 1:`
		`+ raise ValueError("NUM_THREADS_FOR_BUILD_SUBMISSIONS must be >= 1")`
		`+ self._num_threads_for_build_submissions = i`

module_build_service/utils/batches.py

file modified

+1 -3

		`@@ -150,9 +150,7 @@`
		`components_to_build.append(c)`

		`# Start build of components in this batch.`
		`- max_workers = 1`
		`- if config.num_concurrent_builds > 0:`
		`- max_workers = config.num_concurrent_builds`
		`+ max_workers = config.num_threads_for_build_submissions`
		`with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:`
		`futures = {`
		`executor.submit(start_build_component, builder, c): c for c in components_to_build`

mprahl commented 4 years ago

When NUM_CONCURRENT_BUILDS is 0, then multi-threading is disabled when submitting builds to Koji. This is not acceptable, so this commit makes the number of threads configurable.