PR#1993: Always use stream=True when iterating over a request - koji

koji

#1993 Always use stream=True when iterating over a request

Merged 4 years ago by tkopecek. Opened 4 years ago by churchyard.

churchyard/koji requests_stream into master

Always use stream=True when iterating over a request

Miro Hrončok • 4 years ago

2c2c5b3

koji/__init__.py

file modified

+1 -1

		`@@ -1677,7 +1677,7 @@`
		`url = "%s/%s" % (topurl, relpath)`
		`fo = tempfile.TemporaryFile(dir=tempdir)`
		`try:`
		`- resp = requests.get(url)`
		`+ resp = requests.get(url, stream=True)`
		`for chunk in resp.iter_content(chunk_size=8192):`
		`fo.write(chunk)`
		`finally:`

koji/tasks.py

file modified

+1 -1

		`@@ -479,7 +479,7 @@`
		`self.logger.debug("Downloading %s", relpath)`
		`url = "%s/%s" % (self.options.topurl, relpath)`
		`try:`
		`- resp = requests.get(url)`
		`+ resp = requests.get(url, stream=True)`
		`if not os.path.exists(os.path.dirname(fn)):`
		`os.makedirs(os.path.dirname(fn))`
		`with open(fn, 'wb') as fdst:`

churchyard commented 4 years ago

Fixes https://pagure.io/releng/issue/9226

churchyard commented 4 years ago

Note that it should be possible to use shutil.copyfileobj instead of iterating over chunks, however I wanted to make this fix as self contained as possible.