#54 spectool: replace all-in-one-go download with streaming stuff
Closed 2 years ago by ngompa. Opened 2 years ago by decathorpe.

The new spectool implementation naively does response = requests.get(url) which stores everything in memory before writing things to disk, which could cause problems for large downloads, I guess ... and while we're fixing that, maybe add a progress bar back too?

request_download seems to be able to solve both of those problems:
https://github.com/takluyver/requests_download/blob/master/requests_download.py


request_download is a leftpad of Python, but fortunately, we have it packaged in Fedora

while touching it, I think using asyncio to download all sources at once is the new Cool Thing.

I googlead bit for python async progress bar, found this https://gist.github.com/khardix/e8b7e87f9e2e1c9c8e88e3774c52ec77 -- it has no license attached but it is written by @jstanek

Metadata Update from @ngompa:
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata