#3375 `koji download-task` should notice if downloading one package fails and retry
Closed: Fixed 2 months ago by tkopecek. Opened 3 months ago by thl.

I regularly download scratch builds with koji --noauth download-task and sometimes some packages (often big ones) are broken. Happens way more often since a few days for some reason. Turn's out the client doesn't notice if the transfer closes for some reason and just continues as if the download was successful:

Downloading [15/29]: kernel-5.17.11-150.vanilla.1.fc34.src.rpm
[=======================             ]  64%  80.74 MiB

Happens with two different machines here. Downloading with curl also sometimes fails with a message like this:

curl: (18) transfer closed with 536692769 bytes remaining to read

Before this happens the download often becomes quite slow – and I guess the server then is killing the connection for one reason or another.


Is it on Fedora's koji instance?

Metadata Update from @tkopecek:
- Custom field Size adjusted to None

3 months ago

And does rerunning (without deleting partial files) fix it? Or is file still broken? In other words is that file just truncated or contains garbled data?

Is it on Fedora's koji instance?
Yes.

And does rerunning (without deleting partial files) fix it?

Need to try, never tried to do that (didn't even think it was possible).

Fun fact: I tried to reproduce it before writing this comment, but seem at this time of the day it's harder to reproduce (maybe I'm wrong with that, but yesterday around this time the problem was less-frequent, too)

Need to try, never tried to do that (didn't even think it was possible).

I'll wait for non/confirming that. We've added support to continue interrupted downloads. So if it works it is more about adding some warning message with hint that rerunning the command should fix it. Otherwise we need to look into it deeper.

We've added support to continue interrupted downloads.

In which version (I'm running koji-1.28.1-1.fc36.noarch)? Just tried to force things into this scenario, but it seems all files are downloaded again, even those already completely downloaded.

Is it long time (1.21) https://pagure.io/koji/issue/1880

Just tried: continuing interrupted downloads works for me with koji download-build, but not with koji download-task

(BTW: I just join #koji as knurd in case that makes things easier to discuss)

ah, makes sense. In download-build we know that it is the rpm, so we can look to the db for expected size. In download-task it is treated as a generic file without known size. Anyway, we could send HEAD request before and get expected size from the download server.

Metadata Update from @tkopecek:
- Issue set to the milestone: 1.30

3 months ago

Metadata Update from @tkopecek:
- Issue tagged with: testing-ready

2 months ago

PR 3385

Applied it manually and this is a great help for my case, thx. I'd have preferred if the client would directly retry continuing the download when it notices the connection was aborted, but whatever. :-D

PR 3385

Applied it manually and this is a great help for my case, thx. I'd have preferred if the client would directly retry continuing the download when it notices the connection was aborted, but whatever. :-D

@thl Added to the PR.

Metadata Update from @mfilip:
- Issue tagged with: testing-done

2 months ago

Login to comment on this ticket.

Metadata
Related Pull Requests
  • #3385 Merged 2 months ago
  • #3382 Merged 13 days ago