Learn more about these different git repos.
Other Git URLs
We have an issue within Fedora/RISCV Koji infrastructure (separated infrastructure hosted outside of Red Hat datacenters).
fedpkg sources no longer works for the larger sources and thus our buildSRPMFromSCM are failing to generate SRPMs. Example "large source packages" would be kernel, kernel-tools, and similar.
It seems the download must succeed in 15000 or 14999 milliseconds. It attempts 5 times after that it fails. Note that it almost manages to download sources, throws away almost downloaded content before attempting again. I couldn't find in curl, pycurl, fedpkg and rpkg where exactly 15000 milliseconds comes from, but I assume it's dynamically calculated as it's not exactly 15000 milliseconds each time.
I have pulled the latest curl, pycrurl, fedpkg, rpkg from the Fedora 37/Rawhide into Fedora/RISCV (which is still at F33), but that had no affect.
Looking at the main Koji the download speeds are fast (I have seen 75+MB/s), and I haven't seen download take more than 2-3 seconds to happen. I assume that's because everything is within the same datacenter with almost direct path to look aside cache, 1-10GB network connectivity.
I am pulling sources at 8-10M/s and that seems not fast enough to do that within ~15 seconds.
Could we set the timeout (CURLOPT_TIMEOUT or CURLOPT_TIMEOUT_MS) to something more reasonable than ~15 seconds? I have impression this should be done in rpkg package. Probably in pyrpkg/lookaside.py:download.
pyrpkg/lookaside.py:download
Seems that 10 years ago it used to have (in comments) for fetching the sources:
[..] curl.setopt(pycurl.CONNECTTIMEOUT, 30) curl.setopt(pycurl.TIMEOUT, 300) [..]
I haven't tested it yet by rebuilding rpkg (but I plan to do it).
[..] Finish: chroot init INFO: Running in chroot: ['fedpkg', 'sources'] Start: chroot ['fedpkg', 'sources'] % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 123M 0 83067 0 0 99k 0 0:21:17 --:--:-- 0:21:17 99k 3 123M 3 5004k 0 0 2892k 0 0:00:43 0:00:01 0:00:42 2892k 10 123M 10 12.8M 0 0 4818k 0 0:00:26 0:00:02 0:00:24 4817k 16 123M 16 21.0M 0 0 5776k 0 0:00:21 0:00:03 0:00:18 5776k 23 123M 23 29.0M 0 0 6288k 0 0:00:20 0:00:04 0:00:16 6288k 30 123M 30 37.4M 0 0 6690k 0 0:00:18 0:00:05 0:00:13 7785k 36 123M 36 45.5M 0 0 6924k 0 0:00:18 0:00:06 0:00:12 8319k 43 123M 43 53.5M 0 0 7099k 0 0:00:17 0:00:07 0:00:10 8345k 49 123M 49 61.5M 0 0 7224k 0 0:00:17 0:00:08 0:00:09 8304k 56 123M 56 69.6M 0 0 7329k 0 0:00:17 0:00:09 0:00:08 8313k 62 123M 62 77.6M 0 0 7408k 0 0:00:17 0:00:10 0:00:07 8231k 69 123M 69 85.6M 0 0 7477k 0 0:00:16 0:00:11 0:00:05 8222k 75 123M 75 93.7M 0 0 7539k 0 0:00:16 0:00:12 0:00:04 8219k 82 123M 82 101M 0 0 7581k 0 0:00:16 0:00:13 0:00:03 8206k 88 123M 88 109M 0 0 7629k 0 0:00:16 0:00:14 0:00:02 8214k 90 123M 90 111M 0 0 7639k 0 0:00:16 0:00:14 0:00:02 8221k curl: (28) Operation timed out after 14999 milliseconds with 117335163 out of 129790264 bytes received Warning: Problem : timeout. Will retry in 1 seconds. 5 retries left. Throwing away 117335163 bytes 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 123M 0 36620 0 0 59005 0 0:36:39 --:--:-- 0:36:39 59064 3 123M 3 4995k 0 0 3181k 0 0:00:39 0:00:01 0:00:38 3182k 10 123M 10 13.3M 0 0 5336k 0 0:00:23 0:00:02 0:00:21 5336k 17 123M 17 21.8M 0 0 6270k 0 0:00:20 0:00:03 0:00:17 6271k 24 123M 24 30.4M 0 0 6814k 0 0:00:18 0:00:04 0:00:14 6815k 31 123M 31 38.8M 0 0 7141k 0 0:00:17 0:00:05 0:00:12 8029k 38 123M 38 47.2M 0 0 7366k 0 0:00:17 0:00:06 0:00:11 8680k 45 123M 45 55.7M 0 0 7536k 0 0:00:16 0:00:07 0:00:09 8667k 51 123M 51 64.1M 0 0 7664k 0 0:00:16 0:00:08 0:00:08 8659k 58 123M 58 72.5M 0 0 7763k 0 0:00:16 0:00:09 0:00:07 8630k 65 123M 65 81.0M 0 0 7853k 0 0:00:16 0:00:10 0:00:06 8646k 72 123M 72 89.4M 0 0 7918k 0 0:00:16 0:00:11 0:00:05 8645k 79 123M 79 98.0M 0 0 7990k 0 0:00:15 0:00:12 0:00:03 8680k 86 123M 86 106M 0 0 8038k 0 0:00:15 0:00:13 0:00:02 8679k 92 123M 92 114M 0 0 8077k 0 0:00:15 0:00:14 0:00:01 8676k 95 123M 95 118M 0 0 8092k 0 0:00:15 0:00:14 0:00:01 8664k curl: (28) Operation timed out after 15000 milliseconds with 124301148 out of 129790264 bytes received Warning: Problem : timeout. Will retry in 2 seconds. 4 retries left. Throwing away 124301148 bytes 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2 123M 2 3324k 0 0 2374k 0 0:00:53 0:00:01 0:00:52 2374k 9 123M 9 11.8M 0 0 5042k 0 0:00:25 0:00:02 0:00:23 5043k 16 123M 16 20.3M 0 0 6115k 0 0:00:20 0:00:03 0:00:17 6115k 23 123M 23 28.8M 0 0 6707k 0 0:00:18 0:00:04 0:00:14 6707k 30 123M 30 37.3M 0 0 7079k 0 0:00:17 0:00:05 0:00:12 7651k 37 123M 37 45.8M 0 0 7329k 0 0:00:17 0:00:06 0:00:11 8717k 43 123M 43 54.3M 0 0 7516k 0 0:00:16 0:00:07 0:00:09 8704k 50 123M 50 62.8M 0 0 7657k 0 0:00:16 0:00:08 0:00:08 8707k 57 123M 57 71.2M 0 0 7765k 0 0:00:16 0:00:09 0:00:07 8697k 64 123M 64 79.8M 0 0 7858k 0 0:00:16 0:00:10 0:00:06 8699k 71 123M 71 88.2M 0 0 7931k 0 0:00:15 0:00:11 0:00:04 8701k 78 123M 78 96.7M 0 0 7989k 0 0:00:15 0:00:12 0:00:03 8689k 85 123M 85 105M 0 0 8042k 0 0:00:15 0:00:13 0:00:02 8689k 91 123M 91 113M 0 0 8093k 0 0:00:15 0:00:14 0:00:01 8710k 96 123M 96 118M 0 0 8117k 0 0:00:15 0:00:14 0:00:01 8704k curl: (28) Operation timed out after 14999 milliseconds with 124683388 out of 129790264 bytes received Warning: Problem : timeout. Will retry in 4 seconds. 3 retries left. Throwing away 124683388 bytes 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 1 123M 1 2031k 0 0 1650k 0 0:01:16 0:00:01 0:01:15 1650k 8 123M 8 10.5M 0 0 4834k 0 0:00:26 0:00:02 0:00:24 4835k 15 123M 15 19.0M 0 0 6047k 0 0:00:20 0:00:03 0:00:17 6047k 22 123M 22 27.5M 0 0 6677k 0 0:00:18 0:00:04 0:00:14 6677k 29 123M 29 36.0M 0 0 7063k 0 0:00:17 0:00:05 0:00:12 7460k 36 123M 36 44.7M 0 0 7352k 0 0:00:17 0:00:06 0:00:11 8754k 42 123M 42 53.2M 0 0 7536k 0 0:00:16 0:00:07 0:00:09 8742k 49 123M 49 61.6M 0 0 7675k 0 0:00:16 0:00:08 0:00:08 8726k 56 123M 56 70.1M 0 0 7785k 0 0:00:16 0:00:09 0:00:07 8723k 63 123M 63 78.6M 0 0 7871k 0 0:00:16 0:00:10 0:00:06 8718k 70 123M 70 87.1M 0 0 7943k 0 0:00:15 0:00:11 0:00:04 8680k 77 123M 77 95.5M 0 0 8001k 0 0:00:15 0:00:12 0:00:03 8673k 84 123M 84 104M 0 0 8049k 0 0:00:15 0:00:13 0:00:02 8667k 90 123M 90 112M 0 0 8089k 0 0:00:15 0:00:14 0:00:01 8651k 96 123M 96 118M 0 0 8116k 0 0:00:15 0:00:14 0:00:01 8642k curl: (28) Operation timed out after 15000 milliseconds with 124665699 out of 129790264 bytes received Warning: Problem : timeout. Will retry in 8 seconds. 2 retries left. Throwing away 124665699 bytes 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 123M 0 837k 0 0 781k 0 0:02:42 0:00:01 0:02:41 781k 6 123M 6 8657k 0 0 4202k 0 0:00:30 0:00:02 0:00:28 4202k 13 123M 13 16.2M 0 0 5434k 0 0:00:23 0:00:03 0:00:20 5435k 19 123M 19 23.9M 0 0 6038k 0 0:00:20 0:00:04 0:00:16 6038k 25 123M 25 31.6M 0 0 6410k 0 0:00:19 0:00:05 0:00:14 6559k 31 123M 31 39.3M 0 0 6655k 0 0:00:19 0:00:06 0:00:13 7917k 38 123M 38 47.1M 0 0 6833k 0 0:00:18 0:00:07 0:00:11 7917k 44 123M 44 54.7M 0 0 6958k 0 0:00:18 0:00:08 0:00:10 7891k 50 123M 50 62.4M 0 0 7063k 0 0:00:17 0:00:09 0:00:08 7896k 56 123M 56 70.3M 0 0 7160k 0 0:00:17 0:00:10 0:00:07 7918k 63 123M 63 78.0M 0 0 7229k 0 0:00:17 0:00:11 0:00:06 7923k 69 123M 69 85.8M 0 0 7285k 0 0:00:17 0:00:12 0:00:05 7923k 75 123M 75 93.4M 0 0 7329k 0 0:00:17 0:00:13 0:00:04 7928k 81 123M 81 101M 0 0 7363k 0 0:00:17 0:00:14 0:00:03 7907k 87 123M 87 108M 0 0 7397k 0 0:00:17 0:00:14 0:00:03 7880k curl: (28) Operation timed out after 15000 milliseconds with 113614675 out of 129790264 bytes received Warning: Problem : timeout. Will retry in 16 seconds. 1 retries left. Throwing away 113614675 bytes 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 123M 0 387k 0 0 401k 0 0:05:15 --:--:-- 0:05:15 401k 6 123M 6 7779k 0 0 4071k 0 0:00:31 0:00:01 0:00:30 4071k 12 123M 12 16.0M 0 0 5658k 0 0:00:22 0:00:02 0:00:20 5659k 19 123M 19 24.6M 0 0 6453k 0 0:00:19 0:00:03 0:00:16 6454k 26 123M 26 33.2M 0 0 6929k 0 0:00:18 0:00:04 0:00:14 6930k 33 123M 33 41.7M 0 0 7233k 0 0:00:17 0:00:05 0:00:12 8566k 40 123M 40 50.1M 0 0 7434k 0 0:00:17 0:00:06 0:00:11 8718k 47 123M 47 58.7M 0 0 7610k 0 0:00:16 0:00:07 0:00:09 8746k 54 123M 54 67.3M 0 0 7739k 0 0:00:16 0:00:08 0:00:08 8745k 61 123M 61 75.9M 0 0 7849k 0 0:00:16 0:00:09 0:00:07 8753k 68 123M 68 84.5M 0 0 7931k 0 0:00:15 0:00:10 0:00:05 8756k 75 123M 75 93.0M 0 0 7997k 0 0:00:15 0:00:11 0:00:04 8776k 81 123M 81 101M 0 0 8048k 0 0:00:15 0:00:12 0:00:03 8741k 88 123M 88 109M 0 0 8095k 0 0:00:15 0:00:13 0:00:02 8728k 95 123M 95 118M 0 0 8137k 0 0:00:15 0:00:14 0:00:01 8707k 96 123M 96 119M 0 0 8140k 0 0:00:15 0:00:14 0:00:01 8700k curl: (28) Operation timed out after 15000 milliseconds with 125037660 out of 129790264 bytes received Finish: chroot ['fedpkg', 'sources']
The thing that's running in the task is fedpkg-minimal.
The timeout is supposed to be 15 minutes, but since curl expectes the timeout in seconds, the original patch was wrong. This is now fixed in upstream, so once you get latest fedpkg-minimal into your build tag, it should start working correctly.
It seems my eyes stopped after seeing "fedpkg" in Koji logs. My bad. I see it being mentioned in the changelog. Building it now.
I closing this ticket as I expect a newer fedpkg-minimal to resolve this issue. Thanks!
Metadata Update from @davidlt: - Issue close_status updated to: Invalid - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.