#4458 downloads from fedoraproject.org are slow
Closed: Fixed None Opened 5 years ago by chrismurphy.

= bug description =
Downloading ISOs from dl.fedoraproject.org or kojipkgs.fedoraproject.org is very slow, > 5 hours for a Rawhide live ISO.

= bug analysis =

https://kojipkgs.fedoraproject.org/work/tasks/5281/7185281/Fedora-Live-Workstation-x86_64-rawhide-20140723.iso

51-106KB/sec download speed

ming:~ chris$ traceroute kojipkgs.fedoraproject.org
traceroute to kojipkgs.fedoraproject.org (209.132.181.10), 64 hops max, 52 byte packets
1 dd-wrt (192.168.1.1) 3.041 ms 1.056 ms 0.969 ms
2 c-75-70-16-1 (75.70.16.1) 11.533 ms 11.277 ms 12.143 ms
3 xe-11-0-2-32767-sur01.denver.co.denver.comcast.net (68.85.221.161) 26.529 ms 10.017 ms 13.046 ms
4 xe-10-1-0-0-sur02.denver.co.denver.comcast.net (68.86.128.102) 11.186 ms 10.496 ms
xe-10-1-2-0-sur02.denver.co.denver.comcast.net (162.151.8.10) 9.641 ms
5 ae-20-0-ar01.aurora.co.denver.comcast.net (68.86.128.173) 13.631 ms 15.459 ms 12.109 ms
6 te-1-1-0-4-cr01.chicago.il.ibone.comcast.net (68.86.95.205) 46.571 ms
he-3-10-0-0-cr01.denver.co.ibone.comcast.net (68.86.92.25) 15.119 ms
te-1-1-0-4-cr01.chicago.il.ibone.comcast.net (68.86.95.205) 17.111 ms
7 xe-5-1-0-0-pe01.910fifteenth.co.ibone.comcast.net (68.86.82.206) 12.767 ms 12.317 ms 12.154 ms
8 te3-5.ccr01.den03.atlas.cogentco.com (154.54.10.33) 15.325 ms 15.879 ms 13.743 ms
9 te0-1-0-3.ccr22.den01.atlas.cogentco.com (154.54.83.33) 14.416 ms
te0-6-0-3.ccr22.den01.atlas.cogentco.com (154.54.45.185) 16.998 ms
te0-6-0-3.ccr21.den01.atlas.cogentco.com (154.54.45.181) 14.929 ms
10 be2092.ccr21.elp01.atlas.cogentco.com (154.54.31.146) 26.999 ms
be2091.ccr21.elp01.atlas.cogentco.com (154.54.31.142) 29.418 ms
be2092.ccr21.elp01.atlas.cogentco.com (154.54.31.146) 26.118 ms
11 be2254.ccr21.phx02.atlas.cogentco.com (154.54.7.33) 45.669 ms
be2414.ccr22.phx02.atlas.cogentco.com (154.54.42.105) 46.779 ms
be2254.ccr21.phx02.atlas.cogentco.com (154.54.7.33) 47.106 ms
12 te2-1.mag02.phx02.atlas.cogentco.com (154.54.1.230) 46.331 ms
te3-1.mag01.phx02.atlas.cogentco.com (154.54.24.145) 135.465 ms
te2-1.mag02.phx02.atlas.cogentco.com (154.54.1.230) 47.999 ms
13 154.24.18.22 (154.24.18.22) 48.067 ms
154.24.18.26 (154.24.18.26) 48.067 ms
154.24.18.22 (154.24.18.22) 45.752 ms
14 38.88.238.30 (38.88.238.30) 49.577 ms 46.355 ms 45.873 ms
15 ip-162-181-132-209.redhat.com (209.132.181.162) 45.342 ms
ip-166-181-132-209.redhat.com (209.132.181.166) 44.657 ms
ip-162-181-132-209.redhat.com (209.132.181.162) 46.458 ms
16 *
17 *
18 *
19 *
20 *
21 *
22 *
23 *
24 * *

= fix recommendation =


I'm seeing similar problems from here

{{{
traceroute to koji.fedoraproject.org (209.132.181.7), 30 hops max, 60 byte packets
1 192.168.1.1 (192.168.1.1) 272.928 ms 273.245 ms 281.620 ms
2 L100.BSTNMA-VFTTP-86.verizon-gni.net (98.110.150.1) 346.969 ms 637.842 ms 637.482 ms
3 G0-10-3-6.BSTNMA-LCR-22.verizon-gni.net (130.81.212.180) 637.148 ms 636.753 ms 636.287 ms
4 ae10-0.BOS-BB-RTR2.verizon-gni.net (130.81.163.172) 635.563 ms so-5-0-0-0.BOS-BB-RTR2.verizon-gni.net (130.81.151.220) 635.096 ms 635.003 ms
5 0.ae11.XL4.NYC1.ALTER.NET (152.63.20.117) 634.356 ms 633.929 ms 633.454 ms
6 *
7 0.ae2.BR2.NYC4.ALTER.NET (140.222.229.93) 771.537 ms 794.797 ms 795.383 ms
8 204.255.168.110 (204.255.168.110) 836.903 ms 204.255.168.114 (204.255.168.114) 811.350 ms 830.939 ms
9 be2062.mpd21.jfk02.atlas.cogentco.com (154.54.7.13) 795.512 ms be2061.ccr22.jfk02.atlas.cogentco.com (154.54.3.69) 841.783 ms be2062.mpd21.jfk02.atlas.cogentco.com (154.54.7.13) 1049.363 ms
10 be2149.ccr22.dca01.atlas.cogentco.com (154.54.31.125) 1043.843 ms be2148.ccr21.dca01.atlas.cogentco.com (154.54.31.117) 1048.707 ms be2151.mpd22.dca01.atlas.cogentco.com (154.54.40.73) 1057.494 ms
11 be2171.ccr42.atl01.atlas.cogentco.com (154.54.31.109) 1053.869 ms be2168.ccr41.atl01.atlas.cogentco.com (154.54.31.93) 1045.679 ms be2170.ccr41.atl01.atlas.cogentco.com (154.54.31.105) 1022.249 ms
12 be2173.ccr22.iah01.atlas.cogentco.com (154.54.29.118) 1021.511 ms 2314.874 ms be2172.ccr21.iah01.atlas.cogentco.com (154.54.29.18) 2294.497 ms
13 be2294.ccr21.sat01.atlas.cogentco.com (154.54.3.126) 2676.123 ms be2293.ccr21.sat01.atlas.cogentco.com (154.54.3.2) 2737.866 ms be2294.ccr21.sat01.atlas.cogentco.com (154.54.3.126) 2731.854 ms
14 be2301.ccr21.elp01.atlas.cogentco.com (154.54.5.174) 2550.994 ms 2535.722 ms 2549.364 ms
15 be2414.ccr22.phx02.atlas.cogentco.com (154.54.42.105) 2553.910 ms 2549.434 ms 2548.026 ms
16 te3-1.mag01.phx02.atlas.cogentco.com (154.54.24.145) 1834.648 ms te2-1.mag02.phx02.atlas.cogentco.com (154.54.1.230) 1834.644 ms 1834.109 ms
17 154.24.18.22 (154.24.18.22) 1849.364 ms 154.24.18.26 (154.24.18.26) 744.242 ms 154.24.18.22 (154.24.18.22) 958.341 ms
18 38.88.238.30 (38.88.238.30) 561.499 ms 498.085 ms 494.876 ms
19 ip-166-181-132-209.redhat.com (209.132.181.166) 620.619 ms ip-162-181-132-209.redhat.com (209.132.181.162) 638.334 ms ip-166-181-132-209.redhat.com (209.132.181.166) 632.900 ms
20 *
21 *
22 *
23 *
24 *
}}}

traceroute from a connection which does not have the problem (downloads are fast)
traceroute-ok.txt

sorry about the formatting
trace

traceroute for slow connection (around 40 KB/s)
traceroute.2.txt

If I need to download multiple files, I can work around this by downloading them simultaneously. Each download seems to be capped at the same speed.

Another thing I've noticed is that the download starts faster, then the speed drops.

Using aria2c (from the aria2 package) seems to mostly work around the problem, using multiple connections. I'm not familiar with it as I normally don't need it. I seem to be able to get around 200 KB/s with it, as opposed to around 40 KB/s without. I tried using the -j and -s options to increase the maximum number of parallel downloads/connections but it doesn't seem to help. My maximum download speed is 3 Mbits/s (about 350 KB/s).

Ok, so kind of looking like cogentco is the common thread. However, my connection from home here also uses them and is fine. ;(

Several other things to check:

  • Do you get fast speeds if you use http instead of https?

  • If you run mtr to kojipkgs.fedoraproject.org do you see any links drop out or get high ping times after a few minutes?

I'll try and get RHIT networking folks to look at this ticket next week and see if they can see any problems with any of our peering points or the like.

I don't see a difference in speed between http and https. I'm not familiar with mtr so don't know how to interpret the output.

Upon initial connection with mtr, I saw 7% packet loss from ae-20-0-ar01.aurora.co.denver.comcast.net, which does not show up in the attached version; while there are four that have 1.6% loss. 7% seems really excessive but have no idea if that's related to this problem.

Adding traceroute output at request of Kevin(nirik).

{{{
$ traceroute kojipkgs.fedoraproject.org
traceroute to kojipkgs.fedoraproject.org (209.132.181.10), 30 hops max, 60 byte packets
1 192.168.1.1 (192.168.1.1) 0.904 ms 0.884 ms 2.097 ms
2 L100.DLLSTX-VFTTP-58.verizon-gni.net (173.71.9.1) 9.568 ms 11.603 ms 11.735 ms
3 G0-5-0-5.DLLSTX-LCR-22.verizon-gni.net (130.81.58.30) 17.236 ms 17.205 ms 17.174 ms
4 ae8-0.DFW9-BB-RTR2.verizon-gni.net (130.81.162.160) 14.018 ms so-4-1-0-0.DFW9-BB-RTR2.verizon-gni.net (130.81.151.244) 17.210 ms ae4-0.DFW9-BB-RTR2.verizon-gni.net (130.81.199.66) 16.128 ms
5 0.xe-3-3-1.BR1.DFW13.ALTER.NET (152.63.99.21) 16.241 ms 16.869 ms 16.854 ms
6 te0-4-0-29.ccr21.dfw03.atlas.cogentco.com (154.54.12.205) 18.980 ms 9.494 ms 12.447 ms
7 be2032.ccr22.dfw01.atlas.cogentco.com (154.54.6.53) 14.622 ms be2031.ccr21.dfw01.atlas.cogentco.com (154.54.7.45) 14.779 ms 15.192 ms
8 be2146.ccr22.iah01.atlas.cogentco.com (154.54.25.241) 19.630 ms 17.179 ms be2147.mpd22.iah01.atlas.cogentco.com (154.54.27.17) 19.239 ms
9 be2291.ccr21.sat01.atlas.cogentco.com (154.54.2.190) 24.769 ms be2294.ccr21.sat01.atlas.cogentco.com (154.54.3.126) 25.942 ms 26.506 ms
10 be2301.ccr21.elp01.atlas.cogentco.com (154.54.5.174) 49.160 ms 49.255 ms 50.941 ms
11 be2254.ccr21.phx02.atlas.cogentco.com (154.54.7.33) 52.113 ms 51.078 ms be2414.ccr22.phx02.atlas.cogentco.com (154.54.42.105) 51.970 ms
12 te2-1.mag02.phx02.atlas.cogentco.com (154.54.1.230) 42.628 ms te3-1.mag01.phx02.atlas.cogentco.com (154.54.24.145) 46.951 ms 47.931 ms
13 154.24.18.26 (154.24.18.26) 49.306 ms 154.24.18.22 (154.24.18.22) 49.942 ms 49.916 ms
14 38.88.238.30 (38.88.238.30) 49.249 ms 51.783 ms 51.894 ms
15 ip-162-181-132-209.redhat.com (209.132.181.162) 56.361 ms 54.872 ms ip-166-181-132-209.redhat.com (209.132.181.166) 52.027 ms
16 *
17 *
18 *
19 *
20 *
21 *
22 *
23 *
24 *
25 *
26 *
27 *
28 *
29 *
30 * * *

}}}

Downloading from the mirror at download-ib01.fedoraproject.org appears to run at full speed and is unaffected.

I just downloaded boot.iso from http://dl.fedoraproject.org/pub/fedora/linux/development/21/x86_64/os/images/ and it took less than 2 minutes. So on my end the problem isn't happening anymore.

Confirmed, not affecting me either.

From my office on Time Warner Business Class, .iso downloads from Koji take 3 to 4 hours. Traceroute confirms a route through cogento.

{{{
traceroute to kojipkgs.fedoraproject.org (209.132.181.10), 30 hops max, 60 byte packets
1 gateway.macb-sp.net (172.16.4.1) 0.446 ms 0.407 ms 0.367 ms
2 192.168.0.1 (192.168.0.1) 0.719 ms 0.823 ms 0.839 ms
3 *
4 tge7-1.blhttx0202h.texas.rr.com (24.28.132.61) 16.712 ms 16.992 ms 16.928 ms
5 tge8-2.blhttx0201h.texas.rr.com (24.175.34.194) 16.823 ms 20.340 ms 24.678 ms
6 tge0-11-0-3.snantxvy01r.texas.rr.com (24.175.33.218) 34.640 ms 33.934 ms 33.897 ms
7 agg23.dllatxl301r.texas.rr.com (24.175.32.146) 38.197 ms 36.415 ms 40.665 ms
8 107.14.17.136 (107.14.17.136) 45.447 ms 44.024 ms 43.760 ms
9 ae1.pr1.dfw10.tbone.rr.com (107.14.17.234) 46.272 ms ae2.pr1.dfw10.tbone.rr.com (107.14.17.236) 54.137 ms 51.443 ms
10 te0-4-0-17.ccr21.dfw03.atlas.cogentco.com (154.54.11.49) 53.661 ms 43.727 ms 43.587 ms
11 be2031.ccr21.dfw01.atlas.cogentco.com (154.54.7.45) 44.341 ms 43.661 ms be2032.ccr22.dfw01.atlas.cogentco.com (154.54.6.53) 44.437 ms
12 be2147.mpd22.iah01.atlas.cogentco.com (154.54.27.17) 50.318 ms be2146.ccr22.iah01.atlas.cogentco.com (154.54.25.241) 54.918 ms 51.338 ms
13 be2294.ccr21.sat01.atlas.cogentco.com (154.54.3.126) 60.400 ms be2291.ccr21.sat01.atlas.cogentco.com (154.54.2.190) 57.842 ms 53.411 ms
14 be2301.ccr21.elp01.atlas.cogentco.com (154.54.5.174) 88.564 ms 82.053 ms 82.009 ms
15 be2254.ccr21.phx02.atlas.cogentco.com (154.54.7.33) 81.957 ms 81.291 ms 85.893 ms
16 te2-1.mag02.phx02.atlas.cogentco.com (154.54.1.230) 85.088 ms te3-1.mag01.phx02.atlas.cogentco.com (154.54.24.145) 95.723 ms 106.107 ms
17 154.24.18.22 (154.24.18.22) 99.166 ms 99.564 ms 154.24.18.26 (154.24.18.26) 99.135 ms
18 38.88.238.30 (38.88.238.30) 106.042 ms 70.058 ms 70.403 ms
19 ip-162-181-132-209.redhat.com (209.132.181.162) 70.944 ms ip-166-181-132-209.redhat.com (209.132.181.166) 70.345 ms ip-162-181-132-209.redhat.com (209.132.181.162) 70.893 ms
20 *
21 *
22 *
23 *
24 *
25 *
26 *
27 *
28 *
29 *
30 *
}}}

From my home, on AT&T Uverse, .iso downloads take 20 to 30 minutes - I can't run traceroutes from there, but I'm pretty sure AT&T doesn't go through cogento. From my university campus, the Workstation .iso (1.2GB) is taking 20 - 30 minutes (still in progress) - below is the traceroute from school.

{{{
[root@g55 ~]# traceroute kojipkgs.fedoraproject.org
traceroute to kojipkgs.fedoraproject.org (209.132.181.10), 30 hops max, 60 byte packets
1 172.19.127.254 (172.19.127.254) 21.088 ms 21.673 ms 21.670 ms
2 198.49.125.1 (198.49.125.1) 23.173 ms 23.904 ms 24.702 ms
3 192.168.170.2 (192.168.170.2) 36.747 ms 36.756 ms 38.698 ms
4 v501.core1.dal1.he.net (216.66.72.181) 38.698 ms 38.682 ms 38.663 ms
5 *
6 ae7.cr2.dfw2.us.above.net (64.125.20.233) 40.843 ms 13.004 ms 13.500 ms
7 ae2.cr2.iah1.us.us.above.net (64.125.21.62) 19.730 ms 22.802 ms 23.344 ms
8 ae6.mpr4.phx2.us.above.net (64.125.31.66) 45.179 ms 50.972 ms 46.210 ms
9 ae4.mpr3.phx2.us.above.net (64.125.26.57) 53.301 ms 53.295 ms 53.273 ms
10 208.184.108.38.IPYX-072053-007-ZYO.above.net (208.184.108.38) 93.612 ms 95.719 ms 95.622 ms
11 border1.po1-bbnet1.phx004.pnap.net (69.25.168.5) 95.616 ms 95.604 ms 95.597 ms
12 redhat-2.border1.phx004.pnap.net (69.25.121.26) 82.332 ms redhatphx-1.phx004.pnap.net (69.25.120.242) 82.248 ms redhat-2.border1.phx004.pnap.net (69.25.121.26) 82.222 ms
13 ip-164-181-132-209.redhat.com (209.132.181.164) 67.949 ms 82.139 ms 56.191 ms
14 *
15 *
16 *
17 *
18 *
19 *
20 *
21 *
22 *
23 *
24 *
25 *
26 *
27 *
28 *
29 *
30 * * *

}}}

This has become a problem again, except much worse than before. It's saying 4 days left to download today's Fedora-Live-Workstation ISO.

Source URL:
https://kojipkgs.fedoraproject.org/work/tasks/608/7430608/Fedora-Live-Workstation-x86_64-21-20140820.iso

mtr
http://ur1.ca/i0hjg

traceroute
http://ur1.ca/i0hjn

Replying to [comment:13 chrismurphy]:

This has become a problem again, except much worse than before. It's saying 4 days left to download today's Fedora-Live-Workstation ISO.

Source URL:
https://kojipkgs.fedoraproject.org/work/tasks/608/7430608/Fedora-Live-Workstation-x86_64-21-20140820.iso

I tried this download about 90 minutes after you posted, and it was still full-speed for me. But now, it's slow (about 50 KB/s). It doesn't seem to affect everyone at the same time.

kojipkgs is now slow for me, about 12 KB/s:

{{{
mtr -r -c 10 kojipkgs.fedoraproject.org
Start: Thu Aug 21 22:10:15 2014
HOST: genius.invalid Loss% Snt Last Avg Best Wrst StDev
1.|-- leela.invalid 0.0% 10 0.7 0.7 0.6 0.7 0.0
2.|-- rdsl-koln-de81.nw.mediawa 0.0% 10 22.9 27.5 22.4 62.9 12.5
3.|-- xmwc-koln-de02-chan-24.nw 0.0% 10 69.2 38.8 21.9 99.8 28.2
4.|-- ge-3-3-0.mpr1.lhr3.uk.abo 0.0% 10 33.7 33.7 33.3 34.0 0.0
5.|-- xe-0-1-1.mpr3.lhr3.uk.abo 0.0% 10 33.8 38.0 33.8 69.6 11.2
6.|-- xe-2-3-0.cr2.dca2.us.abov 0.0% 10 108.1 108.4 107.6 112.8 1.5
7.|-- ae14.cr2.iah1.us.above.ne 0.0% 10 136.1 137.8 136.1 149.6 4.1
8.|-- ae6.mpr4.phx2.us.above.ne 0.0% 10 186.5 162.7 159.6 186.5 8.4
9.|-- ae4.mpr3.phx2.us.above.ne 0.0% 10 159.6 159.4 159.1 159.8 0.0
10.|-- 208.184.108.38.IPYX-07205 0.0% 10 170.8 170.7 170.1 171.2 0.0
11.|-- border1.po1-bbnet1.phx004 0.0% 10 174.7 174.4 173.8 175.3 0.3
12.|-- redhat-2.border1.phx004.p 20.0% 10 164.1 164.6 164.1 165.6 0.0
13.|-- ip-164-181-132-209.redhat 0.0% 10 165.1 165.3 164.3 166.5 0.5
14.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
15.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
16.|-- kojipkgs.fedoraproject.or 20.0% 10 164.3 164.1 163.3 165.8 0.7
}}}

{{{
HOST: genius.invalid Loss% Snt Last Avg Best Wrst StDev
1.|-- 192.168.3.1 0.0% 10 0.7 0.7 0.6 0.9 0.0
2.|-- 213.20.58.135 0.0% 10 23.2 23.2 22.4 23.7 0.0
3.|-- 195.71.249.117 0.0% 10 22.5 22.3 21.7 23.8 0.5
4.|-- 195.66.236.76 0.0% 10 33.5 37.8 33.5 69.3 11.1
5.|-- 64.125.21.21 10.0% 10 34.2 34.2 33.8 34.6 0.0
6.|-- 64.125.27.165 0.0% 10 110.9 108.3 107.6 110.9 1.1
7.|-- 64.125.21.54 0.0% 10 136.3 137.5 136.0 147.7 3.5
8.|-- 64.125.31.66 0.0% 10 158.2 166.0 157.8 188.0 12.6
9.|-- 64.125.26.57 0.0% 10 190.4 162.6 159.2 190.4 9.8
10.|-- 208.184.108.38 0.0% 10 170.0 170.5 170.0 171.0 0.0
11.|-- 69.25.168.5 0.0% 10 174.1 174.3 174.0 174.5 0.0
12.|-- 69.25.121.26 10.0% 10 165.0 164.5 164.1 165.0 0.0
13.|-- 209.132.181.164 20.0% 10 165.1 165.4 164.7 165.8 0.0
14.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
15.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
16.|-- 209.132.181.10 20.0% 10 164.0 164.7 163.4 167.4 1.1
}}}

{{{
$ traceroute -n kojipkgs.fedoraproject.org
traceroute to kojipkgs.fedoraproject.org (209.132.181.10), 30 hops max, 60 byte packets
1 192.168.3.1 0.888 ms 1.323 ms 1.720 ms
2 213.20.58.135 25.151 ms 25.794 ms 27.088 ms
3 195.71.249.117 25.417 ms 27.146 ms 29.003 ms
4 195.66.236.76 38.358 ms 39.708 ms 40.417 ms
5 64.125.21.21 41.116 ms 42.700 ms 42.856 ms
6 64.125.27.165 115.359 ms 115.419 ms 117.962 ms
7 64.125.21.54 144.899 ms 136.837 ms 138.369 ms
8 64.125.31.66 160.093 ms 161.192 ms 161.267 ms
9 64.125.26.57 161.975 ms 160.094 ms 160.570 ms
10 208.184.108.38 171.487 ms 172.592 ms 171.364 ms
11 69.25.168.69 176.031 ms 174.555 ms 175.945 ms
12 69.25.120.242 165.447 ms 165.272 ms 165.835 ms
13 209.132.181.164 165.689 ms 209.132.181.160 164.109 ms 163.929 ms
}}}

Is anyone still seeing this?

If you are, is it dl.fedoraproject.org, kojipkgs or both?

Please re-open if there's anything we can do from here.

Login to comment on this ticket.

Metadata