#4496 download-i2.fp.org Internet2 route is very slow
Closed: Fixed None Opened 9 years ago by wtogami.

There seems to be a problem with download-i2.fedoraproject.org's Internet2 route. A test download from download-i2 to the networks of mit.edu or hawaii.edu sustain 600KB/sec and 1200KB/sec respectively while downloads from non-Internet2 hosts easily surpass 4000KB/sec.


Can you attach traceroutes from the various places?

And is this persisting still?

Non-internet2 clients can download at 4MB/sec while internet2 seems very slow. Downloading from the non-Internet2 mirror host is considerably faster at 30MB/sec.

MIT swings wildly between zero and 2MB/sec.
{{{
1. 18.9.64.3
2. BACKBONE-RTR-1.MIT.EDU
3. DMZ-RTR-1-BACKBONE-RTR-1.MIT.EDU
4. EXTERNAL-RTR-3-DMZ-RTR-1.MIT.EDU
5. nox1sumgw1-vl-526-nox-mit.nox.org
6. ATM10-430-OC12-GIGAPOPNE.NOX.ORG
7. 192.5.89.222
8. et-3-0-0.101.rtr.wash.net.internet2.edu
9. et-5-0-0.104.rtr.atla.net.internet2.edu
10. chltcrs-gw-to-internet2-atl.ncren.net
11. rtpcrs-gw-to-rlcrs-gw.ncren.net
12. rtp7600-gw-sec-to-rtpcrs-gw.ncren.net
13. redhat-to-rtp7600-gw.ncren.net
14. download-rdu-i2.fedoraproject.org
}}}

Hawaii.edu: 1.2MB/sec
{{{
traceroute to download-i2.fedoraproject.org (204.85.14.12), 30 hops max, 60 byte packets
1 128.171.104.1 (128.171.104.1) 0.413 ms 0.557 ms 0.629 ms
2 uhm-kolanut-vl667.uhnet.net (128.171.64.190) 1.131 ms 1.130 ms 1.030 ms
3 xe-0-1-0-61-ohelo.uhnet.net (205.166.205.32) 1.059 ms 1.044 ms 1.024 ms
4 xe-1-0-0-62-poha.uhnet.net (205.166.205.35) 0.994 ms 0.914 ms 0.876 ms
5 internet2-uhm-losa.uhnet.net (205.166.205.13) 49.671 ms 49.630 ms 49.622 ms
6 et-1-0-0.111.rtr.hous.net.internet2.edu (198.71.45.20) 82.008 ms 81.980 ms 81.993 ms
7 et-10-0-0.105.rtr.atla.net.internet2.edu (198.71.45.12) 105.729 ms 105.718 ms 105.696 ms
8 chltcrs-gw-to-internet2-atl.ncren.net (198.86.17.65) 112.440 ms 112.404 ms 112.536 ms
9 rtpcrs-gw-to-rlcrs-gw.ncren.net (128.109.212.17) 115.338 ms 115.323 ms 115.301 ms
10 rtp7600-gw-sec-to-rtpcrs-gw.ncren.net (128.109.9.69) 113.247 ms 113.066 ms 113.083 ms
11 redhat-to-rtp7600-gw.ncren.net (128.109.70.198) 113.677 ms 113.797 ms 113.818 ms
}}}

Hawaii.edu downloading from dl.fedoraproject.org instead of download-i2.fedoraproject.org can easily do 30MB/sec.
{{{
1 128.171.104.1 (128.171.104.1) 0.328 ms 0.375 ms 0.413 ms
2 xe-4-0-0-669-kolanut.uhnet.net (128.171.213.14) 0.861 ms 0.972 ms 0.948 ms
3 xe-0-1-0-51-ohelo.uhnet.net (205.166.205.22) 1.007 ms 1.006 ms 1.012 ms
4 xe-1-1-0.3505.sttl0.tr-cps.internet2.edu (64.57.21.49) 65.477 ms 65.476 ms 65.464 ms
5 ge-10-0-9.mpr1.sea1.us.above.net (64.125.13.65) 72.846 ms 72.799 ms 72.797 ms
6 ae10.cr1.sjc2.us.above.net (64.125.21.25) 74.014 ms 73.994 ms 74.131 ms
7 xe-0-3-0.cr1.lax112.us.above.net (64.125.26.25) 92.156 ms 92.141 ms 92.163 ms
8 ae7.mpr3.phx2.us.above.net (64.125.31.110) 100.501 ms 100.497 ms 100.495 ms
9 208.184.108.38.IPYX-072053-007-ZYO.above.net (208.184.108.38) 82.359 ms 82.555 ms 82.521 ms
10 border1.po2-bbnet2.phx004.pnap.net (69.25.168.69) 81.378 ms border1.po1-bbnet1.phx004.pnap.net (69.25.168.5) 81.674 ms border1.po2-bbnet2.phx004.pnap.net (69.25.168.69) 81.681 ms
11 redhat-2.border1.phx004.pnap.net (69.25.121.26) 91.580 ms redhatphx-1.phx004.pnap.net (69.25.120.242) 91.621 ms redhat-2.border1.phx004.pnap.net (69.25.121.26) 91.552 ms
12 ip-160-181-132-209.redhat.com (209.132.181.160) 92.875 ms 92.658 ms 92.963 ms
}}}

WPI is connected to the same Internet2 peering point as MIT, Northern Crossroads (NoX) in Boston/Cambridge. We are seeing the same slowness on download-i2. I analyzed my log files from that last few years.

See this graph of WPI's rsync download rates from
download-i2 (red) and dl01-5 (green) over the last few years. In 2011 and 2012 download-i2 was a fairly good 5-7 megabytes/sec (40-60 megabits/sec). I switched to dl01-5 in 2013 which provided a speed boost, then back to download-i2. There is marked decline starting in July 2013 to 2 megabytes/sec, another decline in March 2014 to 1.5 megabytes/sec, and another decline in August 2014 to the current 1 megabyte/sec. At the far right edge of the graph you can see a tiny green spike as I just switched back to dl01 last night, which peaked up to 11 megabytes/sec (90 megabits/sec, and actually I saw instantaneous rates of up to 130mbps).

[[Image(wpi-rsync-rates.png)]]

Traceroute from WPI to download-i2:
{{{
My traceroute [v0.75]
valen.wpi.edu (0.0.0.0) Thu Sep 25 03:41:28 2014
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. rtr-phsr1-flcr-netops.inf.wpi.edu 0.0% 21 0.3 0.4 0.2 4.1 0.8
2. rtr-gpop1-phsr1.inf.wpi.edu 0.0% 21 0.5 1.3 0.4 14.9 3.1
3. wpi-goddard.goddard.gigapop.net 0.0% 21 1.4 2.3 0.7 11.6 2.3
4. nox1sumgw1-vl-591-nox-wpi.nox.org 0.0% 20 2.8 3.3 2.4 3.9 0.4
5. 192.5.89.21 0.0% 20 4.0 4.8 3.8 5.5 0.5
6. nox300gw1-peer-nox-internet2-192-5-89-222.nox.org 0.0% 20 8.5 8.9 8.3 10.3 0.5
7. et-3-0-0.101.rtr.wash.net.internet2.edu 0.0% 20 14.5 15.0 13.9 22.2 1.8
8. et-5-0-0.104.rtr.atla.net.internet2.edu 0.0% 20 27.8 28.0 27.0 29.0 0.4
9. chltcrs-gw-to-internet2-atl.ncren.net 0.0% 20 27.6 29.9 27.6 32.5 1.4
10. 128.109.212.25 0.0% 20 29.9 29.8 27.5 32.7 1.5
11. rtp7600-gw-tg4-2-to-rtp-crs-gw.ncren.net 0.0% 20 29.0 33.2 27.2 132.2 23.3
12. redhat-to-rtp7600-gw.ncren.net 0.0% 20 29.0 28.6 27.8 29.3 0.4
13. download-rdu-i2.fedoraproject.org 0.0% 20 28.8 29.2 28.4 30.4 0.5
}}}

Traceroute from WPI to dl01:

{{{
My traceroute [v0.75]
valen.wpi.edu (0.0.0.0) Thu Sep 25 03:43:26 2014
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. rtr-phsr1-flcr-netops.inf.wpi.edu 0.0% 21 0.3 0.7 0.2 9.8 2.1
2. rtr-gpop1-phsr1.inf.wpi.edu 0.0% 21 0.5 4.7 0.4 25.9 7.7
3. wpi-goddard.goddard.gigapop.net 0.0% 20 1.0 2.9 0.7 28.0 5.9
4. nox1sumgw1-vl-595-nox-wpi.nox.org 0.0% 20 3.1 3.3 2.7 3.7 0.3
5. 192.5.89.21 0.0% 20 4.7 4.8 3.6 5.3 0.4
6. et-5-0-0.120.ny0.tr-cps.internet2.edu 0.0% 20 8.7 9.3 8.2 15.4 1.5
7. above.net 0.0% 20 9.7 9.5 8.2 13.3 1.4
8. ae5.cr2.lga5.us.above.net 0.0% 20 9.0 13.6 8.4 41.6 10.0
9. ae4.cr2.dca2.us.above.net 0.0% 20 14.4 16.9 14.0 38.6 6.5
10. ae14.cr2.iah1.us.above.net 0.0% 20 116.5 59.3 53.7 116.5 14.8
11. ae6.mpr4.phx2.us.above.net 0.0% 20 76.3 79.8 76.0 101.2 6.4
12. ae4.mpr3.phx2.us.above.net 0.0% 20 76.3 79.6 76.1 124.9 10.9
13. 208.184.108.38.IPYX-072053-007-ZYO.above.net 0.0% 20 67.5 67.5 66.4 68.4 0.6
14. border1.po2-bbnet2.phx004.pnap.net 0.0% 20 67.8 68.4 67.1 73.4 1.2
15. redhat-2.border1.phx004.pnap.net 0.0% 20 64.0 64.3 63.1 64.9 0.5
16. ip-160-181-132-209.redhat.com 0.0% 20 64.0 64.2 63.2 64.9 0.5
17. ???
18. ???
19. dl01.fedoraproject.org 0.0% 20 64.1 64.1 63.5 64.5 0.3
}}}

I am taking this ticket and will be checking to see if we can figure out where the speed differences are happening (if they are happening at RH, NCREN, or further up.) We have opened a ticket with NCREN to find out what they can tell.

Going from testing, the problem seems to be pretty close to the source as Ibiblio which was getting 60 mb/s speeds are only getting 5 mb/s speeds. I have passed information to our contact with NCREN to see what is going on.

So, we have talked to lots of networking folks, etc... and it turns out our connection to Internet-2 is just not that great. It's slow, it's hitting it's BW cap and there's not really any plans to improve things.

So, what would you all think to just dropping the download-i2 setup and having everyone use the master I1 mirrors? It would mean you would need to use I1 BW to sync, but it should be a lot faster...

We are retiring the download-i2 mirrors.

Please switch to i1 download master mirrors.

Thanks.

Login to comment on this ticket.

Metadata