#4150 i2 mirroring network issues
Closed: Fixed None Opened 10 years ago by ausil.

= bug description =
aarnet is on i2 and having a very poor experience mirroring for us due to network issues

= bug analysis =
tcp traceroutes to phx2 and rdu
{{{
traceroute to dl.fedoraproject.org (209.132.181.25), 30 hops max, 60 byte packets
1 113.197.1.129 (113.197.1.129) 0.218 ms 0.200 ms 0.142 ms
2 ge-1-1-3.bb1.b.bne.aarnet.net.au (202.158.203.30) 0.429 ms 0.405 ms 0.391 ms
3 so-2-0-0.bb1.b.syd.aarnet.net.au (202.158.194.53) 12.122 ms 12.090 ms 12.054 ms
4 xe-1-0-0.bb1.a.lax.aarnet.net.au (202.158.194.173) 158.682 ms 158.629 ms 158.579 ms
5 xe-0-0-0-16.r05.lsanca03.us.bb.gin.ntt.net (204.1.253.165) 159.857 ms 159.866 ms 159.816 ms
6 ae-5.r04.lsanca03.us.bb.gin.ntt.net (129.250.2.220) 159.209 ms 159.185 ms 159.133 ms
7 los-brdr-01.inet.qwest.net (63.146.27.217) 183.856 ms 183.700 ms 186.495 ms
8 phn-edge-06.inet.qwest.net (205.171.12.142) 197.307 ms 197.727 ms 197.658 ms
9 * 63-232-247-146.dia.static.qwest.net (63.232.247.146) 197.392 ms 197.329 ms
10 ip-166-181-132-209.redhat.com (209.132.181.166) 199.742 ms 197.602 ms ip-162-181-132-209.redhat.com (209.132.181.162) 198.982 ms
11 dl03.fedoraproject.org (209.132.181.25) 199.135 ms 199.156 ms 197.998 ms
12 dl03.fedoraproject.org (209.132.181.25) 199.435 ms 197.054 ms 199.071 ms
13 dl03.fedoraproject.org (209.132.181.25) 197.245 ms 197.195 ms 198.566 ms

traceroute -T download-i2.fedoraproject.org
traceroute to download-i2.fedoraproject.org (204.85.14.12), 30 hops max, 60 byte packets
1 113.197.1.129 (113.197.1.129) 0.168 ms 0.139 ms 0.139 ms
2 xe-1-2-0.bb2.a.syd.aarnet.net.au (202.158.194.106) 17.554 ms 17.482 ms 17.446 ms
3 et-1-0-0.bb1.a.sea.aarnet.net.au (202.158.194.110) 160.133 ms 160.097 ms 160.044 ms
4 abilene-1-is-jmb-776.lsanca.pacificwave.net (207.231.241.131) 184.380 ms 184.342 ms 184.289 ms
5 et-1-0-0.111.rtr.hous.net.internet2.edu (198.71.45.20) 217.084 ms 217.046 ms 216.994 ms
6 ae-1.10.rtr.atla.net.internet2.edu (64.57.28.113) 240.263 ms 240.251 ms 240.183 ms
7 chltcrs-gw-to-internet2-atl.ncren.net (198.86.17.65) 248.290 ms 248.109 ms 248.037 ms
8 chlt7600-gw-sec-to-chltcrs-gw.ncren.net (128.109.9.14) 245.861 ms 245.728 ms 245.653 ms
9 rtp7600-gw-to-chlt7600-gw.ncren.net (128.109.9.18) 249.099 ms 249.251 ms 249.237 ms
10 redhat-to-rtp7600-gw.ncren.net (128.109.70.198) 249.887 ms 253.554 ms 253.472 ms
11 download-rdu-i2.fedoraproject.org (204.85.14.12) 250.068 ms 249.995 ms 250.084 ms
}}}

icmp traceroutes to phx2 and rdu
{{{
traceroute to dl.fedoraproject.org (209.132.181.27), 30 hops max, 60 byte packets
1 113.197.1.129 (113.197.1.129) 0.246 ms 0.238 ms 0.240 ms
2 ge-1-1-3.bb1.b.bne.aarnet.net.au (202.158.203.30) 15.874 ms 15.924 ms 15.970 ms
3 so-2-0-0.bb1.b.syd.aarnet.net.au (202.158.194.53) 12.100 ms 12.125 ms 12.127 ms
4 xe-1-0-0.bb1.a.lax.aarnet.net.au (202.158.194.173) 158.688 ms 158.712 ms 158.715 ms
5 xe-0-0-0-16.r05.lsanca03.us.bb.gin.ntt.net (204.1.253.165) 159.259 ms 159.269 ms 159.280 ms
6 ae-5.r04.lsanca03.us.bb.gin.ntt.net (129.250.2.220) 159.233 ms 159.355 ms 159.147 ms
7 los-brdr-01.inet.qwest.net (63.146.27.217) 186.063 ms 186.436 ms 185.362 ms
8 phn-edge-06.inet.qwest.net (205.171.12.146) 198.579 ms 196.107 ms 197.046 ms
9 63-232-247-146.dia.static.qwest.net (63.232.247.146) 195.942 ms 198.012 ms 197.106 ms
10 ip-162-181-132-209.redhat.com (209.132.181.162) 198.006 ms 197.137 ms 197.047 ms
11 dl05.fedoraproject.org (209.132.181.27) 198.374 ms 197.551 ms 198.424 ms
12 dl05.fedoraproject.org (209.132.181.27) 198.850 ms 198.929 ms 199.741 ms
13 dl05.fedoraproject.org (209.132.181.27) 198.626 ms 198.669 ms 199.640 ms
[motk@aarnet-rk-ws3 ~]$ sudo traceroute -I download-i2.fedoraproject.org
traceroute to download-i2.fedoraproject.org (204.85.14.13), 30 hops max, 60 byte packets
1 113.197.1.129 (113.197.1.129) 0.211 ms 0.218 ms 0.219 ms
2 xe-1-2-0.bb2.a.syd.aarnet.net.au (202.158.194.106) 17.520 ms 17.545 ms 17.548 ms
3 et-1-0-0.bb1.a.sea.aarnet.net.au (202.158.194.110) 160.119 ms 160.144 ms 160.147 ms
4 abilene-1-is-jmb-776.lsanca.pacificwave.net (207.231.241.131) 184.305 ms 184.331 ms 184.335 ms
5 et-1-0-0.111.rtr.hous.net.internet2.edu (198.71.45.20) 216.922 ms 217.015 ms 217.029 ms
6 ae-1.10.rtr.atla.net.internet2.edu (64.57.28.113) 240.211 ms 240.148 ms 240.150 ms
7 chltcrs-gw-to-internet2-atl.ncren.net (198.86.17.65) 248.066 ms 246.812 ms 246.813 ms
8 chlt7600-gw-sec-to-chltcrs-gw.ncren.net (128.109.9.14) 284.530 ms 345.459 ms 345.460 ms
9 rtp7600-gw-to-chlt7600-gw.ncren.net (128.109.9.18) 249.116 ms 249.091 ms 249.091 ms
10 redhat-to-rtp7600-gw.ncren.net (128.109.70.198) 249.908 ms 249.840 ms 249.832 ms
11 download-rdu-i2.fedoraproject.org (204.85.14.13) 249.942 ms 250.164 ms 250.054 ms
}}}

irc conversation

{{{
03:36 < snerd> nirik: we're all native v6, so coming to that
03:37 < nirik> ok, I'm not sure what the state of ipv6 in rdu2 is right now, a while back it was working, but 100MB link, so not very fast.
03:37 < snerd> getting no AAAA at all
03:38 < nirik> ok, yeah, we might have disabled it.
03:38 < snerd> if you can give me an iso image or something to pull that'd be ideal
03:38 < snerd> last time I looks i2 was fantastic til it hit a port somewehre then splat
03:39 < snerd> 10 redhat-to-rtp7600-gw.ncren.net (128.109.70.198) 249.823 ms 249.879 ms 249.821 ms
03:39 < snerd> then walls of nope
03:40 < nirik> how about: http://download-rdu-i2.fedoraproject.org/pub/alt/releases/20/Spins/x86_64/Fedora-Live-Games-x86_64-20-1.iso
03:40 < nirik> well, or rsync
03:41 < snerd> sure
03:41 * nirik is gonna head to bed here before long... want to get up extra early to make sure no release day hiccups.
03:42 < snerd> hitting around 400kB/s not too exciting, certainly not what I'd expect from i2
03:42 < snerd> yeah, I won't keep you up too long :)
03:42 < snerd> 33,352,821 263KB/s eta 2h 13m
03:43 < snerd> looks like the window is leaping up and down
03:43 < nirik> yeah, thats definitely not good.
03:43 < snerd> when our network guy gets back from lunch I'll get him to have a look see
03:43 < nirik> so, if you do 'dl' instead of 'download-rdu-i2' is it better?
03:44 < snerd> looks like it
03:44 < snerd> 151,087,724 10.5MB/s eta 8m 35s
03:44 < nirik> yeah, thats better... wonder why I2 is sucking so bad. ;(
03:48 < snerd> we lose 9k packets for a start
}}}
= fix recommendation =
work with Red Hats newtork team to work out the issue and resolve


Hey Matt, any ideas here?

RHIT has contacted the reporter directly to work the issue. It seems like it might be their provider doing some I2 throttling somewhere, but they are investigating.

Since there's not much for us to do here, closing.

Login to comment on this ticket.

Metadata