Diagnose and fix major network connectivity issues with Fedora infra. Affected Koji, metalink, Bodhi, etc.
The connection hangs before it times out:
$ curl -v https://koji.fedoraproject.org/koji/ * Trying 38.145.60.21:443... * Connected to koji.fedoraproject.org (38.145.60.21) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/pki/tls/certs/ca-bundle.crt * CApath: none * TLSv1.3 (OUT), TLS handshake, Client hello (1):
ASAP.
I have the same issue. It is now very difficult to work with koji via web interface or terminal application.
This could be related to 10230 if you were just seeing it today.
If longer than that it might be something else.
Can you both provide 'traceroute koji.fedoraproject.org' output?
Metadata Update from @mohanboddu: - Issue tagged with: high-trouble, medium-gain, ops
I have this problem at least since 18 september. But it is floating. It can work normal ~10 minutes, then hang and no access for 1 minute, then again work good.
At this moment I not see problem in terminal. But web interface hang just now.
traceroute to koji.fedoraproject.org (38.145.60.20), 30 hops max, 60 byte packets 1 vpnmercury (10.9.0.1) 58.012 ms 57.998 ms 57.988 ms 2 host2-167-211-80.serverdedicati.aruba.it (80.211.167.2) 123.612 ms host3-167-211-80.serverdedicati.aruba.it (80.211.167.3) 133.258 ms 133.308 ms 3 cr1-be107.it1.aruba.it (62.149.185.99) 60.256 ms 60.272 ms 60.250 ms 4 be4458.rcr12.qzo03.atlas.cogentco.com (149.6.18.57) 60.229 ms be4460.rcr11.qzo03.atlas.cogentco.com (149.6.18.49) 60.211 ms 61.004 ms 5 be3138.rcr51.vce01.atlas.cogentco.com (154.54.36.225) 64.954 ms be3137.rcr51.vce01.atlas.cogentco.com (130.117.48.113) 64.964 ms 64.955 ms 6 be3526.rcr21.mil01.atlas.cogentco.com (154.54.59.2) 70.230 ms 68.002 ms 67.947 ms 7 be2864.ccr31.mrs02.atlas.cogentco.com (154.54.74.165) 77.638 ms 76.807 ms 76.791 ms 8 be2776.ccr31.vlc02.atlas.cogentco.com (154.54.72.77) 88.950 ms 88.909 ms 88.900 ms 9 be3355.ccr31.mad05.atlas.cogentco.com (154.54.57.229) 93.261 ms be3356.ccr32.mad05.atlas.cogentco.com (154.54.57.241) 93.045 ms be3355.ccr31.mad05.atlas.cogentco.com (154.54.57.229) 93.032 ms 10 be2325.ccr32.bio02.atlas.cogentco.com (154.54.61.133) 98.168 ms be2324.ccr31.bio02.atlas.cogentco.com (154.54.61.129) 97.608 ms 98.874 ms 11 be2332.ccr42.dca01.atlas.cogentco.com (154.54.85.245) 169.230 ms be2331.ccr41.dca01.atlas.cogentco.com (154.54.85.241) 169.842 ms be2332.ccr42.dca01.atlas.cogentco.com (154.54.85.245) 170.988 ms 12 be3083.ccr41.iad02.atlas.cogentco.com (154.54.30.54) 185.837 ms 185.807 ms be3084.ccr41.iad02.atlas.cogentco.com (154.54.30.66) 185.746 ms 13 38.32.106.90 (38.32.106.90) 171.231 ms 171.162 ms 171.158 ms 14 vrrp2 (209.132.185.253) 185.661 ms 185.635 ms 185.631 ms 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * *
Example from web: 41 second to open url in koji <img alt="Screenshot_20210921_214703.png" src="/fedora-infrastructure/issue/raw/files/90723556a2f619d4e9234887674494d1f973361acb1e82a78c66161fb61191d9-Screenshot_20210921_214703.png" />
Metadata Update from @mohanboddu: - Issue priority set to: Waiting on Assignee (was: Needs Review)
Would you be able to capture a 'tcpdump -w trace.out -s 8192 host koji.fedoraproject.org' while it's being slow and you have made a request to koji and send it to me in email? kevin@scrye.com
Or you could copy it to your fedorapeople.org space and I can copy it down from there?
Can't reproduce it now. May be problem is gone.
Might be it was related to the proxy issues we hit tuesday after all then? not sure...
Anyhow, do keep an eye out and if it starts happening again, reopen this or file a new issue and we will try and track it down.
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.