For instance, I tried just wgeting an ISO from openqa-stg01, and it was transferring at about 600-800KB/sec till I cancelled it...

I can't see anything obviously wrong off hand.

We could try and update kojipkgs01/02 to the new kernel and reboot and see if that helps...

I'm wondering if this is related to the multipath stuff to the netapp?

I doubt it... the kojipkgs instances are not on iscsi storage. They use nfs to get the content...

@kevin did...something...and it seems massively better now. Compare the metadata refresh and package install that happens at the start of these two tests:

• https://openqa.stg.fedoraproject.org/tests/87140/file/video.ogv (8 hours ago)
• https://openqa.stg.fedoraproject.org/tests/87642/file/video.ogv (now)

let's hope it stays! I'll try and keep an eye out for this happening again.

Yeah, I applied updates and rebooted.

So, it could be the move from 4.9.5 kernel to 4.10.5 fixed something in the networking stack.

Or it could be that after some uptime squid isn't processing normally. We have seen it slow down in the past, but have never been able to isolate it. The machines were up for about 65 days, which is longer than we normally go between reboots (just due to freezes and other work).

In any case do let us know if you see this again and we can try and isolate it more.

:octopus:

It's back to being pretty slow again already :/

ISO transfers are going about 1-2MB/sec. It took about an hour each to download the Alpha 1.7 Server DVD ISOs, which are about 3GB in size. When I do a wget test, it shows 1-2MB/sec; by comparison, wget from dl.fedoraproject.org (which resolves to dl.phx2.fedoraproject.org) goes between 10 and 50MB/sec.

So we're pretty sure this is, somehow, squid's fault. From batcave01 you can hit kojipkgs01 and kojipkgs02 on port 80 (which goes through squid) and 8080 (which goes direct to apache). If you try wget'ing the same file on port 80 and 8080, it'll go slow on 80 and very very fast on 8080.

We just don't know what squid's problem is, yet. @smooge notes "the kojipkgs have a ton of connections in TIME_WAIT".

We have switched out squid for varnish and it seems to be doing much much better.

I still think we had something wrong with squid, it should not have acted that way, but switching to varnish was much easier than continuing to debug squid.

Please re-open or file a new ticket if you see this again.

:oncoming_police_car:

Thanks again, everything seems to be working great now. Let's hope it stays that way :)