Yes, we blocked those ips. :(

They were hitting some of our services very very hard, so they got lumped into the "AI scaper" blocking. ;(

I've unblocked them now. Sorry for the trouble.

Would it help to know what sort of traffic we were seeing ? Or just let you know next time before we do anything?

Kevin hi,

We are still blocked. Only the following 2 IPs respond the rest are still blocked:
wildcard.fedoraproject.org. 29 IN A 38.145.60.20
wildcard.fedoraproject.org. 29 IN A 38.145.60.21

$ dig mirrors.fedoraproject.org

; <<>> DiG 9.18.20 <<>> mirrors.fedoraproject.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34627
;; flags: qr rd ra; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;mirrors.fedoraproject.org. IN A

;; ANSWER SECTION:
mirrors.fedoraproject.org. 3 IN CNAME wildcard.fedoraproject.org.
wildcard.fedoraproject.org. 29 IN A 185.141.165.254
wildcard.fedoraproject.org. 29 IN A 18.192.40.85
wildcard.fedoraproject.org. 29 IN A 85.236.55.6
wildcard.fedoraproject.org. 29 IN A 152.19.134.142
wildcard.fedoraproject.org. 29 IN A 38.145.60.20
wildcard.fedoraproject.org. 29 IN A 38.145.60.21
wildcard.fedoraproject.org. 29 IN A 152.19.134.198
wildcard.fedoraproject.org. 29 IN A 18.159.254.57
wildcard.fedoraproject.org. 29 IN A 18.133.140.134

Hum. It might have been blocked in other places.

Let me look...

Yep. It was. Ok, try now?

Thx. IP resolving looks good now.
About our usage:
NVIDIA Networking (aka. Mellanox) develops Linux Drivers for NICs, Switch NOS, HPC Stack, Multimedia, Storage Optimization libraries and many more. All of these projects depend to some extend on Fedora Linux package repositories.
My guess is that we have peak usage during weekends when our "million tests" regression cycles are executed.
Can you provide insights on the load you experience from our side?
Do you have any guidelines for heavy users like ourselves that might help prevent such outages in the future?

Not sure about the details here, but it sounds like a good use case for a private mirror. That way all your downloads would be redirected to a system within your network.

We will implement a private mirror for EPEL and Fedora repositories.
Can we get an estimate on the current size of the repositories?

I think it might be documented somewhere, but on my mirror I see:

$ du -hs epel fedora
556G    epel
4.2T    fedora

Sorry to bug you again, but the block is back for 94.188.199.18.
We started working on creating a local mirror of your repositories, but it will take a couple of weeks to set it up and copy all the data.
Can you please unblock use again?
Do we understand what is the threshold we are crossing on your side?

Sorry about that, you should be unblocked again. We're making a note of it and hopefully it shouldn't happen again.

Thanks.
We've identified a possible cause for the load on your repositories from our side.
Recently we migrated all developer environments to use Fedora and we suspect auto package updates are causing the load on the community repositories.
Immediate action on our side is to disable the package auto update.
In parallel we are creating the local repository mirror.

I'm not sure thats what we are seeing... updates do hit mirrors.fedoraproject.org, but then they go to a mirror (yours locally or one of ours, etc).

This was traffic to src.fedoraproject.org or kojipkgs.fedoraproject.org or koji.fedoraproject.org...

At least I think thats the case, we need to look more closely at our logs and figure out exactly what this traffic was. :)
Hopefully we can get to that soon... sorry for all the blocking. :(

So, I looked around some more and have a theory.

@zlopez what site were you looking at when you blocked things yesterday? Was it just the main fedoraproject.org host?

If so, I think it was due to lots of hits to hotspot.txt.
This is a static file with just "OK" in it. It's used by the NetworkManager-config-connectivity-fedora package to tell if a install is behind a captive portal or not.
While the number of hits could be large here, the amount of traffic is... small.

So, a few options:

• We could make sure not to use that file to tell who is causing traffic
• We could make sure not to just look at the fedoraproject.org host/site, but sites of the things having loading issues directly.
• You could configure your clients to use a different internal site for hotspot.txt (it does need to be sure to serve that file with no caching).
• If you don't need/want the captive portal detection you could remove that package from your clients.

Possibly most or all of those make sense to do. ;)

Thoughts?

I didn't found specific access log for wiki, so I went with fedoraproject.org as wiki is on fedoraproject.org/wiki.

Yeah, true... but do note that there's a lot of other stuff on that virthost... like the hotspot hits

@idlv75 See above... I guess we will close this now from our end, but do look at that theory and some possible actions for your side.

Feel free to reopen or file a new ticket if there's anything more we can do from our side.