6ac1be1
@@ -2,3 +2,5 @@
Disallow: /accounts/
Disallow: /django-admin/
Disallow: /archives/*/vote$
+ User-agent: Bytespider
+ Disallow: /
This crawler is doing about 2M requests a day to lists.fedoraproject.org. This is causing db load on db01, causing services to have issues.
So, block them here for now, but we may want to block them elsewhere too.
Signed-off-by: Kevin Fenzi kevin@scrye.com
Build succeeded. https://fedora.softwarefactory-project.io/zuul/buildset/645a952d193a443d9e1fad4716062bac
Metadata Update from @kevin: - Pull-request tagged with: freeze-break-request
I'm just gonna push this since the db load is causing actual problems.
Pull-Request has been merged by kevin
kevin
commented 13 days ago
This crawler is doing about 2M requests a day to
lists.fedoraproject.org. This is causing db load on db01, causing
services to have issues.
So, block them here for now, but we may want to block them elsewhere
too.
Signed-off-by: Kevin Fenzi kevin@scrye.com