#1606
proxies / koji.fedoraproject.org: disallow robots on koji.fedoraproject.org
Merged a month ago
by
kevin.
Opened 2 months ago
by kevin.
kevin
commented 2 months ago
Turns out back in 2015 (in 623be6f) I added a robots.txt on the koji
hubs to disallow crawling, but since it's behind the proxies, no one
ever saw that robots.txt. Instead they got the default one that just had
a 'crawl-delay 1' in it.
So, lets tell robots to go away for real now, as they are causing load
problems on the koji database and I don't think it's particularly useful
for koji to be indexed. It's a lot of dynamic content anyhow.
Signed-off-by: Kevin Fenzi kevin@scrye.com