@@ -0,0 +1,2 @@
+ User-agent: *
+ Disallow: /
Turns out back in 2015 (in 623be6f) I added a robots.txt on the koji
hubs to disallow crawling, but since it's behind the proxies, no one
ever saw that robots.txt. Instead they got the default one that just had
a 'crawl-delay 1' in it.
So, lets tell robots to go away for real now, as they are causing load
problems on the koji database and I don't think it's particularly useful
for koji to be indexed. It's a lot of dynamic content anyhow.
Signed-off-by: Kevin Fenzi email@example.com
rebased onto aeea2c1
Pull-Request has been merged by kevin