Currently the proxies are set up with the following robots.txt

[root@proxy01 conf.d][PROD]# less /srv/web/greenwave.fedoraproject.org-robots.txt
User-agent: *
Disallow: /

However this looks like it doesn't work because the setup is:
ProxyPass / "balancer://app-os/"
ProxyPassReverse / "balancer://app-os/"

so everything goes to the backend node servers without the front end giving a robots.txt file. I am not sure which part is by design and which is by accident..

Opened a ticket in greenwave for consideration --> https://pagure.io/greenwave/issue/402

OK I think I have found a solution, but I am not sure if it will break other sites. I have sent a pre-FBR to get people to review before I try it.

It looks like this is live:

https://greenwave.fedoraproject.org/robots.txt

User-agent: *
Disallow: /

:green_book:

I think the problem is with https://greenwave.app.os.fedoraproject.org/robots.txt

For historical reason we have 2 routes for greenwave, we could look at removing the old one, but I think this is the route used by other services for example bodhi.

So before this change, neither of those entry points had a robots.txt and now both of them do. I would see if the googlebots honor the robots.txt and if they don't we will add a block for them on app.os

So before this change, neither of those entry points had a robots.txt and now both of them do. I would see if the googlebots honor the robots.txt and if they don't we will add a block for them on app.os

Ok so I ll close this and check if that worked or not in few days :smile:

Thanks