I might have deleted the email ... did you paste the above directly from it? Did it say how many pages it affected and/or how many it was still indexing?

I've looked in the code and don't see anything that should be doing anything like this.

And, maybe more importantly, also tried a few pages ... and don't see that meta html tag, or any suspicious http headers.

mediawiki is supposed to put that out on certain pages, that only humans should view (Eg. history). And has specific keywords users can use to turn it off for specific pages (do we want to go through the DB and see if people have inserted it a bunch?)

There's also some chance that one of the proxies is doing this, so only requests going through that proxy are bad?

The email has:

From: Google Search Console
Subject:
New reason preventing your pages from being indexed
Search Console has identified that some pages on your site are not being
indexed due to the following new reason:
* Blocked by robots.txt
If this reason is not intentional, we recommend that you fix it in order
to get affected pages indexed and appearing on Google.

The above was from:

curl -v -o foobar "https://fedoraproject.org/w/index.php?title=PackagingDrafts/Haskell&diff=337658&oldid=297011"

and looking at foobar... the header is coming back from the wiki in a meta tag in the html it sends back...

There's a 'indexing report' web page that has:

info_filled
Why pages aren’t indexed
Pages that aren’t indexed can’t be served on Google
Reason Source  Validation Trend Pages
Excluded by ‘noindex’ tag   Website Not Started        261,580

So, 261,580 pages affected.

AIUI ... anything that looks like "https://fedoraproject.org/w/index.php?..." is supposed to have that meta tag, because mediawiki doesn't want the robots scrapping 666 versions of the same page with diffs/history.

If you do: curl -v https://fedoraproject.org/wiki/PackagingDrafts/Haskell | fgrep meta then it's fine:

5<meta name="robots" content="max-image-preview:standard">

Ah... interesting.

I wonder why it's being reported now, I don't think it was before.

Perhaps we should just add that to deny in robots.txt ? that would hopefully make it realize it's not supposed to scrape those...

roles/httpd/website/templates/robots/fedoraproject.org-robots.txt

is the one there.

Done: https://pagure.io/fedora-infra/ansible/pull-request/2844

ok, lets close this then unless there's some new reason it has. ;)