fedora-infrastructure

#3596 Robots.txt disallows everything on fedorahosted.org, installation instructions not indexed
Closed: Fixed Opened by adelton.

Closed: Fixed
Reopen Issue

= bug description =

I did a search on Google for Spacewalk installation instructions:

https://www.google.com/search?q=how+to+install+spacewalk

The first result is

{{{
spacewalk - FedoraHosted
https://fedorahosted.org/spacewalk/wiki/HowToInstall
A description for this result is not available because of this site's robots.txt – learn more.
}}}

The net effect is that our users find installations instructions on other web sites and blogs, derived from ours and often outdated, rather than hitting our documentation.

= bug analysis =

{{{
$ GET -S http://fedorahosted.org/robots.txt
GET http://fedorahosted.org/robots.txt
301 Moved Permanently
GET https://fedorahosted.org/robots.txt
200 OK
User-agent: *
Allow: /index/index.aux
Allow: /?a=project_index
Disallow: /
user-agent: AhrefsBot
disallow: /
}}}

= fix recommendation =

Please allow Spacewalk content to be indexed by crawlers.

In fact, I've found multiple infrastructure tickets that talk about robots.txt but the current content of that file does not look like having any of those changes in it. Is this some temporary glitch?

In fact, I've found multiple infrastructure tickets that talk about robots.txt but the current content of that file does not look like having any of those changes in it. Is this some temporary glitch?

Odd. I know of no open tickets on robots.txt. Can you give some pointers to them?

There was a private email I got about this to webmaster, so it was already on the list from that. ;)

Thanks for filing it and we will try and get it fixed soon.

I did

https://fedorahosted.org/fedora-infrastructure/search?q=robots

Not necessarily open tickets, just that the content of robots.txt was discussed in the past.

We have changed this, but not sure how long it will be until google recrawls.

Please re-open if this isn't fixed up after the next google crawl.

Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.