#11841 Categories CentOS dissappear from crawler - random(?)
Closed: Fixed a month ago by poweredge. Opened a month ago by poweredge.

Describe what you would like us to do:

Recently build mirror.netone.nl for CentOS stream, but noticed the categories in the crawler log and subsequent the traffic is dropped. Unsure why this is. Rsync runs every 3 hours to ensure up2date repositories.

It did work for a few hours and then supposedly the mirror becomes somewhat 'invalid'.

When do you need this to be done by?

When convenient. But i love to support the community.

LOG snippet

2024-03-19 07:50:04,249 - INFO - No categories to crawl on host <Host(3005 - mirror.netone.nl)>
2024-03-19 07:50:04,262 - INFO - Ending crawl of <Host(3005 - mirror.netone.nl)> with status 12

This doesn't mean what you think it means. The crawler scans multiple categories at different times. The messages you cited just means that other categories were not scanned on your mirror.
The error message could be better.

Looking at your mirror I see that you rewrite http to https, you should not do that. If you provide both http and https URL users expect to be able to use both. If you do not want to provide http do not mention it. This might be also the reason why the crawler cannot correctly detect the state of your mirror.

Also, if you are able to provide rsync, please do that. That makes the life of our crawler much simpler and effective.

clear clear and clear! :-)

wr http -> ill adjust it
wr to rsync -> that too

Thanks for the quick response!

Cool, thanks. Just add the rsync URL to the existing category. If everything is clear, please close the ticket.

rsync category added by Fabian! (testing seemed to work)... adjusted rewrite of HTTP as well .... wil do some additional checking and then closing.

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: low-trouble, medium-gain, mirrorlists

a month ago

Things looking ok here now?

issue resovled yes - traffic flowing normally - categories added inc rsync. Cool!

Metadata Update from @poweredge:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

a month ago

Login to comment on this ticket.