I have surprising entries in the crawler log if my host (https://mirrormanager.fedoraproject.org/crawler/455.log) , related to directories that I think are inexistant both on the host and on the master rsync server (at the time of the crawling).
For example, for my host, I have these two entries for today (time GMT):
2025-08-12 08:09:57,145 - INFO - Directory pub/epel/testing/9/Everything/source/tree/Packages/o is not up-to-date on this host. 2025-08-12 08:09:57,521 - INFO - Directory pub/epel/testing/9/Everything/x86_64/Packages/k is not up-to-date on this host.
These directories have been normally removed fro the master a couple of hours earlier. Here is the related information from quick-fedora-mirror output (time GMT+0200) :
quick-fedora-mirror
Mirror starting: Tue Aug 12 04:00:00 CEST 2025 >> Log: Run start: cfg /home/bellet/.config/quick-fedora-mirror-epel.conf, tmp /tmp/quick-mirror.agcDTGeUIe ============================================================ >> Log: Remote file list download start Downloading file lists >> Calling /usr/bin/rsync --timeout=600 -aSH -f R .~tmp~ --stats --delay-updates --out-format=@ %i %10l %n%L -v --files-from=filelist-transferlist --no-dirs --relative --compress rsync://dl03.fedoraproject.org/fedora-buffet0/ . +>> Log: calling /usr/bin/rsync --timeout=600 -aSH -f R .~tmp~ --stats --delay-updates --out-format=@ %i %10l %n%L -v --files-from=filelist-transferlist --no-dirs --relative --compress rsync://dl03.fedoraproject.org/fedora-buffet0/ . >> Log: rsync call completed succesfully with return 0 +>> Log: File list download: 46.67KB received, 36.58KB/s >> Log: Remote file list download: end >> Log: Processing start [...] >> Log: Deleting directory epel/testing/9/Everything/source/tree/Packages/o >>>> Removing epel/testing/9/Everything/source/tree/Packages/o >> Log: Deleting directory epel/testing/9/Everything/x86_64/Packages/k >>>> Removing epel/testing/9/Everything/x86_64/Packages/k >> Log: Directory deletion end >> Log: File deletion begin: 226 files Removing 226 stale files. >> Log: File deletion end Saving mirror time to /home/bellet/.cache/quick-fedora-mirror/timefile-epel ========================= Mirror finished: Tue Aug 12 04:02:32 CEST 2025 (0) >> Log: Run end; exiting 0.
These crawling errors are transient, and are mixed with other very valid messages. The common pattern is that they concern directories from testing subdirectories, deleted when content is moved from updates-testing to updates-released repos.
testing
updates-testing
updates-released
From the reading of mm2 source code, I'd say that the mirrormanager filelist database is not always updated when new content is pushed to the master mirrors?
Metadata Update from @zlopez: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: Needs investigation, medium-gain
@abompard Do you know what could cause it?
The scanner does not remove empty directories automatically. This is a manual operation. Currently this is works as designed. The only downside is indeed the log lines which are confusing.
Metadata Update from @adrian: - Issue untagged with: Needs investigation, medium-gain - Issue priority set to: Needs Review (was: Waiting on Assignee)
Thanks @adrian
Metadata Update from @zlopez: - Issue close_status updated to: Will Not/Can Not fix - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.