fedora-infrastructure

#6969 Mirrormanager keeps thinking my mirrors are not up to date
Closed: Fixed Opened by tibbs.

Closed: Fixed
Reopen Issue

I would like to get some help understanding why mirrormanager keeps marking my mirrors as not being up to date for a significant amount of content.

This is not particularly urgent (so no time deadline), but it would be super nice to have my mirrors be used for their intended purpose.

My site is https://admin.fedoraproject.org/mirrormanager/site/1650, under which are two physical hosts. One (pubmirror2.math.uh.edu) has been split into three hosts because the crawler used to choke on a full mirror. I know that's been fixed now. The other host is pubmirror1.math.uh.edu which I had fully re-set up just a few days ago. All of the mirrors (of course) use quick-fedora-mirror and do mirrormanager checkins. They pull from dl-tier1 and should be no more than ten minutes plus transfer time behind the masters.

Looking at a crawl log (https://admin.fedoraproject.org/mirrormanager/crawler/2433.log) I see that it just goes through and decides that a big pile of content is out of date. But as far as I can tell, it still thinks some of the content is correct. So, what criteria is it using when something is marked out of date? It is looking at file properties? The mere presence of files? Or is it comparing directory timestamps or something else?

And the log messages are interesting. I see things like the following:

Directory pub/alt/risc-v/logs/deco-archive-1.7-1.fc24.src.rpm is not up-to-date on this host.

But... that's a file, not a directory. Is that just an incorrect word in the message, or is there some deeper issue?

Edit: it is actually a directory, and that's a complete red herring. Sorry.

I'm of course worried that quick-fedora-mirror has bugs and is somehow failing to transfer things properly. But just running rsync over some of the content which is supposedly outdated doesn't appear to transfer anything. I could do a full refresh by running rsync over everything if it would help (though it would take a while). If I knew more about what the crawler doesn't like about my mirror, I would have more of an idea of what I might be doing wrong.

So, @smooge did a crawler run with debugging, which showed that the crawler doesn't like the size values returned by rsync and in fact complains when converting them to floats.

This is a client side issue. The client rsync version determines what the output of an rsync-generated directory listing looks like. With the rsync in EL7.4 (rsync-3.0.9-18.el7.x86_64):

test-el7:/tmp❯ rsync --temp-dir=/tmp --exclude=.snapshot --exclude='*.~tmp~' --no-motd --timeout 14400 --exclude=lost+found rsync://pubmirror1.math.uh.edu/fedora-enchilada/linux/updates/28/Everything/armhfp/Packages/m/mcabber-1.1.0-1.fc28.1.armv7hl.rpm
-rw-r--r--      311720 2018/05/10 10:21:34 mcabber-1.1.0-1.fc28.1.armv7hl.rpm

After updating to EL7.5 (rsync-3.1.2-4.el7.x86_64):

test-el7:/tmp❯ rsync --temp-dir=/tmp --exclude=.snapshot --exclude='*.~tmp~' --no-motd --timeout 14400 --exclude=lost+found rsync://pubmirror1.math.uh.edu/fedora-enchilada/linux/updates/28/Everything/armhfp/Packages/m/mcabber-1.1.0-1.fc28.1.armv7hl.rpm
-rw-r--r--        311,720 2018/05/10 10:21:34 mcabber-1.1.0-1.fc28.1.armv7hl.rpm

Bottom line: splitting on spaces and taking the second field no longer gives you a number. I see two options:

  • Strip commas from the size field, somewhere around https://github.com/fedora-infra/mirrormanager2/blob/master/utility/mm2_crawler#L942
  • Call rsync with the --no-human-readable flag to revert to the old behavior by changing https://github.com/fedora-infra/mirrormanager2/blob/master/mirrormanager2/lib/sync.py#L78

Metadata Update from @smooge:
- Issue assigned to smooge

I have added --no-human-readable to the configs in the file. Making a change to that line did not add it. I have run it with the updated data and it seems to work

Metadata Update from @smooge:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
Lowest
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.