I'm reporting this issue on behalf of the mirror team in TUNA since their current main admin do not have FAS.
The TUNA mirror team noticed that if they rsync from rsync://download-ib01.fedoraproject.org/fedora-epel/7/x86_64/repodata/ , they will get repomd.xml of Sep 23rd UTC+8, but the rest of the .gz files all have a timestamp of Sep 21st UTC+8.
repomd.xml
.gz
$ rsync rsync://download-ib01.fedoraproject.org/fedora-epel/7/x86_64/repodata/ ---------------------------- Fedora Master Mirror Servers ---------------------------- If you are using bare rsync to sync content, please stop and take a look at https://pagure.io/quick-fedora-mirror. See http://fedoraproject.org/wiki/Infrastructure/Mirroring for further instructions. ---------------------------- You have connected to download-ib01.fedoraproject.org drwxr-xr-x 4,096 2022/09/23 10:49:35 . -rw-r--r-- 2,553,409 2022/09/21 10:18:07 039fa3f8075cd2bfe3dea833828604cbedbb725066ca12915fba6d133ffd312e-other.xml.gz -rw-r--r-- 3,520,460 2022/09/21 10:18:09 03b2706429f3657078959fee5e8766ec417b52362edc37947b34edc59f95f7a5-other.sqlite.bz2 -rw-r--r-- 12,533,117 2022/09/21 10:18:14 20393181bbf4f7660689b7f5181911a54c2aa2e75fe5c825f7361118c2dab308-filelists.sqlite.bz2 -rw-r--r-- 848 2022/09/21 10:18:21 2453eb0346f89d1089176c1f286d679cb6d0dde6c1d11c873c74cf3a15fc9b42-prestodelta.xml.gz -rw-r--r-- 7,336,461 2022/09/21 10:18:10 7e73a0c718b27211276dcf0b119ff89d244f77842eb83bf7b511a5e4a64a3d03-primary.sqlite.bz2 -rw-r--r-- 400,655 2022/09/21 10:17:44 ad987e4a2b74b987dc6272f6ccd44c6eb74cb611001478410d248aa628e0b17d-comps-Everything.x86_64.xml -rw-r--r-- 1,098,286 2022/09/21 10:39:23 b2c2f5a97c2a814f371ec68af171dc5c3f67f8d50b706ae5cd40fa1890cdd082-updateinfo.xml.bz2 -rw-r--r-- 4,081,429 2022/09/21 10:18:07 bdcb65795cb4776ea5673d0035c40c45dfbce980d6f404febed118f165c94743-primary.xml.gz -rw-r--r-- 98,819 2022/09/21 10:18:07 dab4efb89e04560cfbe53711f412c32247da39a0d721d9ec2c9f1e88aed424d1-comps-Everything.x86_64.xml.gz -rw-r--r-- 12,397,815 2022/09/21 10:18:07 f707d7db973f1768d5dc755a980bf2458e985b1e82dd1a5b3ad72aedbfe8e387-filelists.xml.gz -rw-r--r-- 4,849 2022/09/23 10:49:35 repomd.xml
When they check from IAD server, it looks good, all files are in Sep 23rd UTC+8.
$ rsync rsync://dl.fedoraproject.org/fedora-epel/7/x86_64/repodata/ ---------------------------- Fedora Master Mirror Servers ---------------------------- If you are using bare rsync to sync content, please stop and take a look at https://pagure.io/quick-fedora-mirror. See http://fedoraproject.org/wiki/Infrastructure/Mirroring for further instructions. ---------------------------- You have connected to dl-iad01.fedoraproject.org drwxr-xr-x 8,192 2022/09/23 10:49:35 . -rw-r--r-- 12,397,944 2022/09/23 10:28:40 2c1e6534273aa1dd3f1e2d607bb8eee87420d22601e7b8538b5c6b2435c3c2f2-filelists.xml.gz -rw-r--r-- 1,102,384 2022/09/23 10:49:35 6fc37c8f7f7e3102710a440812f8ad1bab3909ab38044acd702e9c3fb312ae8e-updateinfo.xml.bz2 -rw-r--r-- 7,338,605 2022/09/23 10:28:43 7e09d0257e4d6d597cc84629bac5836c3789baf6aff6a46a7e0e6f1404a260b6-primary.sqlite.bz2 -rw-r--r-- 2,553,486 2022/09/23 10:28:40 83f251eba371a4377d7591b710bf9d718761dd0c26fbb32628a785f2b1de3dbc-other.xml.gz -rw-r--r-- 3,521,169 2022/09/23 10:28:43 9bb3727f7c4a320d2b57d4a32b6c9d7585777b613c0ae55f7013fc7efdde0b3b-other.sqlite.bz2 -rw-r--r-- 400,655 2022/09/23 10:28:11 ad987e4a2b74b987dc6272f6ccd44c6eb74cb611001478410d248aa628e0b17d-comps-Everything.x86_64.xml -rw-r--r-- 12,530,186 2022/09/23 10:28:47 c071101485c15b2511c5f0461764ce45baae663d11bdc687c281843b738bff19-filelists.sqlite.bz2 -rw-r--r-- 4,081,871 2022/09/23 10:28:39 cf3463c24b4f63b5cf03b6c86545a706554fb72224ca746853fc28fa3159310a-primary.xml.gz -rw-r--r-- 98,819 2022/09/23 10:28:40 dab4efb89e04560cfbe53711f412c32247da39a0d721d9ec2c9f1e88aed424d1-comps-Everything.x86_64.xml.gz -rw-r--r-- 605 2022/09/23 10:28:55 f5257a1365d32caee737b59d2a97f0ca3d73f0554be5f368231e79e094b7ce4d-prestodelta.xml.gz -rw-r--r-- 4,849 2022/09/23 10:49:35 repomd.xml
That causes issues for their downstream mirrors like mentioned in this email.
So we think it worth reporting to the infra team.
In case it helps, the TUNA mirror is in Beijing, China. Their IP addresses are
101.6.6.170 101.6.6.171 2402:f000:1:416:101:6:6:170 2402:f000:1:416:101:6:6:171
Hope someone can figure out the root cause and get this fixed properly.
Metadata Update from @smooge: - Issue assigned to smooge
The quick-mirror script does not seem to be working correctly for this system. It says nothing needs to be updated but many files and repodata were still stuck from a while ago.
This should be temporarily fixed. Trying to locate true problem.
Metadata Update from @zlopez: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: medium-gain, medium-trouble, ops
Doing a manual update seems to have 'fixed' the issue as all updates since Friday seem to have stayed in sync with upstream dl0X.fedoraproject.org. I will keep this open for this week to see if I can see why it broke (no errors in the logs that I can find).
This needs a nagios check on our systems to see when things aren't synced on them. This may help point out the problem for a manual fix.
Metadata Update from @smooge: - Issue close_status updated to: Initiative Worthy - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.