To maintain a local (private) mirror of Fedora 42 I rsync the I3D mirror. However when doing so, one does not get a usable mirror. For example I just rsynced my local mirror with the public I3D mirror and tried to update my Fedora 42 installation from my local (private) mirror and that results in:
[160/215] perl-Module-CoreList-1:5.20250923-1.fc42.noarch 100% | 872.4 KiB/s | 93.3 KiB | 00m00s [161/215] php-cli-0:8.4.13-1.fc42.x86_64 100% | 1.9 MiB/s | 2.3 MiB | 00m01s >>> Downloading successful, but checksum doesn't match. Calculated: ffdca20f2fc76c17851672f8c831e75a02ddf61f2b81d12bf31b2a8e3279b38d(sha256) Expected: a1186ac6d5a944b21e9143f8b839c3d231f72f6c75efc99838d1822516ee8d7b(sha256) - ht >>> Downloading successful, but checksum doesn't match. Calculated: ffdca20f2fc76c17851672f8c831e75a02ddf61f2b81d12bf31b2a8e3279b38d(sha256) Expected: a1186ac6d5a944b21e9143f8b839c3d231f72f6c75efc99838d1822516ee8d7b(sha256) - ht >>> Downloading successful, but checksum doesn't match. Calculated: ffdca20f2fc76c17851672f8c831e75a02ddf61f2b81d12bf31b2a8e3279b38d(sha256) Expected: a1186ac6d5a944b21e9143f8b839c3d231f72f6c75efc99838d1822516ee8d7b(sha256) - ht >>> Downloading successful, but checksum doesn't match. Calculated: ffdca20f2fc76c17851672f8c831e75a02ddf61f2b81d12bf31b2a8e3279b38d(sha256) Expected: a1186ac6d5a944b21e9143f8b839c3d231f72f6c75efc99838d1822516ee8d7b(sha256) - ht >>> No more mirrors to try - All mirrors were already tried without success
This also happens with other mirrors. In fact it happens with all mirrors I've tried. I've also tried the NLUUG / Surftnet mirror, the University Stuttgart and a couple of others and they all lead to different RPM checksum failures or DNF repo metadata checksum failures.
For example a couple of days ago I tried to rsync the University Stuttgart mirror instead of I3D. DNF then failed because it tried to download de648445a00e1ad54af3cb73ea60afb8155a5696dd3b2fec5134a604f27e7042-primary.xml.zck, which doesn’t exist so it gets an HTTP 404. When I looked at http://ftp-stud.hs-esslingen.de/pub/fedora/linux/updates/42/Everything/x86_64/repodata/repomd.xml I could see the repo metadata index referenced de648445a00e1ad54af3cb73ea60afb8155a5696dd3b2fec5134a604f27e7042 (which is why DNF tried to download it) but when I looked at the repo metada directory: Index of /pub/fedora/linux/updates/42/Everything/x86_64/repodata I could see there was no such file. So as far as I can tell there is nothing wrong with my rsync-ed mirror in the sense that the upstream mirror is botched, so my rsync-ed mirror is botched as well.
de648445a00e1ad54af3cb73ea60afb8155a5696dd3b2fec5134a604f27e7042-primary.xml.zck
de648445a00e1ad54af3cb73ea60afb8155a5696dd3b2fec5134a604f27e7042
I've read the Infrastructure/Mirroring documentation but as far as I can tell I'm doing what I should be doing.
For reference; I initially opened this thread.
The output from the console seems to indicate that this is about the file php-cli-8.4.13-1.fc42.x86_64.rpm, right?
php-cli-8.4.13-1.fc42.x86_64.rpm
# curl -s http://ftp-stud.hs-esslingen.de/pub/Mirrors/dl.fedoraproject.org/fedora/linux/updates/42/Everything/x86_64/Packages/p/php-cli-8.4.13-1.fc42.x86_64.rpm | sha256sum a1186ac6d5a944b21e9143f8b839c3d231f72f6c75efc99838d1822516ee8d7b -
Gives me the correct sha256sum. That is also what your output says as "Expected:".
I also get the same checksum from the primary mirror:
# curl -s https://dl.fedoraproject.org/pub/fedora/linux/updates/42/Everything/x86_64/Packages/p/php-cli-8.4.13-1.fc42.x86_64.rpm | sha256sum a1186ac6d5a944b21e9143f8b839c3d231f72f6c75efc99838d1822516ee8d7b -
One of the mirrors you mentioned has the file with the correct checksum and the primary mirror also.
At the same time you are also reporting about repodata inconsistency.
# curl -s "https://mirrors.fedoraproject.org/metalink?repo=updates-released-f42&arch=x86_64" | grep sha256 <hash type="sha256">e012b07c9a2dee10ee2e8ab7a37b81e124462f3c1e2f0f2c374f9730e1609be9</hash> <hash type="sha256">d2b7944d8c46b82af7d1d0907bb89edaec39e93c409eff4e1759c712b842ec02</hash> <hash type="sha256">67a202638652573f8240f8cfab66d1bf54ff1589811eff022033b74021d70577</hash> <hash type="sha256">8b3768fb93db997ef27a4d2b4a9c1bcbda06ac0ce81a3cb86a1678679ee4a445</hash> # curl -s http://ftp-stud.hs-esslingen.de/pub/Mirrors/dl.fedoraproject.org/fedora/linux/updates/42/Everything/x86_64/repodata/repomd.xml | sha256sum e012b07c9a2dee10ee2e8ab7a37b81e124462f3c1e2f0f2c374f9730e1609be9
This also looks correct.
My guess is that this your are seeing just some rsync in progress inconsistency. Which is unfortunate but that happens with rsync mirroring sometimes. Especially if the changes to the repository are huge. Not sure if that is the case. But that is why DNF falls back to another mirror.
Sorry, I said my mirror was synced with I3D (so also not ftp-stud.hs-esslingen.de ;-) ) but it's actually synced with NLUUG. You can see the NLUUG mirror has this botched file:
ftp-stud.hs-esslingen.de
# curl -s https://ftp.nluug.nl/pub/os/Linux/distr/fedora/linux/updates/42/Everything/x86_64/Packages//p/php-cli-8.4.13-1.fc42.x86_64.rpm | sha256sum ffdca20f2fc76c17851672f8c831e75a02ddf61f2b81d12bf31b2a8e3279b38d -
So at this moment the NLUUG mirror does not seem to be consistent.
I'm sure that if I now change to I3D, Stuttgart, etc. I will get some other checksum failure on some other RPM because thats what's happening all the time for weeks now.
Re-syncing those mirrors later on the day also don't fix this issue. So I don't see how this can be "progress inconsistency".
If there is a file broken on a mirror your best way to solve it is to contact the mirror. The file looks correct on the primary mirror so it would help the mirror admin to know it.
If so many mirrors have broken files all the time I would begin to question the sanity / quality of the primary mirror. I think the only thing keeping the mirrors afloat right now is DNF automatically retrying when it encounters a botched file.
You are the first to report problems with RPMs on different mirrors.
Please provide a list of broken RPMs on which mirror so that we can start to investigate.
I'm willing to put in the work. If I have locally rsync-ed a mirror, is there any way I can check if the metadata checksums with all the RPM's? Is there any tooling which does that out of the box?
You could do something like this:
$ wget https://dl.fedoraproject.org/pub/fedora/linux/updates/42/Everything/x86_64/Packages/p/php-cli-8.4.13-1.fc42.x86_64.rpm Saving 'php-cli-8.4.13-1.fc42.x86_64.rpm' HTTP response 200 OK [https://dl.fedoraproject.org/pub/fedora/linux/updates/42/Everything/x86_64/Packages/p/php-cli-8.4.13-1.fc42.x86_64.rpm] php-cli-8.4.13-1.fc4 100% [==========================================================================================================================================================================================>] 2.32M 7.80MB/s [Files: 1 Bytes: 2.32M [1.88MB/s] Redirects: 0 Todo: 0 Errors: 0 ] $ rpm -Kv php-cli-8.4.13-1.fc42.x86_64.rpm php-cli-8.4.13-1.fc42.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 105ef944: NOKEY Header SHA256 digest: OK Header SHA1 digest: OK Payload SHA256 digest: OK MD5 digest: OK
Something like that checks the package against its internal checksum.
Metadata Update from @phsmoura: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: low-gain, low-trouble, ops
Worth mentioning that mirroring with https://pagure.io/quick-fedora-mirror if you can will be quicker than rsync by itself and also will check all the checksums of the repomd.xml files when syncing.
I noticed the corrupt files all consist of only zero's (binary zero's, not the ASCII zero character). They do have the correct file size, their contents just consist of only zero's. This is also why rsync doesn't correct this corruption on subsequent syncs. Because rsync only looks at file modification time and the size. That is, unless the --checksum flag is specified. However the --checksum flag is way too slow for large mirrors like Fedora and these public rsync mirror servers usually don't accept the --checksum flag. Meaning this silent data corruption is only fixed if there happens to be a new version of the package to be published.
Personally, I never realized rsync is that susceptible to silent data corruption. I always assumed that even if there was some sort of bitflip subsequent rsync's would fix that. This turns out not to be the case.
I've mailed with the NLUUG mirror administrator and they removed the corrupt php-cli-8.4.13-1.fc42.x86_64.rpm package after which they resynced their mirror. I then resynced with the NLUUG mirror and used the command below to spot files which consist of only zero's:
----8<-------- $ find /export/pub/fedora/42/testing/.zfs/snapshot/2025-10-27_15:05 -type f ! -empty -exec grep -zvFxL '' {} + /export/pub/fedora/42/testing/.zfs/snapshot/2025-10-27_15:05/Everything/x86_64/updates/Packages/r/remmina-1.4.41-1.fc42.x86_64.rpm /export/pub/fedora/42/testing/.zfs/snapshot/2025-10-27_15:05/Everything/x86_64/updates/Packages/o/openjph-0.21.5-1.fc42.x86_64.rpm /export/pub/fedora/42/testing/.zfs/snapshot/2025-10-27_15:05/Everything/x86_64/updates/Packages/p/php-8.4.13-1.fc42.x86_64.rpm /export/pub/fedora/42/testing/.zfs/snapshot/2025-10-27_15:05/Everything/x86_64/updates/Packages/p/php-bcmath-8.4.13-1.fc42.x86_64.rpm /export/pub/fedora/42/testing/.zfs/snapshot/2025-10-27_15:05/Everything/x86_64/updates/Packages/z/zeitfetch-0.1.14-3.fc42.x86_64.rpm ----8<--------
To verify this wasn't just locally I ran:
----8<-------- curl -s https://ftp.nluug.nl/pub/os/Linux/distr/fedora/linux/updates/42/Everything/x86_64/Packages/z/zeitfetch-0.1.14-3.fc42.x86_64.rpm | sha256sum 92f8adac20f888156d3eb25296f856a1b5af664dc514df987c8cf7792d9311e4 - ----8<--------
I also downloaded zeitfetch-0.1.14-3.fc42.x86_64.rpm for the NLUUG mirror on my desktop and manually confirmed the file consisted of only zero's.
By now the NLUUG mirror has also removed these files.
I have no idea how these initial zero filled files have been created on these mirrors. From what I read in various places there are circumstances under which rsync can create these 0 filled files while doing a sync. Without the --checksum this silent corruption won't fix itself unless an updated version of the package is released.
--checksum
The insight here might be that using quick-fedora-mirror is the only reliable way of doing a sync (since it checks checksums). Or alternatively doing a find /repo -type f ! -empty -exec grep -zvFxL '' {} + after each rsync to see if no "zero filled" files have been pulled in.
quick-fedora-mirror
find /repo -type f ! -empty -exec grep -zvFxL '' {} +
I am wondering to what extend you folks were aware of this behavior of rsync? For me it was kind of a surprise actually.
Sure, thats all understandable to me, except for how the file got that way in the first place. They would have had to rsync the file initially and have it be all zeros, but I don't understand how that could happen. The files are fine on the master mirror... so under what condition did the zeros file get initially created?
We do recommend everyone use quick-fedora-mirror... it's a lot better for you and for us.
I'm not sure what we can do to prevent this aside from that unless we can figure out a cause. ;(
I guess lets close this now? Feel free to reopen if there's anything further we can do.
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.