Giving one repository as an example for which I see these errors:
[centos-kmods-testing] name=CentOS Stream $releasever - Kmods - Testing baseurl=https://buildlogs.centos.org/centos/$stream/kmods/$basearch/packages-main/ gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Kmods gpgcheck=1 repo_gpgcheck=0 metadata_expire=6h enabled=0
$stream is 9-stream and $basearch is x86_64.
The error is:
[MIRROR] kmod-3w-9xxx-5.14.0~210-1.el9s.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated: f655265e19d0a6faaebdb95e14754e248a919160d9487aa3c4e479f342580a3c(sha256) Expected: 1fd53d15a58dbaf62c327218e474889b6d82b5de70310491d6a8f97f08b56b14(sha256)
Downloading the file manually using
wget https://buildlogs.centos.org/centos/9-stream/kmods/x86_64/packages-main/Packages/k/kmod-3w-9xxx-5.14.0~210-1.el9s.x86_64.rpm
I get a file with the following sha256:
SHA256 (kmod-3w-9xxx-5.14.0~210-1.el9s.x86_64.rpm) = f655265e19d0a6faaebdb95e14754e248a919160d9487aa3c4e479f342580a3c
I.e., not the expected checksum.
My observations from NFV SIG where we are hitting similar issue:
ovn22.12-22.12.0-34.el9s openvswitch-selinux-extra-policy-1.0-31.el9s ovn22.09-22.09.0-31.el9s openvswitch3.1-3.1.0-17.el9s
Given that, both kmod (correct me if i'm wrong) and nvf sig are building packages in C9S and RHEL9 BR, i wonder if that may be related....
Also observed in cloudsig testing repo from https://buildlogs.centos.org/centos/9-stream/cloud/x86_64/openstack-antelope/ :
[MIRROR] libsodium-1.0.18-7.el9s.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated: 1f2a5abe61bdd68bcbee962cf9a3b81158d1e5b987b24f421cb902924d75bb89(sha256) Expected: 8147c75c0454e7e58f5ac3a351e08b8b734ac3ddef803885c085515523ad8327(sha256) (14/194): openstack-barbican-worker-16.0.0-2.el9s.noarch.rpm 2.1 kB/s | 8.4 kB 00:04 [MIRROR] python-oslo-cache-lang-3.3.1-1.el9s.noarch.rpm: Downloading successful, but checksum doesn't match. Calculated: 7344f476bdf799f62457a6a34bfee00a845656eca4b19429d25e97dfde1c2b38(sha256) Expected: 125622fdeb3a2e02aa4cdcbc25a9a631677aba4c4de97d823ddb3c5230ecb714(sha256) [MIRROR] python-oslo-concurrency-lang-5.1.1-2.el9s.noarch.rpm: Downloading successful, but checksum doesn't match. Calculated: cd53374a430a54df57a6482c2805e931dfd0030778a1a16fcf6e8b97a2dda69c(sha256) Expected: e0c44f79afc2568b30e85f3aeb128f375129694c77166bfa07a73c3bacd6bb53(sha256) [MIRROR] python-oslo-cache-lang-3.3.1-1.el9s.noarch.rpm: Downloading successful, but checksum doesn't match. Calculated: 7344f476bdf799f62457a6a34bfee00a845656eca4b19429d25e97dfde1c2b38(sha256) Expected: 125622fdeb3a2e02aa4cdcbc25a9a631677aba4c4de97d823ddb3c5230ecb714(sha256)
Thanks to @tkopecek , we found that koji distrepo tasks is using createrepo_c with some caching options, so as packages for one tag had the same ENVR but different checksums (as now being signed) the packages were the correct ones but generated repodata were still referencing cached checksums.
a manual clean-up to delete previously created repos-dist layout for tag cloud9s-openstack-antelope-testing confirmed that generated repodata was now correct :
cloud9s-openstack-antelope-testing
sha256sum Packages/l/libsodium-1.0.18-7.el9s.x86_64.rpm 1f2a5abe61bdd68bcbee962cf9a3b81158d1e5b987b24f421cb902924d75bb89 Packages/l/libsodium-1.0.18-7.el9s.x86_64.rpm
From primary.xml.gz repodata file :
<package type="rpm"> <name>libsodium</name> <arch>x86_64</arch> <version epoch="0" ver="1.0.18" rel="7.el9s"/> <checksum type="sha256" pkgid="YES">1f2a5abe61bdd68bcbee962cf9a3b81158d1e5b987b24f421cb902924d75bb89</checksum> <summary>The Sodium crypto library</summary>
So we believe that issue was found and we manually deleted all previous cached copies of generated repos-dist and triggered again in the queue all -testing tags.
FWIW, @tkopecek also created https://pagure.io/koji/issue/3829 as a RFE to have a new option to force a koji dist-repo task to force a metadata regen too
koji dist-repo
Metadata Update from @arrfab: - Issue assigned to arrfab
Metadata Update from @arrfab: - Issue priority set to: 🔥 Urgent 🔥 (was: Needs Review) - Issue tagged with: cbs, centos-build-pipeline, high-gain, medium-trouble
Just wanted to report that I do not see the issues I reported in the initial post anymore. Thanks!
I'm not closing this issue myself as others have reported similar issues here as well and want to give them a chance to report back.
thanks for the feedback. As we got also feedback on irc, I'll close this one
Metadata Update from @arrfab: - Issue close_status updated to: Fixed with Explanation - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.