Since a couple of hours ago I noticed that all C8S CI jobs started failing. The first fail looked like this:
============================================================================================== Package Arch Version Repository Size ============================================================================================== Installing: util-linux-core x86_64 2.38-0.3.el8 mrc0mmand-systemd-centos-ci-centos8-stream8 610 k replacing hardlink.x86_64 1:1.3-6.el8 Upgrading: centos-gpg-keys noarch 1:8-5.el8 extras 14 k centos-stream-repos noarch 8-5.el8 extras 20 k epel-next-release noarch 8-15.el8 epel 10 k epel-release noarch 8-15.el8 epel 23 k libblkid x86_64 2.38-0.3.el8 mrc0mmand-systemd-centos-ci-centos8-stream8 226 k libfdisk x86_64 2.38-0.3.el8 mrc0mmand-systemd-centos-ci-centos8-stream8 279 k libmount x86_64 2.38-0.3.el8 mrc0mmand-systemd-centos-ci-centos8-stream8 249 k libsmartcols x86_64 2.38-0.3.el8 mrc0mmand-systemd-centos-ci-centos8-stream8 190 k libuuid x86_64 2.38-0.3.el8 mrc0mmand-systemd-centos-ci-centos8-stream8 97 k util-linux x86_64 2.38-0.3.el8 mrc0mmand-systemd-centos-ci-centos8-stream8 2.6 M util-linux-user x86_64 2.38-0.3.el8 mrc0mmand-systemd-centos-ci-centos8-stream8 103 k Transaction Summary ============================================================================================== Install 1 Package Upgrade 11 Packages Total download size: 4.3 M Downloading Packages: (1/12): centos-gpg-keys-8-5.el8.noarch.rpm 879 kB/s | 14 kB 00:00 [MIRROR] centos-stream-repos-8-5.el8.noarch.rpm: Status code: 404 for http://mirror.ci.centos.org/centos/8-stream/extras/x86_64/os/Packages/centos-stream-repos-8-5.el8.noarch.rpm (IP: 172.19.0.7) [MIRROR] centos-stream-repos-8-5.el8.noarch.rpm: Status code: 404 for http://mirror2.ci.centos.org/centos/8-stream/extras/x86_64/os/Packages/centos-stream-repos-8-5.el8.noarch.rpm (IP: 172.19.0.7) [MIRROR] centos-stream-repos-8-5.el8.noarch.rpm: Status code: 404 for http://mirror3.ci.centos.org/centos/8-stream/extras/x86_64/os/Packages/centos-stream-repos-8-5.el8.noarch.rpm (IP: 172.19.0.7) [MIRROR] centos-stream-repos-8-5.el8.noarch.rpm: Status code: 404 for http://mirror4.ci.centos.org/centos/8-stream/extras/x86_64/os/Packages/centos-stream-repos-8-5.el8.noarch.rpm (IP: 172.19.0.7) [MIRROR] centos-stream-repos-8-5.el8.noarch.rpm: Status code: 404 for http://mirror.centos.org/centos/8-stream/extras/x86_64/os/Packages/centos-stream-repos-8-5.el8.noarch.rpm (IP: 172.19.0.7) [FAILED] centos-stream-repos-8-5.el8.noarch.rpm: No more mirrors to try - All mirrors were already tried without success The downloaded packages were saved in cache until the next successful transaction. You can remove cached packages by executing 'dnf clean packages'. Error: Error downloading packages: centos-stream-repos-8-5.el8.noarch: Cannot download, all mirrors were already tried without success
All subsequent jobs currently fail with:
Error: Problem: package centos-stream-repos-8-4.el8.noarch requires centos-gpg-keys = 1:8-4.el8, but none of the providers can be installed - cannot install both centos-gpg-keys-1:8-5.el8.noarch and centos-gpg-keys-1:8-4.el8.noarch - cannot install the best update candidate for package centos-stream-repos-8-4.el8.noarch - cannot install the best update candidate for package centos-gpg-keys-1:8-4.el8.noarch (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
All this is caused by the Extras repo shipping a newer version of both centos-gpg-keys and centos-stream-repos packages (centos-gpg-keys-8-5.el8.noarch.rpm and /centos-stream-repos-8-5.el8.noarch.rpm respectively). The centos-stream-repos package was since then deleted from the Extras repo, but the centos-gpg-keys remains there[0], causing fails during system installation/upgrade, as the BaseOS repo provides only centos-stream-repos-8-4.el8.noarch.rpm and centos-gpg-keys-8-4.el8.noarch.rpm [1][2].
centos-gpg-keys
centos-stream-repos
centos-gpg-keys-8-5.el8.noarch.rpm
/centos-stream-repos-8-5.el8.noarch.rpm
centos-stream-repos-8-4.el8.noarch.rpm
centos-gpg-keys-8-4.el8.noarch.rpm
[0] http://mirror.centos.org/centos/8-stream/extras/x86_64/os/Packages/centos-gpg-keys-8-5.el8.noarch.rpm [1] http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/centos-stream-repos-8-4.el8.noarch.rpm [2] http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/centos-gpg-keys-8-4.el8.noarch.rpm
I'm not sure if this is the same problem, but this is what I am seeing on recent c8s builds:
================================================================================ Package Arch Version Repository Size ================================================================================ Upgrading: centos-gpg-keys noarch 1:8-5.el8 extras 14 k Removing dependent packages: centos-stream-release noarch 8.6-1.el8 @baseos 26 k centos-stream-repos noarch 8-4.el8 @baseos 26 k setup noarch 2.12.2-6.el8 @baseos 708 k shadow-utils x86_64 2:4.6-16.el8 @baseos 3.9 M Transaction Summary ================================================================================ Running transaction Preparing : 1/1 Upgrading : centos-gpg-keys-1:8-5.el8.noarch 1/6 Erasing : shadow-utils-2:4.6-16.el8.x86_64 2/6 Erasing : setup-2.12.2-6.el8.noarch 3/6 warning: /etc/subuid saved as /etc/subuid.rpmsave warning: /etc/subgid saved as /etc/subgid.rpmsave warning: /etc/shadow saved as /etc/shadow.rpmsave warning: /etc/passwd saved as /etc/passwd.rpmsave warning: /etc/gshadow saved as /etc/gshadow.rpmsave warning: /etc/group saved as /etc/group.rpmsave Erasing : centos-stream-release-8.6-1.el8.noarch 4/6 Erasing : centos-stream-repos-8-4.el8.noarch 5/6 warning: file /etc/yum.repos.d/CentOS-Stream-Sources.repo: remove failed: No such file or directory warning: file /etc/yum.repos.d/CentOS-Stream-ResilientStorage.repo: remove failed: No such file or directory warning: file /etc/yum.repos.d/CentOS-Stream-RealTime.repo: remove failed: No such file or directory warning: file /etc/yum.repos.d/CentOS-Stream-PowerTools.repo: remove failed: No such file or directory warning: file /etc/yum.repos.d/CentOS-Stream-NFV.repo: remove failed: No such file or directory warning: file /etc/yum.repos.d/CentOS-Stream-Media.repo: remove failed: No such file or directory warning: file /etc/yum.repos.d/CentOS-Stream-HighAvailability.repo: remove failed: No such file or directory warning: file /etc/yum.repos.d/CentOS-Stream-Extras.repo: remove failed: No such file or directory warning: file /etc/yum.repos.d/CentOS-Stream-Debuginfo.repo: remove failed: No such file or directory warning: file /etc/yum.repos.d/CentOS-Stream-BaseOS.repo: remove failed: No such file or directory warning: file /etc/yum.repos.d/CentOS-Stream-AppStream.repo: remove failed: No such file or directory warning: file /etc/yum.repos.d: remove failed: No such file or directory warning: /etc/dnf/vars/stream saved as /etc/dnf/vars/stream.rpmsave warning: /etc/dnf/vars/contentdir saved as /etc/dnf/vars/contentdir.rpmsave Cleanup : centos-gpg-keys-1:8-4.el8.noarch 6/6 Verifying : centos-gpg-keys-1:8-5.el8.noarch 1/6 Verifying : centos-gpg-keys-1:8-4.el8.noarch 2/6 Verifying : centos-stream-release-8.6-1.el8.noarch 3/6 Verifying : centos-stream-repos-8-4.el8.noarch 4/6 Verifying : setup-2.12.2-6.el8.noarch 5/6 Verifying : shadow-utils-2:4.6-16.el8.x86_64 6/6 Upgraded: centos-gpg-keys-1:8-5.el8.noarch Removed: centos-stream-release-8.6-1.el8.noarch centos-stream-repos-8-4.el8.noarch setup-2.12.2-6.el8.noarch shadow-utils-2:4.6-16.el8.x86_64 Complete!
https://copr-be.cloud.fedoraproject.org/results/packit/cockpit-project-cockpit-machines-642/centos-stream-8-x86_64/03909100-cockpit-machines/builder-live.log.gz
Mismatch on pkgs versions on mirrors, but I just had a look and it seems @hughesjr has updated pkgs that should fix it but now in "staged" area :
/BaseOS/x86_64/os/Packages/centos-stream-repos-8-6.el8.noarch.rpm /BaseOS/x86_64/os/Packages/centos-gpg-keys-8-6.el8.noarch.rpm
So once it will go out to mirrors and these will be picked by mirror crawler and so fix this.
@ttomecek : What I'm really concerned about is more the issue about centos-stream-release itself being removed though
Interesting, it that particular case the 'protected packages' feature should kick in, as setup is a protected package:
setup
# dnf upgrade --allowerasing Last metadata expiration check: 0:02:50 ago on Tue Mar 29 08:07:50 2022. Error: Problem: The operation would result in removing the following protected packages: setup (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
As @hughesjr will probably not show up before several hours, what I manually did was removing centos-gpg-keys-8-5.el8.noarch.rpm that landed initially in Extras repo (and it was "fine" as long as centos-gpg-keys-1:8-5.el8 was there but it seems it was manually removed yesterday
Extras
centos-gpg-keys-1:8-5.el8
I ran quickly creatrerepo to regen repodata and I manually sync in the CI infra to verify that it's now working.
@mrc0mmand : can you "dnf clean all && dnf update" to confirm that workaround works for you ? Once hughesjr will have pushed the newer tree with 8-6 it will be back to normal
Metadata Update from @zlopez: - Issue tagged with: high-gain
As @hughesjr will probably not show up before several hours, what I manually did was removing centos-gpg-keys-8-5.el8.noarch.rpm that landed initially in Extras repo (and it was "fine" as long as centos-gpg-keys-1:8-5.el8 was there but it seems it was manually removed yesterday I ran quickly creatrerepo to regen repodata and I manually sync in the CI infra to verify that it's now working. @mrc0mmand : can you "dnf clean all && dnf update" to confirm that workaround works for you ? Once hughesjr will have pushed the newer tree with 8-6 it will be back to normal
That indeed works, thanks!
Metadata Update from @arrfab: - Issue assigned to amoloney - Issue tagged with: centos-build-pipeline, centos-stream, medium-trouble
As @arrfab notified me on IRC, both the CI and CentOS mirrors got new versions of centos-gpg-keys and centos-stream-repos) and the upgrades are working as expected:
14:40:36 Dependencies resolved. 14:40:36 ===================================================================================================== 14:40:36 Package Arch Version Repository Size 14:40:36 ===================================================================================================== 14:40:36 Installing: 14:40:36 util-linux-core x86_64 2.38-0.3.el8 mrc0mmand-systemd-centos-ci-centos8-stream8 610 k 14:40:36 replacing hardlink.x86_64 1:1.3-6.el8 14:40:36 Upgrading: 14:40:36 centos-gpg-keys noarch 1:8-6.el8 baseos 14 k 14:40:36 centos-stream-repos noarch 8-6.el8 baseos 20 k 14:40:36 epel-release noarch 8-15.el8 epel 23 k 14:40:36 iproute x86_64 5.15.0-4.el8 baseos 798 k 14:40:36 iputils x86_64 20180629-10.el8 baseos 148 k 14:40:36 libblkid x86_64 2.38-0.3.el8 mrc0mmand-systemd-centos-ci-centos8-stream8 226 k 14:40:36 libfdisk x86_64 2.38-0.3.el8 mrc0mmand-systemd-centos-ci-centos8-stream8 279 k 14:40:36 libmount x86_64 2.38-0.3.el8 mrc0mmand-systemd-centos-ci-centos8-stream8 249 k 14:40:36 libsmartcols x86_64 2.38-0.3.el8 mrc0mmand-systemd-centos-ci-centos8-stream8 190 k 14:40:36 libuuid x86_64 2.38-0.3.el8 mrc0mmand-systemd-centos-ci-centos8-stream8 97 k 14:40:36 openssl x86_64 1:1.1.1k-6.el8 baseos 709 k 14:40:36 openssl-libs x86_64 1:1.1.1k-6.el8 baseos 1.5 M 14:40:36 platform-python x86_64 3.6.8-46.el8 baseos 86 k 14:40:36 procps-ng x86_64 3.3.15-7.el8 baseos 329 k 14:40:36 python3-libs x86_64 3.6.8-46.el8 baseos 7.8 M 14:40:36 selinux-policy noarch 3.14.3-95.el8 baseos 645 k 14:40:36 selinux-policy-targeted noarch 3.14.3-95.el8 baseos 15 M 14:40:36 util-linux x86_64 2.38-0.3.el8 mrc0mmand-systemd-centos-ci-centos8-stream8 2.6 M 14:40:36 util-linux-user x86_64 2.38-0.3.el8 mrc0mmand-systemd-centos-ci-centos8-stream8 103 k 14:40:36 Installing weak dependencies: 14:40:36 epel-next-release noarch 8-15.el8 epel 10 k 14:40:36 14:40:36 Transaction Summary 14:40:36 ===================================================================================================== 14:40:36 Install 2 Packages 14:40:36 Upgrade 19 Packages ... 4:41:21 Upgraded: 14:41:21 centos-gpg-keys-1:8-6.el8.noarch 14:41:21 centos-stream-repos-8-6.el8.noarch 14:41:21 epel-release-8-15.el8.noarch 14:41:21 iproute-5.15.0-4.el8.x86_64 14:41:21 iputils-20180629-10.el8.x86_64 14:41:21 libblkid-2.38-0.3.el8.x86_64 14:41:21 libfdisk-2.38-0.3.el8.x86_64 14:41:21 libmount-2.38-0.3.el8.x86_64 14:41:21 libsmartcols-2.38-0.3.el8.x86_64 14:41:21 libuuid-2.38-0.3.el8.x86_64 14:41:21 openssl-1:1.1.1k-6.el8.x86_64 14:41:21 openssl-libs-1:1.1.1k-6.el8.x86_64 14:41:21 platform-python-3.6.8-46.el8.x86_64 14:41:21 procps-ng-3.3.15-7.el8.x86_64 14:41:21 python3-libs-3.6.8-46.el8.x86_64 14:41:21 selinux-policy-3.14.3-95.el8.noarch 14:41:21 selinux-policy-targeted-3.14.3-95.el8.noarch 14:41:21 util-linux-2.38-0.3.el8.x86_64 14:41:21 util-linux-user-2.38-0.3.el8.x86_64 14:41:21 Installed: 14:41:21 epel-next-release-8-15.el8.noarch util-linux-core-2.38-0.3.el8.x86_64 14:41:21 14:41:21 Complete!
Thanks!
Closing, per discussion with @amoloney
Metadata Update from @arrfab: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.