Summary: when the 4.19.6 kernel update went out, several to many mirrors deleted all of their 4.19.5 packages without picking up the new 4.19.6 packages. This resulted in users being unable to install kernel-related packages for at least the last 36 hours.
Narrative:
Setting up for a Kubecon demo yesterday afternoon (Pacific time), I spun up 5 Fedora Cloud Base servers in AWS. In order to run GlusterFS on these, I attempted to install kernel-modules and kernel-devel packages. I got this:
[MIRROR] kernel-devel-4.19.5-300.fc29.x86_64.rpm: Status code: 404 for http://mirror.cs.princeton.edu/pub/mirrors/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-devel-4.19.5-300.fc29 .x86_64.rpm [MIRROR] kernel-devel-4.19.5-300.fc29.x86_64.rpm: Status code: 404 for http://mirror.math.princeton.edu/pub/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-devel-4.19.5-300.fc29.x86_6 4.rpm [MIRROR] kernel-devel-4.19.5-300.fc29.x86_64.rpm: Status code: 404 for http://kdeforge2.unl.edu/mirrors/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-devel-4.19.5-300.fc29.x86_64.rp m
Multiple attempts to refresh metadata, clean the DNF cache, etc, were without result:
[root@ip-172-31-14-197 ~]# dnf clean all 29 files removed [root@ip-172-31-14-197 ~]# dnf update --refresh Fedora Modular 29 - x86_64 440 kB/s | 1.5 MB 00:03 Fedora Modular 29 - x86_64 - Updates 521 kB/s | 1.7 MB 00:03 Fedora 29 - x86_64 - Updates 10 MB/s | 16 MB 00:01 Fedora 29 - x86_64 24 MB/s | 62 MB 00:02 Dependencies resolved. Nothing to do. Complete! [root@ip-172-31-14-197 ~]# dnf install kernel Last metadata expiration check: 0:00:33 ago on Fri 07 Dec 2018 12:54:29 AM UTC. Dependencies resolved. =================================================================================================================================================================================================== Package Arch Version Repository Size =================================================================================================================================================================================================== Installing: kernel x86_64 4.19.5-300.fc29 updates 26 k Installing dependencies: kernel-modules x86_64 4.19.5-300.fc29 updates 28 M Transaction Summary =================================================================================================================================================================================================== Install 2 Packages Total download size: 28 M Installed size: 27 M
The only thing that did work (with help of user dowdle) was to manually edit fedora-updates.repo and have it direct to a specific mirror that was known to have 4.19.6 (missouri.edu). Even uncommenting baseurl and having dnf hit downloads.fedoraproject.org directly still resulted in getting a bad mirror and bad metadata.
This started about 16 hours after the 4.19.6 release and finished 24 hours after.
What appears to have happened is that several mirrors deleted their 4.19.5 rpms, while failing to update metadata, and possibly failing to sync the 4.19.6 rpms as well.
This was due to our aws mirror sync taking too long.
We are working on fixing that up...
CC: @codeblock
Metadata Update from @kevin: - Issue assigned to codeblock - Issue priority set to: Waiting on Assignee (was: Needs Review)
Lets track this all in issue #6022
Hopefully it's better now, but we need to make sure its all fixed...
Metadata Update from @kevin: - Issue close_status updated to: Duplicate - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.