#7428 Mirror network is failing on kernel updates
Closed: Duplicate 5 years ago Opened 5 years ago by jberkus.

Summary: when the 4.19.6 kernel update went out, several to many mirrors deleted all of their 4.19.5 packages without picking up the new 4.19.6 packages. This resulted in users being unable to install kernel-related packages for at least the last 36 hours.

Narrative:

Setting up for a Kubecon demo yesterday afternoon (Pacific time), I spun up 5 Fedora Cloud Base servers in AWS. In order to run GlusterFS on these, I attempted to install kernel-modules and kernel-devel packages. I got this:

[MIRROR] kernel-devel-4.19.5-300.fc29.x86_64.rpm: Status code: 404 for http://mirror.cs.princeton.edu/pub/mirrors/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-devel-4.19.5-300.fc29
.x86_64.rpm                                                                                                                                                                                       
[MIRROR] kernel-devel-4.19.5-300.fc29.x86_64.rpm: Status code: 404 for http://mirror.math.princeton.edu/pub/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-devel-4.19.5-300.fc29.x86_6
4.rpm                                                                                                                                                                                             
[MIRROR] kernel-devel-4.19.5-300.fc29.x86_64.rpm: Status code: 404 for http://kdeforge2.unl.edu/mirrors/fedora/linux/updates/29/Everything/x86_64/Packages/k/kernel-devel-4.19.5-300.fc29.x86_64.rp
m      

Multiple attempts to refresh metadata, clean the DNF cache, etc, were without result:

[root@ip-172-31-14-197 ~]# dnf clean all
29 files removed
[root@ip-172-31-14-197 ~]# dnf update --refresh
Fedora Modular 29 - x86_64                                                                                                                                         440 kB/s | 1.5 MB     00:03    
Fedora Modular 29 - x86_64 - Updates                                                                                                                               521 kB/s | 1.7 MB     00:03    
Fedora 29 - x86_64 - Updates                                                                                                                                        10 MB/s |  16 MB     00:01    
Fedora 29 - x86_64                                                                                                                                                  24 MB/s |  62 MB     00:02    
Dependencies resolved.
Nothing to do.
Complete!
[root@ip-172-31-14-197 ~]# dnf install kernel
Last metadata expiration check: 0:00:33 ago on Fri 07 Dec 2018 12:54:29 AM UTC.
Dependencies resolved.
===================================================================================================================================================================================================
 Package                                           Arch                                      Version                                              Repository                                  Size
===================================================================================================================================================================================================
Installing:
 kernel                                            x86_64                                    4.19.5-300.fc29                                      updates                                     26 k
Installing dependencies:
 kernel-modules                                    x86_64                                    4.19.5-300.fc29                                      updates                                     28 M

Transaction Summary
===================================================================================================================================================================================================
Install  2 Packages

Total download size: 28 M
Installed size: 27 M

The only thing that did work (with help of user dowdle) was to manually edit fedora-updates.repo and have it direct to a specific mirror that was known to have 4.19.6 (missouri.edu). Even uncommenting baseurl and having dnf hit downloads.fedoraproject.org directly still resulted in getting a bad mirror and bad metadata.

This started about 16 hours after the 4.19.6 release and finished 24 hours after.

What appears to have happened is that several mirrors deleted their 4.19.5 rpms, while failing to update metadata, and possibly failing to sync the 4.19.6 rpms as well.


This was due to our aws mirror sync taking too long.

We are working on fixing that up...

CC: @codeblock

Metadata Update from @kevin:
- Issue assigned to codeblock
- Issue priority set to: Waiting on Assignee (was: Needs Review)

5 years ago

Lets track this all in issue #6022

Hopefully it's better now, but we need to make sure its all fixed...

Metadata Update from @kevin:
- Issue close_status updated to: Duplicate
- Issue status updated to: Closed (was: Open)

5 years ago

Login to comment on this ticket.

Metadata