Issue #609: CentOS-8 stream contents giving 404 while pulling it from opendev mirror - centos-infra

centos-infra

#609 CentOS-8 stream contents giving 404 while pulling it from opendev mirror

Closed: Fixed 2 years ago by zlopez. Opened 2 years ago by chandankumar.

OpenDev AFS mirror rysnc centos stream 8 content from mirror.dal10.us.leaseweb.net mirror(https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/centos-mirror-update#L42) (It is near by TX).
The OpenStack jobs started failing with following error

error: Status code: 404 for https://mirror-int.dfw.rax.opendev.org/centos/8-stream/AppStream/x86_64/os/repodata/09255ba7c10e01afeb0d343667190f9c3e42d0a6099f887619abcb92ea0378db-filelists.xml.gz (IP: 10.209.161.66)

for that we have logged a bug in the launchpad: https://bugs.launchpad.net/tripleo/+bug/1958510

As per the OpenDev Infra investigation (http://lists.openstack.org/pipermail/openstack-discuss/2022-January/026831.html) it seems that there was
some (likely global) upheaval for CentOS Stream 8 package indices,

After checking with arrfab on #centos-devel, something is broken on the mirror.
I am also pasting the IRC discussion here for further investigation

08:09 <raukadah> on OpenStack side, we rsync cs8 content to openstack afs mirrors
08:09 <raukadah> we are seeing 404 on  Status code: 404 for https://mirror-int.ord.rax.opendev.org/centos/8-stream/AppStream/x86_64/os/repodata/8436c3dcc9cb11facf086b1da2dcc1be03e96045dd53c872486e9044ec58d95b-primary.xml.gz (IP: 10.209.128.57)
08:10 <raukadah> It rsyncs from mirror.dal10.us.leaseweb.net
08:10 <raukadah> we have seen this issue and it got resolved automatically but it is coming again today
08:10 --> RemiFedora (~RemiFedor@fedora/remi) has joined #centos-devel
08:10 <raukadah> can someone please look at mirror side
08:10 <raukadah> ? Thanks!
08:12 <arrfab> raukadah: we don't control when external mirrors are pulling from official mirrors either
08:13 <raukadah> arrfab: any specific offical mirror we can use to pull the content from?
08:13 <raukadah> to avoid the issue
08:13 <arrfab> we authorize public mirror to pull from us, but restricted for the others
08:13 --> vbenes (~vbenes@31.96.broadband16.iol.cz) has joined #centos-devel
08:13 <arrfab> why are you pulling from that particular leaseweb.net node ?
08:14 <arrfab> something you decided ?
08:14 <raukadah> based on this https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/centos-mirror-update#L42
08:15 <raukadah> there is no reason given but it somewhere in TX so we are using it
08:15 <raukadah> recently we have switched to facebook mirror for centos stream 9 content
08:15 <arrfab> probably worth sticking to one known and validated mirror that wouldn't have issues
08:16 <arrfab> but it seems there was a push during the night too
08:16 <arrfab> well, I take that back
08:16 <arrfab> nothing was pushed it seems
08:16 <arrfab> http://mirror.centos.org/centos/8-stream/COMPOSE_ID
08:17 <raukadah> http://lists.openstack.org/pipermail/openstack-discuss/2022-January/026831.html <- there was a  upheaval for CentOS Stream 8 package indices
08:17 <arrfab> which is strange .. as some pkgs clearly landed for BaseOS so worth verifying with hughesjr
08:19 <raukadah> arrfab: thank you for looking into this, will wait for hughesjr 's reply :-) 
08:19 <arrfab> something is b0rken, compose_id is from the past but even newer .iso images are pushed so you should ask hughesjr
08:19 <raukadah> In the mean time, we will ask openstack infra to switch to facebook mirror.
08:25 --> jcajka_ (~jcajka@2a02:8308:a106:d100::4ac5) has joined #centos-devel
08:26 <arrfab> raukadah: what can happen though is that mirror is pulling from one up2date mirror, then not current, and then again from up2date
08:26 <arrfab> until mirror network is stable enough
08:34 <raukadah> arrfab: ok, I will pass this info to the system admin, they might look into that .
08:37 <arrfab> raukadah: I meant that leasewe node
08:38 <arrfab> let me also circle that back as 8-stream is special versus previous setup, so let me verify that they still push correctly
08:38 <arrfab> as I think they don't wait enough, and so that can explain the issue you're having
08:39 <arrfab> raukadah: do you mind creating (for ref) a ticket on pagure.io/centos-infra/issues ? I'll take it from there and discuss with stream team (not using that tracket but I'll then create a jira ticket or bz, or else, for them)

Currently we have proposed a patch to switch to facebook mirror https://review.opendev.org/c/opendev/system-config/+/825446 (Facebook mirror might be validation one and does not have issue.)

Metadata Update from @arrfab:
- Issue assigned to bstinson
- Issue tagged with: centos-stream, doc, high-gain, low-trouble

2 years ago

arrfab commented 2 years ago

@bstinson we also had another report from @amoralej in #centos-stream about that the other day and I'd like to request some time for you, @hughesjr and me to sit together and review how 8-stream is pushed, as I'm not sure that the initial discussed plan is always followed

amoralej commented 2 years ago

I've found other issues related with CS8 mirrors. Sympthoms looks different but they may share root cause. On 2021-01-21 we hit several times followint errors between 00:00 and

From last night (2022-01-21 between 00:00 and 04:00 UTC):

DEBUG: Error:
DEBUG: Problem: cannot install the best update candidate for package glibc-2.28-184.el8.x86_64
DEBUG: - problem with installed package glibc-2.28-184.el8.x86_64
DEBUG: - glibc-gconv-extra-2.28-167.el8.i686 has inferior architecture
DEBUG: - package redhat-rpm-config-127-1.el8.noarch requires glibc-gconv-extra, but none of the providers can be installed
DEBUG: - glibc-gconv-extra-2.28-170.el8.i686 has inferior architecture
DEBUG: - glibc-gconv-extra-2.28-174.el8.i686 has inferior architecture
DEBUG: - glibc-gconv-extra-2.28-180.el8.i686 has inferior architecture
DEBUG: - glibc-gconv-extra-2.28-181.el8.i686 has inferior architecture
DEBUG: - package glibc-gconv-extra-2.28-167.el8.x86_64 requires glibc(x86-64) = 2.28-167.el8, but none of the providers can be installed
DEBUG: - package glibc-gconv-extra-2.28-170.el8.x86_64 requires glibc(x86-64) = 2.28-170.el8, but none of the providers can be installed
DEBUG: - package glibc-gconv-extra-2.28-174.el8.x86_64 requires glibc(x86-64) = 2.28-174.el8, but none of the providers can be installed
DEBUG: - package glibc-gconv-extra-2.28-180.el8.x86_64 requires glibc(x86-64) = 2.28-180.el8, but none of the providers can be installed
DEBUG: - package glibc-gconv-extra-2.28-181.el8.x86_64 requires glibc(x86-64) = 2.28-181.el8, but none of the providers can be installed
DEBUG: - cannot install both glibc-2.28-167.el8.x86_64 and glibc-2.28-184.el8.x86_64
DEBUG: - cannot install both glibc-2.28-170.el8.x86_64 and glibc-2.28-184.el8.x86_64
DEBUG: - cannot install both glibc-2.28-174.el8.x86_64 and glibc-2.28-184.el8.x86_64
DEBUG: - cannot install both glibc-2.28-180.el8.x86_64 and glibc-2.28-184.el8.x86_64
DEBUG: - cannot install both glibc-2.28-181.el8.x86_64 and glibc-2.28-184.el8.x86_64
DEBUG: - problem with installed package redhat-rpm-config-125-1.el8.noarch
DEBUG: - cannot install the best update candidate for package redhat-rpm-config-125-1.el8.noarch

Also, we got some occurrences of following error from 21-01-2022 to 13:30-14:00:

DEBUG: Error:
DEBUG: Problem 1: problem with installed package platform-python-devel-3.6.8-44.el8.x86_64
DEBUG: - cannot install the best update candidate for package platform-python-devel-3.6.8-44.el8.x86_64
DEBUG: - nothing provides python3-libs(x86-64) = 3.6.8-45.el8 needed by platform-python-devel-3.6.8-45.el8.x86_64
DEBUG: Problem 2: problem with installed package glibc-gconv-extra-2.28-181.el8.x86_64
DEBUG: - cannot install the best update candidate for package glibc-gconv-extra-2.28-181.el8.x86_64
DEBUG: - nothing provides glibc-common = 2.28-184.el8 needed by glibc-gconv-extra-2.28-184.el8.x86_64
DEBUG: - nothing provides glibc(x86-64) = 2.28-184.el8 needed by glibc-gconv-extra-2.28-184.el8.x86_64

amoralej commented 2 years ago

One more data point at 2022-01-21 15:12 (UTC):

arrfab commented 2 years ago

@hughesjr , @amoralej , @bstinson : is that now ok and fixed at the push side ? If so can we close the ticket ?

amoralej commented 2 years ago

One more issue from yesterday 2-Feb-2022 with extras repos in CS8. We observed issues for about 1 hour:

Metadata Update from @zlopez:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

Metadata

Assignee

bstinson

Tags

Blocking

None

Depending on

None

Priority

Needs Review

centos-infra

Source Code

#609 CentOS-8 stream contents giving 404 while pulling it from opendev mirror Closed: Fixed 2 years ago by zlopez. Opened 2 years ago by chandankumar.

Metadata

centos-stream low-trouble high-gain doc

#609 CentOS-8 stream contents giving 404 while pulling it from opendev mirror

Closed: Fixed 2 years ago by zlopez. Opened 2 years ago by chandankumar.