#680 Source repos on vault.centos.org are unreliable
Closed: Fixed 2 years ago by arrfab. Opened 2 years ago by mrc0mmand.

Hello!

In the recent weeks I noticed suspiciously high frequency of issues with downloading metadata/packages from the CentOS 8-stream source repository hosted on vault.centos.org:

Today (2022-02-28) @ ~20:30 CET

enabling appstream-source repository
enabling baseos-source repository
enabling extras-source repository
enabling powertools-source repository
enabling epel-source repository
enabling epel-modular-source repository
enabling epel-next-source repository
CentOS Stream 8 - BaseOS - Source               0.0  B/s |   0  B     02:16    
Errors during downloading metadata for repository 'baseos-source':
  - Curl error (28): Timeout was reached for http://vault.centos.org/centos/8-stream/BaseOS/Source/repodata/repomd.xml [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds]
  - Curl error (28): Timeout was reached for http://vault.centos.org/centos/8-stream/BaseOS/Source/repodata/repomd.xml [Connection timed out after 30000 milliseconds]
Error: Failed to download metadata for repo 'baseos-source': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
[cmd_retry] Command 'dnf -y builddep systemd' failed (EC: 1) [try 5/5]

Today (2022-02-28) @ ~01:30 CET

enabling appstream-source repository
enabling baseos-source repository
enabling extras-source repository
enabling powertools-source repository
enabling epel-source repository
enabling epel-modular-source repository
enabling epel-next-source repository
CentOS Stream 8 - BaseOS - Source               0.0  B/s |   0  B     02:22    
Errors during downloading metadata for repository 'baseos-source':
  - Curl error (28): Timeout was reached for http://vault.centos.org/centos/8-stream/BaseOS/Source/repodata/ba3596678c500f6cd43317443da5e42123b3655de0f949e5031f7becf4e3d605-filelists.xml.gz [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds]
  - Curl error (28): Timeout was reached for http://vault.centos.org/centos/8-stream/BaseOS/Source/repodata/7b24adb0b51a1e8f5ce2263d108ad9c478a7bf1f36b14b940265a95c62a7e1f9-primary.xml.gz [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds]
  - Curl error (28): Timeout was reached for http://vault.centos.org/centos/8-stream/BaseOS/Source/repodata/7b24adb0b51a1e8f5ce2263d108ad9c478a7bf1f36b14b940265a95c62a7e1f9-primary.xml.gz [Connection timed out after 30000 milliseconds]
Error: Failed to download metadata for repo 'baseos-source': Yum repo downloading error: Downloading error(s): repodata/7b24adb0b51a1e8f5ce2263d108ad9c478a7bf1f36b14b940265a95c62a7e1f9-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/ba3596678c500f6cd43317443da5e42123b3655de0f949e5031f7becf4e3d605-filelists.xml.gz - Cannot download, all mirrors were already tried without success
[cmd_retry] Command 'dnf -y builddep systemd' failed (EC: 1) [try 5/5]

2022-02-25 @ ~01:30

enabling appstream-source repository
enabling baseos-source repository
enabling extras-source repository
enabling powertools-source repository
enabling epel-source repository
enabling epel-modular-source repository
enabling epel-next-source repository
CentOS Stream 8 - AppStream - Source            6.7 kB/s | 1.2 MB     02:55    
CentOS Stream 8 - PowerTools - Source           1.0 kB/s | 216 kB     03:31    
CentOS Stream 8 - Extras - Source               0.0  B/s |   0  B     02:02    
Errors during downloading metadata for repository 'extras-source':
  - Curl error (28): Timeout was reached for http://vault.centos.org/centos/8-stream/extras/Source/repodata/repomd.xml [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds]
  - Curl error (28): Timeout was reached for http://vault.centos.org/centos/8-stream/extras/Source/repodata/repomd.xml [Connection timed out after 30000 milliseconds]
Error: Failed to download metadata for repo 'extras-source': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
[cmd_retry] Command 'dnf -y builddep systemd' failed (EC: 1) [try 5/5]

2022-02-24 @ ~02:15

enabling appstream-source repository
enabling baseos-source repository
enabling extras-source repository
enabling powertools-source repository
enabling epel-source repository
enabling epel-modular-source repository
enabling epel-next-source repository
CentOS Stream 8 - BaseOS - Source               2.9 kB/s | 607 kB     03:31    
CentOS Stream 8 - AppStream - Source            3.8 kB/s | 745 kB     03:17    
Errors during downloading metadata for repository 'appstream-source':
  - Curl error (28): Timeout was reached for http://vault.centos.org/centos/8-stream/AppStream/Source/repodata/d10622cf0d014ded9f93a37f93fd21f2f3031c3993b52e1bec7ffb0c17d0bf28-filelists.xml.gz [Connection timed out after 30000 milliseconds]
  - Curl error (28): Timeout was reached for http://vault.centos.org/centos/8-stream/AppStream/Source/repodata/repomd.xml [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds]
  - Curl error (28): Timeout was reached for https://vault.centos.org/centos/8-stream/AppStream/Source/repodata/d10622cf0d014ded9f93a37f93fd21f2f3031c3993b52e1bec7ffb0c17d0bf28-filelists.xml.gz [Connection timed out after 30386 milliseconds]
Error: Failed to download metadata for repo 'appstream-source': Yum repo downloading error: Downloading error(s): repodata/d10622cf0d014ded9f93a37f93fd21f2f3031c3993b52e1bec7ffb0c17d0bf28-filelists.xml.gz - Cannot download, all mirrors were already tried without success
[cmd_retry] Command 'dnf -y builddep systemd' failed (EC: 1) [try 5/5]

and many more...

I don't remember the source repo being that unreliable before, so maybe there's an actual issue hidden behind this? I wrapped the build-dep command in a loop which retries the download up to 5 times (with a multiplicative delay, which makes the loop run up to ~620 seconds in case all attempts fail) to avoid transient network issues, but it's apparently still not enough.


The change is that with CentOS-8 being removed a lot of traffic is now hitting vault.centos.org and it is not able to keep up. This server/service is mainly meant for one time downloads, and is highly overloaded. For regular usage it would be better if you made a local mirror of the site that you rely on for your system.

I see, thank you. I'm hitting this in CentOS CI and iirc @arrfab had some plans for a mirror of these repositories for the CI environment - not sure what's the status though.

I guess for the time being I can replace the dnf builddep systemd step by cloning the systemd centos-git repo & using the specfile directly, or, in the worst case, use an explicit list of dependencies) thus completely avoiding the sources repo, which should help in (at least) my use case.

Metadata Update from @arrfab:
- Issue tagged with: centos-common-infra, medium-gain, medium-trouble

2 years ago

@mrc0mmand our monitoring platform confirms that part of the mirror network is under kind of "abuse" at the moment, and vault.centos.org seems to be part of the victims, so yeah, it was flapping. It has to be investigated when some engineers will have time to work on this and will be allocated.

WRT your RFE for internal vault.centos.org in the CI infra, yes, that would be a good idea and what we tried to accomplish (like we have internal mirror.centos.org vhost there).
Worth knowing that we don't have machine with enough storage space (~4TiB needed) and still under warranty to consider configuring one internal vault mirror. We can eventually try to host one on older node and if it fails, just redirect to the public vault.centos.org pool. Would you mind creating though a different ticket for that RFE, which is different from solving the existing issue on vault.centos.org. Thanks :)

@mrc0mmand our monitoring platform confirms that part of the mirror network is under kind of "abuse" at the moment, and vault.centos.org seems to be part of the victims, so yeah, it was flapping. It has to be investigated when some engineers will have time to work on this and will be allocated.

WRT your RFE for internal vault.centos.org in the CI infra, yes, that would be a good idea and what we tried to accomplish (like we have internal mirror.centos.org vhost there).
Worth knowing that we don't have machine with enough storage space (~4TiB needed) and still under warranty to consider configuring one internal vault mirror. We can eventually try to host one on older node and if it fails, just redirect to the public vault.centos.org pool. Would you mind creating though a different ticket for that RFE, which is different from solving the existing issue on vault.centos.org. Thanks :)

Sure, https://pagure.io/centos-infra/issue/687 :-)

Metadata Update from @arrfab:
- Issue assigned to arrfab

2 years ago

Should (partially) be closed by #682 , and have reopened #687 as a feature-request for internal mirror and we can then have a look at this RFE and find a place to host it (if we still have storage). But initial issue (this ticket) should be now closed

Metadata Update from @arrfab:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)

2 years ago

Not sure what went wrong, but for the past hour or so all requests to vault.centos.org fail with 403:

enabling appstream-source repository
enabling baseos-source repository
enabling extras-source repository
enabling powertools-source repository
enabling epel-source repository
enabling epel-modular-source repository
enabling epel-next-source repository
CentOS Stream 8 - BaseOS - Source               4.2 kB/s | 919  B     00:00    
Errors during downloading metadata for repository 'baseos-source':
  - Status code: 403 for https://vault.centos.org/centos/8-stream/BaseOS/Source/repodata/repomd.xml (IP: 18.67.76.55)
Error: Failed to download metadata for repo 'baseos-source': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
[cmd_retry] Command 'dnf -y builddep systemd' failed (EC: 1) [try 5/5]

Manual curl:

$ curl -s https://vault.centos.org/centos/8-stream/BaseOS/Source/repodata/repomd.xml
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>403 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
Request blocked.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: yImOvvXXTsmYe-jiVc6Wb7AFl0P2PfRKVfMAf_NEJFuWE9_zTvW2vA==
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>

Metadata Update from @mrc0mmand:
- Issue status updated to: Open (was: Closed)

2 years ago

Works for me from 24.211.138.219/2603:6080:5840:71e:4de6:dae6:4c5d:ae2f at 2022-03-08T11:59+00:00 . I think the problem was transient or if still going we need to know what ip address you came from to see if it is 'local' to AWS zone.

yes, already discussed in #centos-ci irc channel, and confirmed it's working.
Clearly some default WAF rules (automated/managed by AWS) aren't happy with community cage somehow. I had to remove these rules, so it's now back to "open bar" mode

Closing for now but we should open a new one about cloudfront AWS shield/waf rules tuning , that we can reuse for both fedora and centos

Metadata Update from @arrfab:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata
Boards 1