#8625 broken aarch64 mirror for copr builds?
Closed: Fixed a month ago by smooge. Opened 2 months ago by praiskup.

From this build in copr today:
https://download.copr.fedorainfracloud.org/results/praiskup/ping/epel-7-aarch64/01223250-dummy-pkg/builder-live.log.gz

I see:

2020-02-06T12:28:14Z INFO Downloading: http://mirrorlist.centos.org/?release=7&arch=aarch64&repo=os
2020-02-06T12:28:14Z INFO Downloading: http://mirrorlist.centos.org/?release=7&arch=aarch64&repo=updates
2020-02-06T12:28:14Z INFO Downloading: http://mirrors.fedoraproject.org/mirrorlist?repo=epel-7&arch=aarch64
...

In the librepo log (see below), and the mirror returned seem to be broken, because:

2020-02-06T12:28:29Z INFO Error during transfer: Status code: 404 for http://d36uatko69830t.cloudfront.net/altarch/7/os/aarch64/Packages/gcc-c++-4.8.5-39.el7.aarch64.rpm (IP: 13.249.39.56)

https://copr-be.cloud.fedoraproject.org/results/praiskup/ping/epel-7-aarch64/01223250-dummy-pkg/chroot_scan/var/lib/mock/1223250-epel-7-aarch64-1580992070.245452/root/var/log/dnf.librepo.log


@jmracek, @mblaha shouldn't dnf actually pick different mirror from mirrorlist in this case?

@arrfab This looks to be due to the changes to CentOS mirrors in AWS.

@praiskup this is pulling information from CentOS and not Fedora systems. They implemented a change recently that nodes in AWS will pull from cloudfront. A node outside of AWS trying to reach such a mirror will not work.

Metadata Update from @smooge:
- Issue tagged with: copr

2 months ago

Let me verify but pretty sure that the pkg is in the Origin used by cloudfront. but the fact that it has ++ in the name can be the issue with cloudfront.
That reminds me a discussion with Kevin about such issue.

@praiskup : seems to be an issue with the way objects are stored in the S3 bucket and so cloudfront can't retrieve it (yes, using + in the name seems to be the issue with S3)
I'll verify what can be done, but I have already regenerated a list of mirrors, falling back to mirror.centos.org, so can you retry ?
It should , if can't download from internal cloudfront, fall back to mirror.centos.org (as a workaround for now)

Do note: EPEL7 for aarch64 is not supported/active anymore. We do still have the bits available, but they are not in sync with RHEL7.7 in any way.

That said, yeah, there's a problem with + in names... will try and find the reference. @codeblock may have a pointer.

Metadata Update from @smooge:
- Issue assigned to smooge

2 months ago

Metadata Update from @smooge:
- Issue priority set to: Waiting on Assignee (was: Needs Review)

2 months ago

This was brought up in the Red Hat CPE Sysadmin meeting today. There are several issues going on:

  1. builders and systems using yum and el-6 or el-7 will work because yum and urlgrabber sends to amazon a %2B for every +
  2. builders and systems using dnf on el-8 or fedora-30/31 will not work easily because dnf doesn't know about this and asks for + which S3 interprets as a special character.
  3. a work-around is in place where the primary mirror sent to the client is the S3 one with a second one sent which is outside of AWS. This will raise an warning in dnf of a 404 and then it will try the next mirror. We do not know if this works inside of builders or other tools.

Please open if still happening but I think we have done all we can do at this time.

Metadata Update from @smooge:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

a month ago

Login to comment on this ticket.

Metadata