fedora-infrastructure

#3469 MM: files in category topdirs have extra / prepended which fails on S3
Closed: Fixed Opened by mdomsch.

Closed: Fixed
Reopen Issue

= bug description =
From: ​mirror-admin-bounces@fedoraproject.org ​mirror-admin-bounces@fedoraproject.org On Behalf Of Dmitry Zamaruev Sent: Monday, September 10, 2012 5:21 AM To: ​mirror-admin@fedoraproject.org Subject: Wrong mirror redirection

Hi,

I'm using global download URL ​http://download.fedoraproject.org/ - it redirects me to nearest mirror. But this mirror (URL in Location header) does have an error - it have double slash. While this is not an error for most web-servers (they will just compact it to one slash) - this is a different Object for Amazon S3 system. So when I try to download anything from ​http://download.fedoraproject.org/ from inside Amazon EC2 instance - I will receive 404 error, because Location headers suggest using "s3-mirror-us-east-1.fedoraproject.org" server (Amazon S3) with double slash in URL. This renders URL ​http://download.fedoraproject.org/ unusable in Amazon EC2 instances.

Example of transcript: =========== START # curl -v -L ​http://download.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-6

About to connect() to download.fedoraproject.org port 80 (#0)
Trying 209.132.181.16... connected
Connected to download.fedoraproject.org (209.132.181.16) port 80 (#0)
GET /pub/epel/RPM-GPG-KEY-EPEL-6 HTTP/1.1 User-Agent: curl/7.19.7 (i386-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2 Host: download.fedoraproject.org Accept: /

< HTTP/1.1 302 FOUND < Date: Mon, 10 Sep 2012 10:05:16 GMT < Server: Apache/2.2.15 (Red Hat) < cache-control: no-cache < location: ​http://s3-mirror-us-east-1.fedoraproject.org/pub/epel//RPM-GPG-KEY-EPEL-6 < AppTime?: D=23268 < AppServer?: app04.phx2.fedoraproject.org < Content-Length: 0 < Content-Type: text/html; charset=utf-8 < ProxyTime?: D=27088 < ProxyServer?: proxy01.phx2.fedoraproject.org <

Connection #0 to host download.fedoraproject.org left intact
Issue another request to this URL:
'​http://s3-mirror-us-east-1.fedoraproject.org/pub/epel//RPM-GPG-KEY-EPEL-6'

About to connect() to s3-mirror-us-east-1.fedoraproject.org port 80 (#1)
Trying 72.21.211.197... connected
Connected to s3-mirror-us-east-1.fedoraproject.org (72.21.211.197) port 80 (#1)
GET /pub/epelRPM-GPG-KEY-EPEL-6 HTTP/1.1 User-Agent: curl/7.19.7 (i386-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2 Host: s3-mirror-us-east-1.fedoraproject.org Accept: /

< HTTP/1.1 404 Not Found < x-amz-request-id: 1C6C7CDAFA210776 < x-amz-id-2: zPltQwGXM5IvoFQBBVaVrMOrTVm8BCqIQGtSqOXAz7bWlCYO+mPGywacvu36kn6i < Content-Type: text/html; charset=utf-8 < Content-Length: 351 < Date: Mon, 10 Sep 2012 10:05:16 GMT < Server: AmazonS3 < 404 Not Found

404 Not Found

  • Code: NoSuchKey?
  • Message: The specified key does not exist.
  • Key: pub/epelRPM-GPG-KEY-EPEL-6
  • RequestId?: 1C6C7CDAFA210776
  • HostId?: zPltQwGXM5IvoFQBBVaVrMOrTVm8BCqIQGtSqOXAz7bWlCYO+mPGywacvu36kn6i

Connection #1 to host s3-mirror-us-east-1.fedoraproject.org left intact
Closing connection #0
Closing connection #1
=========== END

With best regards, Dmitry Zamaruev.

= bug analysis =

This only happens on file requests to the topdir of a category, not on file requests to files in a subdirectory. The failure is specific to S3, as other web servers squash the second / of the pair, but S3 does not. This thus doesn't affect downloads of the packages from S3 (except perhaps after the initial run which downloads the repository key from S3), but does affect downloading of the GPG keys from the topdirs. Once the key is retrieved from another server, yum should then start using S3 for future package downloads.

= fix recommendation =

proposed fix:
{{{
diff --git a/modules/hotfix/files/mirrormanager/mirrorlist_server.py b/modules/hotfix/files/mirrormanager/mirrorlist_server.py
index fb2ae29..605a121 100755
--- a/modules/hotfix/files/mirrormanager/mirrorlist_server.py
+++ b/modules/hotfix/files/mirrormanager/mirrorlist_server.py
@@ -366,7 +366,9 @@ def append_path(hosts, cache, file, pathIsDirectory=False):
if file is None and pathIsDirectory:
s += "/"
if file is not None:
- s += "/" + file
+ if not s.endswidth('/'):
+ s += "/"
+ s += file
hcurls.append(s)
results.append((hostid, hcurls))
return results
}}}

fixed in MM 1.4 now in production.
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.