#3469 MM: files in category topdirs have extra / prepended which fails on S3
Closed: Fixed None by mdomsch. Opened 6 years ago by mdomsch.

= bug description = From: ​mirror-admin-bounces@fedoraproject.org ​mirror-admin-bounces@fedoraproject.org On Behalf Of Dmitry Zamaruev Sent: Monday, September 10, 2012 5:21 AM To: ​mirror-admin@fedoraproject.org Subject: Wrong mirror redirection Hi, I'm using global download URL ​http://download.fedoraproject.org/ - it redirects me to nearest mirror. But this mirror (URL in Location header) does have an error - it have double slash. While this is not an error for most web-servers (they will just compact it to one slash) - this is a different Object for Amazon S3 system. So when I try to download anything from ​http://download.fedoraproject.org/ from inside Amazon EC2 instance - I will receive 404 error, because Location headers suggest using "s3-mirror-us-east-1.fedoraproject.org" server (Amazon S3) with double slash in URL. This renders URL ​http://download.fedoraproject.org/ unusable in Amazon EC2 instances. Example of transcript: =========== START # curl -v -L ​http://download.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-6 About to connect() to download.fedoraproject.org port 80 (#0) Trying connected Connected to download.fedoraproject.org ( port 80 (#0) GET /pub/epel/RPM-GPG-KEY-EPEL-6 HTTP/1.1 User-Agent: curl/7.19.7 (i386-redhat-linux-gnu) libcurl/7.19.7 NSS/ zlib/1.2.3 libidn/1.18 libssh2/1.2.2 Host: download.fedoraproject.org Accept: */* < HTTP/1.1 302 FOUND < Date: Mon, 10 Sep 2012 10:05:16 GMT < Server: Apache/2.2.15 (Red Hat) < cache-control: no-cache < location: ​http://s3-mirror-us-east-1.fedoraproject.org/pub/epel//RPM-GPG-KEY-EPEL-6 < AppTime?: D=23268 < AppServer?: app04.phx2.fedoraproject.org < Content-Length: 0 < Content-Type: text/html; charset=utf-8 < ProxyTime?: D=27088 < ProxyServer?: proxy01.phx2.fedoraproject.org < Connection #0 to host download.fedoraproject.org left intact Issue another request to this URL: '​http://s3-mirror-us-east-1.fedoraproject.org/pub/epel//RPM-GPG-KEY-EPEL-6' About to connect() to s3-mirror-us-east-1.fedoraproject.org port 80 (#1) Trying connected Connected to s3-mirror-us-east-1.fedoraproject.org ( port 80 (#1) GET /pub/epelRPM-GPG-KEY-EPEL-6 HTTP/1.1 User-Agent: curl/7.19.7 (i386-redhat-linux-gnu) libcurl/7.19.7 NSS/ zlib/1.2.3 libidn/1.18 libssh2/1.2.2 Host: s3-mirror-us-east-1.fedoraproject.org Accept: */* < HTTP/1.1 404 Not Found < x-amz-request-id: 1C6C7CDAFA210776 < x-amz-id-2: zPltQwGXM5IvoFQBBVaVrMOrTVm8BCqIQGtSqOXAz7bWlCYO+mPGywacvu36kn6i < Content-Type: text/html; charset=utf-8 < Content-Length: 351 < Date: Mon, 10 Sep 2012 10:05:16 GMT < Server: AmazonS3 < <html> <head><title>404 Not Found</title></head> <body>

404 Not Found

  • Code: NoSuchKey?
  • Message: The specified key does not exist.
  • Key: pub/epelRPM-GPG-KEY-EPEL-6
  • RequestId?: 1C6C7CDAFA210776
  • HostId?: zPltQwGXM5IvoFQBBVaVrMOrTVm8BCqIQGtSqOXAz7bWlCYO+mPGywacvu36kn6i

</body> </html> Connection #1 to host s3-mirror-us-east-1.fedoraproject.org left intact Closing connection #0 Closing connection #1 =========== END With best regards, Dmitry Zamaruev. = bug analysis = This only happens on file requests to the topdir of a category, not on file requests to files in a subdirectory. The failure is specific to S3, as other web servers squash the second / of the pair, but S3 does not. This thus doesn't affect downloads of the packages from S3 (except perhaps after the initial run which downloads the repository key from S3), but does affect downloading of the GPG keys from the topdirs. Once the key is retrieved from _another_ server, yum should then start using S3 for future package downloads. = fix recommendation =

proposed fix:
diff --git a/modules/hotfix/files/mirrormanager/mirrorlist_server.py b/modules/hotfix/files/mirrormanager/mirrorlist_server.py
index fb2ae29..605a121 100755
--- a/modules/hotfix/files/mirrormanager/mirrorlist_server.py
+++ b/modules/hotfix/files/mirrormanager/mirrorlist_server.py
@@ -366,7 +366,9 @@ def append_path(hosts, cache, file, pathIsDirectory=False):
if file is None and pathIsDirectory:
s += "/"
if file is not None:
- s += "/" + file
+ if not s.endswidth('/'):
+ s += "/"
+ s += file
results.append((hostid, hcurls))
return results

fixed in MM 1.4 now in production.

Login to comment on this ticket.