I am trying to build f31/s2i-core and f31/s2i-base containers but the builds fail due to Failed to establish a new connection: [Errno 101] Network is unreachable.
Failed to establish a new connection: [Errno 101] Network is unreachable
Could you please take a look at this and help me investigate the problem?
sooner is better
Metadata Update from @smooge: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: OSBS, high-trouble, medium-gain
I forgot to add some links to failed builds so there are my last two attempts: - s2i-core https://koji.fedoraproject.org/koji/taskinfo?taskID=46728967 - s2I-base https://koji.fedoraproject.org/koji/taskinfo?taskID=46732708
Hum ok this is because the IP addresses of the registry cdn we are using are not static and we have these specified in the ip tables of OSBS. https://pagure.io/fedora-infra/ansible/c/5e78f7011840287269bac4aa83bed399549132c8?branch=master
I am not sure if OSBS could talk directly to the registry box instead of going through the CDN that way we can have a fixed ip address.
I've seen this as well, for more days already: https://pagure.io/releng/issue/9581
IIUIC and the problem is a different address, could it be work-arounded for now by adding the new IP address to the iptables filter? It would at least unblock building container images for now and some proper solution can be found in upcoming days.
Metadata Update from @cverna: - Issue assigned to cverna
So I tried using the registry-no-cdn.fedoraproject.org in OSBS configuration but that will not work since registry.fedoraproject.org is in each container Dockerfile so that is what is used in the end.
registry-no-cdn.fedoraproject.org
registry.fedoraproject.org
@kevin do you know if we could do something on the proxies about it ?
Fwiw, this seems to be related:
RuntimeError: Registry specified in dockerfile image doesn't match configured one. Dockerfile: 'registry.fedoraproject.org/flatpak-build-base:latest'; expected registry: 'registry-no-cdn.fedoraproject.org'
Got that in a flatpak build (for which yesterday it had the 101 network error).
https://koji.fedoraproject.org/koji/getfile?taskID=46855607&volume=DEFAULT&name=orchestrator.log&offset=-4000
ok, so we have in roles/httpd/reverseproxy/templates/reversepassproxy.registry-generic.conf:
RewriteCond %{HTTP:VIA} !cdn77 RewriteCond %{HTTP:VIA} !cloudfront RewriteCond %{SERVER_NAME} !^registry-no-cdn.fedoraproject.org$ RewriteCond %{REQUEST_METHOD} !^(PATCH|POST|PUT|DELETE|HEAD)$ RewriteRule ^/v2/(.)/blobs/([a-zA-Z0-9:]) https://cdn.registry.fedoraproject.org/v2/$1/blobs/$2 [R]
Is there some header or condition we could match for here to avoid the cdn re-write?
ok, so we have in roles/httpd/reverseproxy/templates/reversepassproxy.registry-generic.conf: RewriteCond %{HTTP:VIA} !cdn77 RewriteCond %{HTTP:VIA} !cloudfront RewriteCond %{SERVER_NAME} !^registry-no-cdn.fedoraproject.org$ RewriteCond %{REQUEST_METHOD} !^(PATCH|POST|PUT|DELETE|HEAD)$ RewriteRule ^/v2/(.)/blobs/([a-zA-Z0-9:]) https://cdn.registry.fedoraproject.org/v2/$1/blobs/$2 [R] Is there some header or condition we could match for here to avoid the cdn re-write?
Maybe we could use the REMOTE_HOST variable and not redirect if the hostname has osbs-* in it ?
REMOTE_HOST
osbs-*
Fixed by https://pagure.io/fedora-infra/ansible/pull-request/166
Sorry it took so long but container builds should be working now https://koji.fedoraproject.org/koji/taskinfo?taskID=47274413
Metadata Update from @cverna: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
I have no idea whether this is related or not, but a module build https://mbs.fedoraproject.org/module-build-service/2/module-builds/9201 https://koji.fedoraproject.org/koji/taskinfo?taskID=43024369 just gave me this at the end (watching command line), though the build seems to have completed:
Could not execute module_build: The following error occurred while getting information on module build #9201: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>502 Proxy Error</title> </head><body> <h1>Proxy Error</h1> <p>The proxy server received an invalid response from an upstream server.<br /> The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p> <hr> <address>Apache Server at mbs.fedoraproject.org Port 443</address> </body></html>
The subsequent flatpak container build then appears to have completed fine https://koji.fedoraproject.org/koji/taskinfo?taskID=47304499 https://koji.fedoraproject.org/koji/buildinfo?buildID=1542522
But maybe I'm missing something..
Today, I've rebuilt at least 6 Fedora containers and everything was ok. Thank you.
Login to comment on this ticket.