#9116 Container builds don't work
Closed: Fixed a month ago by cverna. Opened a month ago by lbalhar.

I am trying to build f31/s2i-core and f31/s2i-base containers but the builds fail due to Failed to establish a new connection: [Errno 101] Network is unreachable.

Could you please take a look at this and help me investigate the problem?

When do you need this to be done by? (YYYY/MM/DD)

sooner is better


Metadata Update from @smooge:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: OSBS, high-trouble, medium-gain

a month ago

I forgot to add some links to failed builds so there are my last two attempts:
- s2i-core https://koji.fedoraproject.org/koji/taskinfo?taskID=46728967
- s2I-base https://koji.fedoraproject.org/koji/taskinfo?taskID=46732708

Hum ok this is because the IP addresses of the registry cdn we are using are not static and we have these specified in the ip tables of OSBS. https://pagure.io/fedora-infra/ansible/c/5e78f7011840287269bac4aa83bed399549132c8?branch=master

I am not sure if OSBS could talk directly to the registry box instead of going through the CDN that way we can have a fixed ip address.

I've seen this as well, for more days already: https://pagure.io/releng/issue/9581

IIUIC and the problem is a different address, could it be work-arounded for now by adding the new IP address to the iptables filter? It would at least unblock building container images for now and some proper solution can be found in upcoming days.

Metadata Update from @cverna:
- Issue assigned to cverna

a month ago

So I tried using the registry-no-cdn.fedoraproject.org in OSBS configuration but that will not work since registry.fedoraproject.org is in each container Dockerfile so that is what is used in the end.

@kevin do you know if we could do something on the proxies about it ?

Fwiw, this seems to be related:

RuntimeError: Registry specified in dockerfile image doesn't match configured one. Dockerfile: 'registry.fedoraproject.org/flatpak-build-base:latest'; expected registry: 'registry-no-cdn.fedoraproject.org'

Got that in a flatpak build (for which yesterday it had the 101 network error).

https://koji.fedoraproject.org/koji/getfile?taskID=46855607&volume=DEFAULT&name=orchestrator.log&offset=-4000

ok, so we have in roles/httpd/reverseproxy/templates/reversepassproxy.registry-generic.conf:

RewriteCond %{HTTP:VIA} !cdn77
RewriteCond %{HTTP:VIA} !cloudfront
RewriteCond %{SERVER_NAME} !^registry-no-cdn.fedoraproject.org$
RewriteCond %{REQUEST_METHOD} !^(PATCH|POST|PUT|DELETE|HEAD)$
RewriteRule ^/v2/(.)/blobs/([a-zA-Z0-9:]) https://cdn.registry.fedoraproject.org/v2/$1/blobs/$2 [R]

Is there some header or condition we could match for here to avoid the cdn re-write?

ok, so we have in roles/httpd/reverseproxy/templates/reversepassproxy.registry-generic.conf:
RewriteCond %{HTTP:VIA} !cdn77
RewriteCond %{HTTP:VIA} !cloudfront
RewriteCond %{SERVER_NAME} !^registry-no-cdn.fedoraproject.org$
RewriteCond %{REQUEST_METHOD} !^(PATCH|POST|PUT|DELETE|HEAD)$
RewriteRule ^/v2/(.)/blobs/([a-zA-Z0-9:]) https://cdn.registry.fedoraproject.org/v2/$1/blobs/$2 [R]
Is there some header or condition we could match for here to avoid the cdn re-write?

Maybe we could use the REMOTE_HOST variable and not redirect if the hostname has osbs-* in it ?

Metadata Update from @cverna:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

a month ago

I have no idea whether this is related or not, but a module build
https://mbs.fedoraproject.org/module-build-service/2/module-builds/9201
https://koji.fedoraproject.org/koji/taskinfo?taskID=43024369
just gave me this at the end (watching command line), though the build seems to have completed:

Could not execute module_build: The following error occurred while getting information on module build #9201:
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p>
<hr>
<address>Apache Server at mbs.fedoraproject.org Port 443</address>
</body></html>

The subsequent flatpak container build then appears to have completed fine
https://koji.fedoraproject.org/koji/taskinfo?taskID=47304499
https://koji.fedoraproject.org/koji/buildinfo?buildID=1542522

But maybe I'm missing something..

Today, I've rebuilt at least 6 Fedora containers and everything was ok. Thank you.

Login to comment on this ticket.

Metadata