Flatpak builds fail with error 503 when connecting to osbs.fedoraproject.org:
$ fedpkg flatpak-build Created task: 50379979 Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=50379979 Watching tasks (this may be safely interrupted)... 50379979 buildContainer (noarch): free 50379979 buildContainer (noarch): free -> open (buildvm-x86-04.iad2.fedoraproject.org) 50379979 buildContainer (noarch): open (buildvm-x86-04.iad2.fedoraproject.org) -> FAILED: Fault: <Fault 1: 'Traceback (most recent call last):\n File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 439, in send\n resp = conn.urlopen(\n File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 833, in urlopen\n return self.urlopen(\n File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 833, in urlopen\n return self.urlopen(\n File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 833, in urlopen\n return self.urlopen(\n [Previous line repeated 5 more times]\n File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 819, in urlopen\n retries = retries.increment(method, url, response=response, _pool=self)\n File "/usr/lib/python3.8/site-packages/urllib3/util/retry.py", line 436, in increment\n raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=\'osbs.fedoraproject.org\', port=443): Max retries exceeded with url: /apis/build.openshift.io/v1/namespaces/osbs-fedora/builds/?labelSelector=koji-task-id%3D50379979 (Caused by ResponseError(\'too many 503 error responses\'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3.8/site-packages/osbs/http.py", line 59, in request\n stream = HttpStream(url, *args, verbose=self.verbose, **kwargs)\n File "/usr/lib/python3.8/site-packages/osbs/http.py", line 156, in __init__\n self.req = self.session.request(method, url, **args)\n File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 533, in request\n resp = self.send(prep, **send_kwargs)\n File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 646, in send\n r = adapter.send(request, **kwargs)\n File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 507, in send\n raise RetryError(e, request=request)\nrequests.exceptions.RetryError: HTTPSConnectionPool(host=\'osbs.fedoraproject.org\', port=443): Max retries exceeded with url: /apis/build.openshift.io/v1/namespaces/osbs-fedora/builds/?labelSelector=koji-task-id%3D50379979 (Caused by ResponseError(\'too many 503 error responses\'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3.8/site-packages/koji/daemon.py", line 1356, in runTask\n response = (handler.run(),)\n File "/usr/lib/python3.8/site-packages/koji/tasks.py", line 328, in run\n return koji.util.call_with_argcheck(self.handler, self.params, self.opts)\n File "/usr/lib/python3.8/site-packages/koji/util.py", line 270, in call_with_argcheck\n return func(*args, **kwargs)\n File "/usr/lib/koji-builder-plugins/builder_containerbuild.py", line 889, in handler\n results = self.runBuilds(src, target_info, archlist,\n File "/usr/lib/koji-builder-plugins/builder_containerbuild.py", line 558, in runBuilds\n semi_results = self.createContainer(**kwargs)\n File "/usr/lib/koji-builder-plugins/builder_containerbuild.py", line 630, in createContainer\n build_response = create_method(**orchestrator_create_build_args)\n File "/usr/lib/python3.8/site-packages/osbs/api.py", line 68, in catch_exceptions\n return func(*args, **kwargs)\n File "/usr/lib/python3.8/site-packages/osbs/api.py", line 1009, in create_orchestrator_build\n return self._do_create_prod_build(**kwargs)\n File "/usr/lib/python3.8/site-packages/osbs/api.py", line 811, in _do_create_prod_build\n builds_for_koji_task = self._get_not_cancelled_builds_for_koji_task(koji_task_id)\n File "/usr/lib/python3.8/site-packages/osbs/api.py", line 288, in _get_not_cancelled_builds_for_koji_task\n all_builds_for_task = self.os.list_builds(koji_task_id=koji_task_id).json()[\'items\']\n File "/usr/lib/python3.8/site-packages/osbs/core.py", line 594, in list_builds\n return self._get(url)\n File "/usr/lib/python3.8/site-packages/osbs/core.py", line 191, in _get\n return self._con.get(\n File "/usr/lib/python3.8/site-packages/osbs/http.py", line 46, in get\n return self.request(url, "get", **kwargs)\n File "/usr/lib/python3.8/site-packages/osbs/http.py", line 68, in request\n raise OsbsNetworkException(url, str(ex), \'\',\nosbs.exceptions.OsbsNetworkException: () HTTPSConnectionPool(host=\'osbs.fedoraproject.org\', port=443): Max retries exceeded with url: /apis/build.openshift.io/v1/namespaces/osbs-fedora/builds/?labelSelector=koji-task-id%3D50379979 (Caused by ResponseError(\'too many 503 error responses\'))\n'> 0 free 0 open 0 done 1 failed 50379979 buildContainer (noarch) failed
I've retried 3 times and got the same error every time. Flatpak builds worked just fine 2 days ago, so must be something that has changed in the mean time.
Metadata Update from @mobrien: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: medium-gain, medium-trouble, ops
I encountered the same issue when trying to build fedora-toolbox image.
fedora-toolbox
$ fedpkg container-build Created task: 50493095 Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=50493095 Watching tasks (this may be safely interrupted)... 50493095 buildContainer (noarch): free 50493095 buildContainer (noarch): free -> open (buildvm-x86-04.iad2.fedoraproject.org) 50493095 buildContainer (noarch): open (buildvm-x86-04.iad2.fedoraproject.org) -> FAILED: Fault: <Fault 1: 'Traceback (most recent call last):\n File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 439, in send\n resp = conn.urlopen(\n File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 833, in urlopen\n return self.urlopen(\n File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 833, in urlopen\n return self.urlopen(\n File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 833, in urlopen\n return self.urlopen(\n [Previous line repeated 5 more times]\n File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 819, in urlopen\n retries = retries.increment(method, url, response=response, _pool=self)\n File "/usr/lib/python3.8/site-packages/urllib3/util/retry.py", line 436, in increment\n raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=\'osbs.fedoraproject.org\', port=443): Max retries exceeded with url: /apis/build.openshift.io/v1/namespaces/osbs-fedora/builds/?labelSelector=koji-task-id%3D50493095 (Caused by ResponseError(\'too many 503 error responses\'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3.8/site-packages/osbs/http.py", line 59, in request\n stream = HttpStream(url, *args, verbose=self.verbose, **kwargs)\n File "/usr/lib/python3.8/site-packages/osbs/http.py", line 156, in __init__\n self.req = self.session.request(method, url, **args)\n File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 533, in request\n resp = self.send(prep, **send_kwargs)\n File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 646, in send\n r = adapter.send(request, **kwargs)\n File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 507, in send\n raise RetryError(e, request=request)\nrequests.exceptions.RetryError: HTTPSConnectionPool(host=\'osbs.fedoraproject.org\', port=443): Max retries exceeded with url: /apis/build.openshift.io/v1/namespaces/osbs-fedora/builds/?labelSelector=koji-task-id%3D50493095 (Caused by ResponseError(\'too many 503 error responses\'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3.8/site-packages/koji/daemon.py", line 1356, in runTask\n response = (handler.run(),)\n File "/usr/lib/python3.8/site-packages/koji/tasks.py", line 328, in run\n return koji.util.call_with_argcheck(self.handler, self.params, self.opts)\n File "/usr/lib/python3.8/site-packages/koji/util.py", line 270, in call_with_argcheck\n return func(*args, **kwargs)\n File "/usr/lib/koji-builder-plugins/builder_containerbuild.py", line 889, in handler\n results = self.runBuilds(src, target_info, archlist,\n File "/usr/lib/koji-builder-plugins/builder_containerbuild.py", line 558, in runBuilds\n semi_results = self.createContainer(**kwargs)\n File "/usr/lib/koji-builder-plugins/builder_containerbuild.py", line 630, in createContainer\n build_response = create_method(**orchestrator_create_build_args)\n File "/usr/lib/python3.8/site-packages/osbs/api.py", line 68, in catch_exceptions\n return func(*args, **kwargs)\n File "/usr/lib/python3.8/site-packages/osbs/api.py", line 1009, in create_orchestrator_build\n return self._do_create_prod_build(**kwargs)\n File "/usr/lib/python3.8/site-packages/osbs/api.py", line 811, in _do_create_prod_build\n builds_for_koji_task = self._get_not_cancelled_builds_for_koji_task(koji_task_id)\n File "/usr/lib/python3.8/site-packages/osbs/api.py", line 288, in _get_not_cancelled_builds_for_koji_task\n all_builds_for_task = self.os.list_builds(koji_task_id=koji_task_id).json()[\'items\']\n File "/usr/lib/python3.8/site-packages/osbs/core.py", line 594, in list_builds\n return self._get(url)\n File "/usr/lib/python3.8/site-packages/osbs/core.py", line 191, in _get\n return self._con.get(\n File "/usr/lib/python3.8/site-packages/osbs/http.py", line 46, in get\n return self.request(url, "get", **kwargs)\n File "/usr/lib/python3.8/site-packages/osbs/http.py", line 68, in request\n raise OsbsNetworkException(url, str(ex), \'\',\nosbs.exceptions.OsbsNetworkException: () HTTPSConnectionPool(host=\'osbs.fedoraproject.org\', port=443): Max retries exceeded with url: /apis/build.openshift.io/v1/namespaces/osbs-fedora/builds/?labelSelector=koji-task-id%3D50493095 (Caused by ResponseError(\'too many 503 error responses\'))\n'> 0 free 0 open 0 done 1 failed 50493095 buildContainer (noarch) failed
@cverna Do you happen to know what might be wrong with the container build system this time? :)
It looks like osbs.fedoraproject.org does not point to the correct service anymore. It should be directing the request to osbs-master01.iad2.fedoraproject.org but that does not seems to happen.
Might be something broken in haproxy but I can't remember how to check that :(
Ahh, thanks for looking into it! Do you know who would be able to debug the proxy setup?
Yes, it looks like the last round of RHIT firewall adjustments to get sudo working on some machines, broke this. ;(
We will get them to fix it.
Awesome, thanks to both of you!
I think this is fixed now. Can you try again?
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Yep, confirmed fixed. Thanks!
Login to comment on this ticket.