Here is the full log:
Process Details --------------- - host: hotness01.phx2.fedoraproject.org - PID: 3093 - name: fedmsg-hub - command: /usr/bin/python2 /usr/bin/fedmsg-hub - msg_id: Callstack that lead to the logging statement -------------------------------------------- File "/usr/lib64/python2.7/threading.py", line 785 in __bootstrap self.__bootstrap_inner() File "/usr/lib64/python2.7/threading.py", line 812 in __bootstrap_inner self.run() File "/usr/lib64/python2.7/threading.py", line 765 in run self.__target(*self.__args, **self.__kwargs) File "/usr/lib64/python2.7/site-packages/twisted/python/threadpool.py", line 167 in _worker result = context.call(ctx, function, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 118 in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 81 in callWithContext return func(*args,**kw) File "/usr/lib/python2.7/site-packages/moksha/hub/api/consumer.py", line 186 in _work_loop self._do_work(message) File "/usr/lib/python2.7/site-packages/moksha/hub/api/consumer.py", line 210 in _do_work self.log.exception(message) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/moksha/hub/api/consumer.py", line 207, in _do_work self.consume(message) File "/usr/lib/python2.7/site-packages/hotness/consumers.py", line 173, in consume self.handle_anitya_map_new(msg) File "/usr/lib/python2.7/site-packages/hotness/consumers.py", line 244, in handle_anitya_map_new anitya.force_check(msg['msg']['project']) File "/usr/lib/python2.7/site-packages/hotness/anitya.py", line 196, in force_check data = self.send_request(url, verb='POST', data=dict(id=idx)) File "/usr/lib/python2.7/site-packages/fedora/client/openidbaseclient.py", line 244, in send_request output = func(method, **kwargs) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 507, in post return self.request('POST', url, data=data, json=json, **kwargs) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 464, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send r = adapter.send(request, **kwargs) File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 419, in send raise ConnectTimeout(e, request=request) ConnectTimeout: HTTPSConnectionPool(host='release-monitoring.org', port=443): Max retries exceeded with url: /api/version/get (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe7cbf10bd0>, 'Connection to release-monitoring.org timed out. (connect timeout=120)'))
When do you need this? (YYYY/MM/DD) As soon as possible
When is this no longer needed or useful? (YYYY/MM/DD) When the-new-hotness in OpenShift will replace the VM one
If we cannot complete your request, what is the impact? Probably some notification will be lost
So one issue I am seeing is that hotness is trying to connect to release-monitoring.org port 9940. However release-monitoring.org is actually just the front end proxies and not the openshift app you are wanting to talk with. So 9940 is not going to answer.
What I think is needed is for the application to talk to the openshift app.os.fedoraproject.org backend.
Thanks @smooge , but I'm not sure where you see the port 9940 (this is used for fedmsg). In the trace above there is port 443.
Metadata Update from @mizdebsk: - Issue assigned to mizdebsk - Issue priority set to: Waiting on Assignee (was: Needs Review)
release-monitoring.org DNS resolves to public IPv4 addresses for proxy01 and proxy10. Internal PXH2 hosts like hotness01 can't connect to these addresses. A proper fix would be converting release-monitoring.org to a templated zone. A quick workaround is adding /etc/hosts entry on hotness to point release-monitoring.org to either proxy101 or proxy110.
Workaround applied. Please test and let me know whether it works.
Metadata Update from @mizdebsk: - Issue priority set to: Waiting on Reporter (was: Waiting on Assignee)
Oh, so it's the same issue like with the partner-bugzilla and others URLs for internal services.
Hm, I need to apply this also on the the-new-hotness in Openshift. Otherwise it will have the same problems.
It looks like the workaround is working, I'm not receiving any new mail about this issue from the-new-hotness.
Thanks @mizdebsk
No, it's not the same issue. partner-bugzilla resolved to an internal IP address where it should resolve to an external one. Case of release-monitoring is reversed: it resolves to external IP addresses where it should resolve to internal ones.
Ok, but there will be the same issue with OpenShift instance of the-new-hotness, if I'm not mistaken.
Thanks for explanation
@mizdebsk Didn't see the issue for a while. I'm closing this.
Metadata Update from @zlopez: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.