fedora-infrastructure

#7615 the-new-hotness - connectivity to release-monitoring.org
Closed: Fixed by zlopez. Opened by zlopez.

Closed: Fixed
Reopen Issue
  • Describe what you need us to do:
    I'm getting errors about the-new-hotness timeout when connecting to release-monitoring.org. Anitya is running fine and the API is normally accessible from my computer. It is probably some connectivity issue between the-new-hotness and release-monitoring.org.

Here is the full log:

Process Details
---------------
- host:     hotness01.phx2.fedoraproject.org
- PID:      3093
- name:     fedmsg-hub
- command:  /usr/bin/python2 /usr/bin/fedmsg-hub
- msg_id:   
Callstack that lead to the logging statement
--------------------------------------------
  File "/usr/lib64/python2.7/threading.py", line 785 in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib64/python2.7/threading.py", line 812 in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765 in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/site-packages/twisted/python/threadpool.py", line 167 in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 118 in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 81 in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.7/site-packages/moksha/hub/api/consumer.py", line 186 in _work_loop
    self._do_work(message)
  File "/usr/lib/python2.7/site-packages/moksha/hub/api/consumer.py", line 210 in _do_work
    self.log.exception(message)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/moksha/hub/api/consumer.py", line 207, in _do_work
    self.consume(message)
  File "/usr/lib/python2.7/site-packages/hotness/consumers.py", line 173, in consume
    self.handle_anitya_map_new(msg)
  File "/usr/lib/python2.7/site-packages/hotness/consumers.py", line 244, in handle_anitya_map_new
    anitya.force_check(msg['msg']['project'])
  File "/usr/lib/python2.7/site-packages/hotness/anitya.py", line 196, in force_check
    data = self.send_request(url, verb='POST', data=dict(id=idx))
  File "/usr/lib/python2.7/site-packages/fedora/client/openidbaseclient.py", line 244, in send_request
    output = func(method, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 507, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 464, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 419, in send
    raise ConnectTimeout(e, request=request)
ConnectTimeout: HTTPSConnectionPool(host='release-monitoring.org', port=443): Max retries exceeded with url: /api/version/get (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe7cbf10bd0>, 'Connection to release-monitoring.org timed out. (connect timeout=120)'))

  • When do you need this? (YYYY/MM/DD)
    As soon as possible

  • When is this no longer needed or useful? (YYYY/MM/DD)
    When the-new-hotness in OpenShift will replace the VM one

  • If we cannot complete your request, what is the impact?
    Probably some notification will be lost

So one issue I am seeing is that hotness is trying to connect to release-monitoring.org port 9940. However release-monitoring.org is actually just the front end proxies and not the openshift app you are wanting to talk with. So 9940 is not going to answer.

What I think is needed is for the application to talk to the openshift app.os.fedoraproject.org backend.

Thanks @smooge , but I'm not sure where you see the port 9940 (this is used for fedmsg). In the trace above there is port 443.

Metadata Update from @mizdebsk:
- Issue assigned to mizdebsk
- Issue priority set to: Waiting on Assignee (was: Needs Review)

release-monitoring.org DNS resolves to public IPv4 addresses for proxy01 and proxy10. Internal PXH2 hosts like hotness01 can't connect to these addresses. A proper fix would be converting release-monitoring.org to a templated zone. A quick workaround is adding /etc/hosts entry on hotness to point release-monitoring.org to either proxy101 or proxy110.

Workaround applied. Please test and let me know whether it works.

Metadata Update from @mizdebsk:
- Issue priority set to: Waiting on Reporter (was: Waiting on Assignee)

Oh, so it's the same issue like with the partner-bugzilla and others URLs for internal services.

Hm, I need to apply this also on the the-new-hotness in Openshift. Otherwise it will have the same problems.

It looks like the workaround is working, I'm not receiving any new mail about this issue from the-new-hotness.

Thanks @mizdebsk

No, it's not the same issue. partner-bugzilla resolved to an internal IP address where it should resolve to an external one. Case of release-monitoring is reversed: it resolves to external IP addresses where it should resolve to internal ones.

Ok, but there will be the same issue with OpenShift instance of the-new-hotness, if I'm not mistaken.

Thanks for explanation

@mizdebsk
Didn't see the issue for a while. I'm closing this.

Metadata Update from @zlopez:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
High
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.