Today I wanted to test new release of the-new-hotness on staging by triggering messages in Anitya. I noticed that messages aren't published, to be sure that this isn't just some temporary issue I did a rebuild of the Anitya in OpenShift to have a fresh deployment.
There are currently three pods running on staging OpenShift for Anitya, one is the web server, second is service that checks for new versions and the last one is checking libraries.io for new projects. All three should emit messages.
The error for the web server pod is:
[Mon Jun 01 09:36:57.323365 2020] [wsgi:error] [pid 13:tid 140493386503936] [twisted INFO] Starting factory FedoraMessagingFactoryV2(parameters=<URLParameters host=rabbitmq.stg.fedoraproject.org port=5671 virtual_host=/pubsub ssl=True>, confirms=True) [Mon Jun 01 09:36:57.361852 2020] [wsgi:error] [pid 13:tid 140493378111232] [pika.connection INFO] AMQP stack terminated, failed to connect, or aborted: error-arg=None; pending-error=ConnectionDone() [Mon Jun 01 09:36:57.362095 2020] [wsgi:error] [pid 13:tid 140493378111232] [pika.connection INFO] Connection setup terminated due to ConnectionDone() [Mon Jun 01 09:36:57.364795 2020] [wsgi:error] [pid 13:tid 140493386503936] [twisted CRITICAL] Unhandled error in Deferred: [Mon Jun 01 09:36:57.365099 2020] [wsgi:error] [pid 13:tid 140493386503936] [twisted CRITICAL] [Mon Jun 01 09:36:57.365112 2020] [wsgi:error] [pid 13:tid 140493386503936] fedora_messaging.exceptions.ConnectionException [Mon Jun 01 09:36:57.365204 2020] [wsgi:error] [pid 13:tid 140493386503936] [twisted INFO] <twisted.internet.tcp.Connector instance at 0x7fc73504e290 disconnected IPv4Address(type='TCP', host='rabbitmq.stg.fedoraproject.org', port=5671)> will retry in 3043 seconds [Mon Jun 01 09:36:57.365350 2020] [wsgi:error] [pid 13:tid 140493386503936] [twisted INFO] Stopping factory FedoraMessagingFactoryV2(parameters=<URLParameters host=rabbitmq.stg.fedoraproject.org port=5671 virtual_host=/pubsub ssl=True>, confirms=True)
Error for the check service is:
[fedora_messaging.api ERROR] Traceback (most recent call last): File "/usr/lib64/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks result = g.send(result) File "/usr/lib/python3.7/site-packages/fedora_messaging/api.py", line 261, in _twisted_publish yield _twisted_service._service.factory.publish(message, exchange=exchange) File "/usr/lib64/python3.7/site-packages/twisted/internet/defer.py", line 1613, in unwindGenerator return _cancellableInlineCallbacks(gen) File "/usr/lib64/python3.7/site-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks _inlineCallbacks(None, g, status) --- <exception caught here> --- File "/usr/lib/python3.7/site-packages/fedora_messaging/api.py", line 261, in _twisted_publish yield _twisted_service._service.factory.publish(message, exchange=exchange) File "/usr/lib64/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks result = g.send(result) File "/usr/lib/python3.7/site-packages/fedora_messaging/twisted/factory.py", line 432, in publish yield protocol.publish(message, exchange) builtins.AttributeError: 'NoneType' object has no attribute 'publish' [anitya ERROR] 'NoneType' object has no attribute 'publish' Traceback (most recent call last): File "/usr/local/bin/check_service.py", line 224, in run future.result() File "/usr/lib64/python3.7/concurrent/futures/_base.py", line 428, in result return self.__get_result() File "/usr/lib64/python3.7/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/lib64/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/bin/check_service.py", line 106, in update_project utilities.check_project_release(project, session) File "/usr/local/lib/python3.7/site-packages/anitya/lib/utilities.py", line 173, in check_project_release odd_change=False, File "/usr/local/lib/python3.7/site-packages/anitya/lib/utilities.py", line 236, in log topic=topic, msg=dict(project=project, distro=distro, message=message) File "/usr/local/lib/python3.7/site-packages/anitya/lib/utilities.py", line 59, in fedmsg_publish topic="anitya.{}".format(kwargs["topic"]), body=kwargs["msg"] File "/usr/lib/python3.7/site-packages/fedora_messaging/api.py", line 313, in publish eventual_result.wait(timeout=timeout) File "/usr/lib/python3.7/site-packages/crochet/_eventloop.py", line 241, in wait result.raiseException() File "/usr/lib64/python3.7/site-packages/twisted/python/failure.py", line 488, in raiseException raise self.value.with_traceback(self.tb) AttributeError: 'NoneType' object has no attribute 'publish'
It will be best to solve this after the move is done.
Metadata Update from @smooge: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: backlog, groomed, high-gain, medium-trouble, staging
I think messages on staging aren't emitted by a couple of weeks, at least. Also Bodhi staging shows stuck updates since mid May.
@mattia It's possible that you are right, I'm using the staging only when I need to test a new release or play with something.
so, staging is gone right now and we will be rebuilding it later this month.
Lets just close this now and deal with it if there it still an issue after the bringup.
Metadata Update from @kevin: - Issue close_status updated to: Will Not/Can Not fix - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.