I'm looking at the status on the RabbitMQ server, and it seems that the the-new-hotness queue has no consumer and is just accumulating messages. There's probably something wrong going on.
This may be related to the bugzilla issue, where the-new-hotness cannot file new bugs.
I see the pod is crashing over and over in prod with:
[hotness.consumers INFO] OK, 2.0.0 is newer than 1.4.0-3.fc31 [fedora_messaging.twisted.protocol ERROR] Received unexpected exception from consumer Consumer(queue=the-new-hotness, callback=<hotness.consumers.BugzillaTicketFiler object at 0x7f72c3e052b0>) Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/fedora_messaging/twisted/protocol.py", line 239, in _read yield threads.deferToThread(consumer.callback, message) File "/usr/lib64/python3.7/site-packages/twisted/python/threadpool.py", line 250, in inContext result = inContext.theWork() File "/usr/lib64/python3.7/site-packages/twisted/python/threadpool.py", line 266, in <lambda> inContext.theWork = lambda: context.call(ctx, func, *args, **kw) File "/usr/lib64/python3.7/site-packages/twisted/python/context.py", line 122, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib64/python3.7/site-packages/twisted/python/context.py", line 85, in callWithContext return func(*args,**kw) File "/usr/local/lib/python3.7/site-packages/hotness/consumers.py", line 161, in __call__ self.handle_anitya_version_update(msg) File "/usr/local/lib/python3.7/site-packages/hotness/consumers.py", line 206, in handle_anitya_version_update self._handle_anitya_update(upstream, pname, msg) File "/usr/local/lib/python3.7/site-packages/hotness/consumers.py", line 281, in _handle_anitya_update projectid, package, upstream, version, release, url File "/usr/local/lib/python3.7/site-packages/hotness/bz.py", line 133, in handle bug = self.create_bug(**kwargs) File "/usr/local/lib/python3.7/site-packages/hotness/bz.py", line 268, in create_bug new_bug = self.bugzilla.createbug(**bug_dict) File "/usr/lib/python3.7/site-packages/bugzilla/base.py", line 1741, in createbug rawbug = self._proxy.Bug.create(data) File "/usr/lib64/python3.7/xmlrpc/client.py", line 1112, in __call__ return self.__send(self.__name, args) File "/usr/lib/python3.7/site-packages/bugzilla/transport.py", line 104, in _ServerProxy__request self)._ServerProxy__request(methodname, params) File "/usr/lib64/python3.7/xmlrpc/client.py", line 1452, in __request verbose=self.__verbose File "/usr/lib/python3.7/site-packages/bugzilla/transport.py", line 201, in request return self._request_helper(url, request_body) File "/usr/lib/python3.7/site-packages/bugzilla/transport.py", line 179, in _request_helper return self.parse_response(response) File "/usr/lib/python3.7/site-packages/bugzilla/transport.py", line 154, in parse_response return unmarshaller.close() File "/usr/lib64/python3.7/xmlrpc/client.py", line 656, in close raise Fault(**self._stack[0]) xmlrpc.client.Fault: <Fault -32000: 'Can\'t call method "sub_component_count" on an undefined value at /var/www/html/bugzilla/extensions/SubComponents/Extension.pm line 931.\n'> [fedora_messaging.cli ERROR] Unexpected error occurred in consumer Consumer(queue=the-new-hotness, callback=<hotness.consumers.BugzillaTicketFiler object at 0x7f72c3e052b0>): <twisted.python.failure.Failure xmlrpc.client.Fault: <Fault -32000: 'Can\'t call method "sub_component_count" on an undefined value at /var/www/html/bugzilla/extensions/SubComponents/Extension.pm line 931.\n'>> [fedora_messaging.twisted.protocol INFO] Waiting for 0 consumer(s) to finish processing before halting [fedora_messaging.twisted.protocol INFO] Finished canceling 0 consumers [fedora_messaging.twisted.protocol INFO] Disconnect requested, but AMQP connection already gone
@zlopez can you please take a look? if this is the bugzilla issue, can it perhaps handle it more gracefully? Or are you hoping to process all the old messages once it is fixed?
Metadata Update from @kevin: - Issue assigned to zlopez - Issue priority set to: Waiting on Assignee (was: Needs Review)
This is a known issue and it is on Bugzilla side. See https://bugzilla.redhat.com/show_bug.cgi?id=1679385
I'm still waiting for the fix on their side, there isn't much else I can do right now.
This is now fixed:
https://admin.fedoraproject.org/collectd/bin/graph.cgi?hostname=rabbitmq_slash_pubsub;plugin=queues;plugin_instance=the-new-hotness;type=messages;begin=-86400
:fuelpump:
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.