#8069 No consumer is reading from the the-new-hotness bus message queue
Closed: Fixed 2 years ago by kevin. Opened 2 years ago by abompard.

I'm looking at the status on the RabbitMQ server, and it seems that the the-new-hotness queue has no consumer and is just accumulating messages.
There's probably something wrong going on.

This may be related to the bugzilla issue, where the-new-hotness cannot file new bugs.

I see the pod is crashing over and over in prod with:

[hotness.consumers INFO] OK, 2.0.0 is newer than 1.4.0-3.fc31
[fedora_messaging.twisted.protocol ERROR] Received unexpected exception from consumer Consumer(queue=the-new-hotness, callback=<hotness.consumers.BugzillaTicketFiler object at 0x7f72c3e052b0>)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/fedora_messaging/twisted/protocol.py", line 239, in _read
    yield threads.deferToThread(consumer.callback, message)
  File "/usr/lib64/python3.7/site-packages/twisted/python/threadpool.py", line 250, in inContext
    result = inContext.theWork()
  File "/usr/lib64/python3.7/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/lib64/python3.7/site-packages/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib64/python3.7/site-packages/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
  File "/usr/local/lib/python3.7/site-packages/hotness/consumers.py", line 161, in __call__
  File "/usr/local/lib/python3.7/site-packages/hotness/consumers.py", line 206, in handle_anitya_version_update
    self._handle_anitya_update(upstream, pname, msg)
  File "/usr/local/lib/python3.7/site-packages/hotness/consumers.py", line 281, in _handle_anitya_update
    projectid, package, upstream, version, release, url
  File "/usr/local/lib/python3.7/site-packages/hotness/bz.py", line 133, in handle
    bug = self.create_bug(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/hotness/bz.py", line 268, in create_bug
    new_bug = self.bugzilla.createbug(**bug_dict)
  File "/usr/lib/python3.7/site-packages/bugzilla/base.py", line 1741, in createbug
    rawbug = self._proxy.Bug.create(data)
  File "/usr/lib64/python3.7/xmlrpc/client.py", line 1112, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python3.7/site-packages/bugzilla/transport.py", line 104, in _ServerProxy__request
    self)._ServerProxy__request(methodname, params)
  File "/usr/lib64/python3.7/xmlrpc/client.py", line 1452, in __request
  File "/usr/lib/python3.7/site-packages/bugzilla/transport.py", line 201, in request
    return self._request_helper(url, request_body)
  File "/usr/lib/python3.7/site-packages/bugzilla/transport.py", line 179, in _request_helper
    return self.parse_response(response)
  File "/usr/lib/python3.7/site-packages/bugzilla/transport.py", line 154, in parse_response
    return unmarshaller.close()
  File "/usr/lib64/python3.7/xmlrpc/client.py", line 656, in close
    raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault -32000: 'Can\'t call method "sub_component_count" on an undefined value at /var/www/html/bugzilla/extensions/SubComponents/Extension.pm line 931.\n'>
[fedora_messaging.cli ERROR] Unexpected error occurred in consumer Consumer(queue=the-new-hotness, callback=<hotness.consumers.BugzillaTicketFiler object at 0x7f72c3e052b0>): <twisted.python.failure.Failure xmlrpc.client.Fault: <Fault -32000: 'Can\'t call method "sub_component_count" on an undefined value at /var/www/html/bugzilla/extensions/SubComponents/Extension.pm line 931.\n'>>
[fedora_messaging.twisted.protocol INFO] Waiting for 0 consumer(s) to finish processing before halting
[fedora_messaging.twisted.protocol INFO] Finished canceling 0 consumers
[fedora_messaging.twisted.protocol INFO] Disconnect requested, but AMQP connection already gone

@zlopez can you please take a look? if this is the bugzilla issue, can it perhaps handle it more gracefully? Or are you hoping to process all the old messages once it is fixed?

Metadata Update from @kevin:
- Issue assigned to zlopez
- Issue priority set to: Waiting on Assignee (was: Needs Review)

2 years ago

This is a known issue and it is on Bugzilla side. See https://bugzilla.redhat.com/show_bug.cgi?id=1679385

I'm still waiting for the fix on their side, there isn't much else I can do right now.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.