Issue #7815: greenwave is not sending messages - fedora-infrastructure

fedora-infrastructure

#7815 greenwave is not sending messages

Closed: Fixed 4 years ago by kevin. Opened 4 years ago by pingou.

Nagios has triggered on receiving no messages from greenwave in a while.

Yesterday, we tried to rebuild greenwave with fedora-messaging which lead to following traceback:

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/fedora_messaging/twisted/protocol.py", line 239, in _read
    yield threads.deferToThread(consumer.callback, message)
  File "/usr/lib64/python3.7/site-packages/twisted/python/threadpool.py", line 250, in inContext
    result = inContext.theWork()
  File "/usr/lib64/python3.7/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/lib64/python3.7/site-packages/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib64/python3.7/site-packages/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
  File "/usr/local/lib/python3.7/site-packages/greenwave/consumers/fedora_messaging_consumer.py", line 47, in fedora_messaging_callback
    messaging_rx_counter.inc()
  File "/usr/lib/python3.7/site-packages/prometheus_client/metrics.py", line 239, in inc
    self._value.inc(amount)
AttributeError: 'Counter' object has no attribute '_value'
[fedora_messaging.cli ERROR] Unexpected error occurred in consumer Consumer(queue=greenwave, callback=<function fedora_messaging_callback at 0x7fa5cd2c7950>): <twisted.python.failure.Failure builtins.AttributeError: 'Counter' object has no attribute '_value'>

So we rollback to use the previous configuration which I believe is still not sending messages.
So it would be nice to easier fix the fedora-messaging deployment (preferred) or fix the current state.

One thing I've observed is that on the web pod the folder /etc/fedmsg-greenwave.d is not mounted while a fedmsg configuration file is pointing to it. This may or may not be related to the failure to send messages though.