This seems to have broken in our recent update/reboot cycle:

Sep 03 19:46:56 loopabull01.phx2.fedoraproject.org fedmsg-hub[27438]: [2019-09-03 19:46:56][moksha.hub   ERRO
R] Failed to load 'fedmsg_rabbitmq_serializer_consumer' consumer.
Sep 03 19:46:56 loopabull01.phx2.fedoraproject.org fedmsg-hub[27438]: Traceback (most recent call last):
Sep 03 19:46:56 loopabull01.phx2.fedoraproject.org fedmsg-hub[27438]: File "/usr/lib/python2.7/site-packages/
moksha/hub/hub.py", line 397, in __init_consumers
Sep 03 19:46:56 loopabull01.phx2.fedoraproject.org fedmsg-hub[27438]: c = consumer.load()
Sep 03 19:46:56 loopabull01.phx2.fedoraproject.org fedmsg-hub[27438]: File "/usr/lib/python2.7/site-packages/
pkg_resources.py", line 2260, in load
Sep 03 19:46:56 loopabull01.phx2.fedoraproject.org fedmsg-hub[27438]: entry = __import__(self.module_name, gl
obals(),globals(), ['__name__'])
Sep 03 19:46:56 loopabull01.phx2.fedoraproject.org fedmsg-hub[27438]: File "/usr/lib/python2.7/site-packages/
fedmsg_rabbitmq_serializer/consumer.py", line 16, in <module>
Sep 03 19:46:56 loopabull01.phx2.fedoraproject.org fedmsg-hub[27438]: retry_delay=2,
Sep 03 19:46:56 loopabull01.phx2.fedoraproject.org fedmsg-hub[27438]: File "/usr/lib/python2.7/site-packages/
pika/connection.py", line 658, in __init__
Sep 03 19:46:56 loopabull01.phx2.fedoraproject.org fedmsg-hub[27438]: raise TypeError('unexpected kwargs: %r'
 % (kwargs,))
Sep 03 19:46:56 loopabull01.phx2.fedoraproject.org fedmsg-hub[27438]: TypeError: unexpected kwargs: {'heartbe
at_interval': 0}

This was due to pika and rabbitmq versions.

We have downgraded pika and it should be working again now...

I have a freeze break in to upgrade rabbitmq and that should hopefully also work.

Sadly, upgrading rabbitmq-server didn't seem to fix it. ;(

I guess needs a bit more investigation...in the mean time I have rolled it back to the working versions.

Doesn't appear to be fixed - https://src.fedoraproject.org/rpms/lorax/pull-request/3

Several attempts, including new pushes and requesting [citest] have not triggered Fedora CI test.

Yeah, seems it didn't like downgrading. ;(

Will need to get a rabbitmq person to look... @abompard can you have a look on loopabull01?

Sure, but I don't have the rights to login to that box. I'll also need to be able to sudo there.

It looks like the rabbitmq-server failed to start, trying to restart it and checking the logs

Ok, we nuke /var/lib/rabbitmq/mnesia/ and we could then restart rabbitmq with the newer version. We've lost the message that were in the queue at that time but at least we're back working.

From https://src.fedoraproject.org/rpms/fedora-gather-easyfix/pull-request/5 and https://src.fedoraproject.org/rpms/fedora-gather-easyfix/pull-request/6 it looks like it is working :)

Let's close as fixed and please re-open if you're seeing any more issues :)

https://src.fedoraproject.org/tests/python/pull-request/11 no Ci flag. I cannot reopen.

The CI pipeline change its format a few minutes ago which has broke things, I'm working with @bgoncalv to bring the format back :)

FTR the formatting issue was fixed yesterday and things are back in business :)

ok, then, closing again... please file new or reopen if there's anything further or something breaks again.