Issue #8493: fedora-messaging consumers outages - fedora-infrastructure

fedora-infrastructure

#8493 fedora-messaging consumers outages

Closed: Invalid 4 years ago by jpopelka. Opened 4 years ago by jpopelka.

Hi,

we have 2 instances of our fedora-messaging consumer in Packit service, both configured :

amqp_url = "amqps://fedora:@rabbitmq.fedoraproject.org/%2Fpublic_pubsub"
callback = "fedora_messaging.example:printer"

[exchanges."amq.topic"]
type = "topic"
durable = true
auto_delete = false
arguments = {}

[queues.xyz]
durable = true
auto_delete = true
exclusive = true
arguments = {}

[[bindings]]
queue = "xyz"
exchange = "amq.topic"
routing_keys = ["org.fedoraproject.prod.copr.build.#"]

[qos]
prefetch_size = 0
prefetch_count = 25

They both stopped consuming messages on 2019-25-12 (according to logs).
I restarted them today, but they've been consuming messages intermittently since then.

I've been playing with fedora-messaging consume locally and realized that unless I change routing_keys from ["org.fedoraproject.prod.copr.build.#"] to ["org.fedoraproject.prod.copr.#"] it indeed doesn't consume anything.
And also, once the locally running consumer starts consuming, all of a sudden, the 2 consumers running in cloud start consuming too, until I stop the local one.
They all use different queues (uuids) with exclusive = true so I'm out of ideas.

Can you help me debug that? Thank you!

Metadata Update from @abompard:
- Issue assigned to abompard

4 years ago

abompard commented 4 years ago

Hey jpopleka, yeah that seems weird, can we find some time on IRC to debug this?

abompard commented 4 years ago

Can you, for starters, add a section like the [[bindings]] section you already have, but changing amq.topic to zmq.topic please? This will subscribe your queue to the zmq.topic exchange where messages coming from fedmsg are published (I think copr is still publishing on fedmsg).

jpopelka commented 4 years ago

Aurelien, thanks for the suggestion, however the zmq.topic exchange doesn't seem to help.
I tried to add

[exchanges."zmq.topic"]
type = "topic"
durable = true
auto_delete = false
arguments = {}

[[bindings]]
queue = "my"
exchange = "zmq.topic"
routing_keys = ["org.fedoraproject.prod.copr.build.#"]

as well as use only "zmq.topic" and neither helped.

What still makes it work is changing the routing_keys to either ["org.fedoraproject.prod.copr.#"] or ["org.fedoraproject.prod.copr.build.start", "org.fedoraproject.prod.copr.build.end"]