#8493 fedora-messaging consumers outages
Closed: Invalid 2 years ago by jpopelka. Opened 2 years ago by jpopelka.

Hi,

we have 2 instances of our fedora-messaging consumer in Packit service, both configured :

amqp_url = "amqps://fedora:@rabbitmq.fedoraproject.org/%2Fpublic_pubsub"
callback = "fedora_messaging.example:printer"

[exchanges."amq.topic"]
type = "topic"
durable = true
auto_delete = false
arguments = {}

[queues.xyz]
durable = true
auto_delete = true
exclusive = true
arguments = {}

[[bindings]]
queue = "xyz"
exchange = "amq.topic"
routing_keys = ["org.fedoraproject.prod.copr.build.#"]

[qos]
prefetch_size = 0
prefetch_count = 25

They both stopped consuming messages on 2019-25-12 (according to logs).
I restarted them today, but they've been consuming messages intermittently since then.

I've been playing with fedora-messaging consume locally and realized that unless I change routing_keys from ["org.fedoraproject.prod.copr.build.#"] to ["org.fedoraproject.prod.copr.#"] it indeed doesn't consume anything.
And also, once the locally running consumer starts consuming, all of a sudden, the 2 consumers running in cloud start consuming too, until I stop the local one.
They all use different queues (uuids) with exclusive = true so I'm out of ideas.

Can you help me debug that? Thank you!


Metadata Update from @abompard:
- Issue assigned to abompard

2 years ago

Hey jpopleka, yeah that seems weird, can we find some time on IRC to debug this?

Can you, for starters, add a section like the [[bindings]] section you already have, but changing amq.topic to zmq.topic please? This will subscribe your queue to the zmq.topic exchange where messages coming from fedmsg are published (I think copr is still publishing on fedmsg).

Aurelien, thanks for the suggestion, however the zmq.topic exchange doesn't seem to help.
I tried to add

[exchanges."zmq.topic"]
type = "topic"
durable = true
auto_delete = false
arguments = {}

[[bindings]]
queue = "my"
exchange = "zmq.topic"
routing_keys = ["org.fedoraproject.prod.copr.build.#"]

as well as use only "zmq.topic" and neither helped.

What still makes it work is changing the routing_keys to either ["org.fedoraproject.prod.copr.#"] or ["org.fedoraproject.prod.copr.build.start", "org.fedoraproject.prod.copr.build.end"]

OK, can we hop on IRC to try to debug this please? It'll be easier. Thanks

I'm on #fedora-admin and #fedora-noc

So the above suggested change has worked after all. Thank you!

Metadata Update from @jpopelka:
- Issue close_status updated to: Invalid
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata