#9170 Cleaning up temporary queues in /bodhi?
Closed: Fixed 3 years ago by kevin. Opened 3 years ago by pingou.

Describe what you would like us to do:


Temporary queues on /pubsub are identified by their name being uuid, and according to the fedora-messaging docs: https://fedora-messaging.readthedocs.io/en/stable/quick-start.html#fedora-s-public-broker these queues are automatically deleted if they have no consumers after approximately an hour.

Checking the rabbitmq server, we have a few queues on the /bodhi virtual host, whose name are uuid, who do not seem to have consumers but are currently accumulating messages.

Could it be that we're automatically cleaning up queues in /pubsub and aren't doing so in /bodhi?
Should we allow uuid-based queues in /bodhi?

When do you need this to be done by? (YYYY/MM/DD)


Pending messages take memory on the server, but the amount of messages is currently fairly low (2.5K atm, we went to above 100K last week w/o too much troubles).

Cc @cverna could these queues be coming from bodhi itself? (I'm thinking of celery-beat or so)


Metadata Update from @mobrien:
- Issue tagged with: groomed, medium-gain, medium-trouble

3 years ago

Metadata Update from @mohanboddu:
- Issue priority set to: Waiting on Assignee (was: Needs Review)

3 years ago

@cverna or @abompard if you have an idea (especially around the question of: could bodhi (via celert?) itself create uuid-based queues in rabbitmq?) :)

I still see these, but I am not sure they are growing (in number of queues)

There are currently 15 of these.

I guess we could pull a message off them and see what it looks like?

I pulled a message from a few queue:

queue:
21696a7d-9b30-35f1-82e3-ac6ed4940442
message:
{"task_id": "f8a16f94-119b-4f0e-9c48-e554421286ff", "status": "SUCCESS", "result": null, "traceback": null, "children": []}

---

queue:
2aac2291-3f5f-31a0-9968-b0295bf89a38
message:
{"task_id": "291025f1-04f9-468c-8db3-80599544b701", "status": "SUCCESS", "result": null, "traceback": null, "children": []}

---


queue:
623afe73-5788-37a7-949f-54fd01063016
message:
{"task_id": "140b3480-daea-42b3-a8ad-2d14bf41f179", "status": "SUCCESS", "result": null, "traceback": null, "children": []}

---

queue:
70403b6e-1a76-311d-a506-0d22493dd7fb
message:
{"task_id": "e2cd674f-135d-46e7-a95a-612672a9707c", "status": "SUCCESS", "result": null, "traceback": null, "children": []}

I see 9 queues with an UUID and 3 starting with celeryev, 3 other starting with celery@ and 1 just named: celery.

The messages looks like celery tasks results which IIRC bodhi does not use. I guess a new queue is created everytime a celery worker pod is created.

The messages looks like celery tasks results which IIRC bodhi does not use. I guess a new queue is created everytime a celery worker pod is created.

Can we drop them then? And do think we can prevent more queues from being created?

The messages looks like celery tasks results which IIRC bodhi does not use. I guess a new queue is created everytime a celery worker pod is created.

Can we drop them then? And do think we can prevent more queues from being created?

Yeah I think it is safe to do so, would be good to check on the celery worker pods after tho just to make sure that did not impact them.

Metadata Update from @smooge:
- Issue tagged with: dev, ops

3 years ago

Not sure what there is to develop on this one

Lets just get rabbitmq to clean these per it's policy! :)

https://pagure.io/fedora-infra/ansible/pull-request/387

Login to comment on this ticket.

Metadata
Boards 2
ops Status: Done
dev Status: Done
Related Pull Requests
  • #387 Merged 3 years ago