#847 Move from Redis to pika (rabbitmq)
Opened 9 years ago by pingou. Modified 2 years ago

Currently we are using redis to communicate between our different services. But Redis is fire and forget, so if the service is down, the message is lost.

Switching to rabbitmq (via https://pypi.python.org/pypi/pika ?) would allow making sure messages sent are delivered, otherwise rabbitmq just queues it up and retries until the services acknowledge receiving the message


@pingou Can we discuss more on this?

We can, I don't think this is a high priority though because it will be a rather large change since all the services will need to be adjusted.

Okay. I'll catch up more on this with you over IRC.

The last update was 6 years ago, no further requests, updates or actionable tasks since then, I'm going to close this issue for now to reduce our backlog.

Metadata Update from @wombelix:
- Issue close_status updated to: Won't Fix
- Issue status updated to: Closed (was: Open)

2 years ago

That one is also potentially interesting to keep, we're currently using redis as
a queueing mechanism which while it can be used that way isn't really what it
was built for.
So moving to a proper queueing solution (be that pika or something else) could
still be of interest.

That one is also potentially interesting to keep, we're currently using redis as
a queueing mechanism which while it can be used that way isn't really what it
was built for.

I always through that it's quite a valid use case for Redis?

When re-reading the initial request, it was mainly about reliability:

Currently we are using redis to communicate between our different services. But Redis is fire and forget, so if the service is down, the message is lost.

I guess that could already somehow improved by using Redis persistence. The actual logic to acknowledge and remove from queue is handled by Celery as far I'm aware of.

But sure, let's re-open it, put it on 6.0 and evaluate what would be necessary to introduce an abstraction layer so that either Redis or RabbitMQ can be used, that would probably be the most flexible solution.

Metadata Update from @wombelix:
- Issue status updated to: Open (was: Closed)

2 years ago

Metadata Update from @wombelix:
- Issue set to the milestone: 6.0

2 years ago

But sure, let's re-open it, put it on 6.0 and evaluate what would be necessary to introduce an abstraction layer so that either Redis or RabbitMQ can be used, that would probably be the most flexible solution.

Note: it could be we already have this, in which case this would be a case of
testing/validating, updating the documentation and potentially switching to it
in prod.

Log in to comment on this ticket.

Metadata