#8167 Adding topic authorization to our RabbitMQ instances
Opened 2 years ago by abompard. Modified 3 days ago

Community applications will want to publish messages on the bus (like election in its future CommunityShift home). Currently, any read-write account can publish to any topic, which can be a security issue.

Starting with RabbitMQ 3.7.0, topic authorization is possible, but the version we are running is 3.6.0 since that's what's in EPEL7. If we want to have topic authorizations, we need to upgrade RabbitMQ. Making an infra-specific package seems like a bad idea because of the maintenance it involves. The other way would be to upgrade the servers to RHEL8.


Turns out, rhel8 doesn't seem to have the rabbitmq server in it, only the client libraries. :(

Will have to ponder on a solution here...

Metadata Update from @kevin:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: rabbitmq

2 years ago

Metadata Update from @cverna:
- Issue tagged with: backlog

2 years ago

A Fedora host?

@abompard do you have any experience/clues on how to upgrade a rabbitmq cluster
from an OS version to another?

I don't... yet! But @jcline may know, and it's very probably in the docs. Others have had this need before us.

EPEL actually comes with 3.3 or something, we are getting it from the OpenStack channel as far as I know. From what I understand 3.7 will come with the next OpenStack release, no idea on the timeline.

Upgrade docs are https://www.rabbitmq.com/upgrade.html

OS15 comes with 3.7.x... for rhel8.

So, we need fasClient working on rhel8 and enough epel8 stuff for us to run things on rhel8 and then we can use the newer one from os15.

I believe fasClient is now working on RHEL8.

So we need to figure out the dependency list we want in EPEL8.

So, we updated staging with newer version and rhel8.

I guess the next step is to adjust the rabbitmq config and get that all working, then move to prod?

Metadata Update from @cverna:
- Issue untagged with: backlog
- Issue tagged with: high-gain, low-trouble

a year ago

This is live in prod now also because the iad2 datacenter prod has the updated version.

Metadata Update from @smooge:
- Issue tagged with: dev

8 months ago

hello, what's the status of this? Is this planned work?

I would love to see it fixed to see localization events part of our event system (see: https://pagure.io/fedora-infrastructure/issue/8291).

For reference, the topic authorization doc is here: https://www.rabbitmq.com/access-control.html#topic-authorisation

Reading this doc I can see:

Topic authorisation targets protocols like STOMP and MQTT, which are structured around topics and use topic exchanges under the hood.
[..]
The concept of topic authorisation only really makes sense for the topic-oriented protocols such as MQTT and STOMP. In AMQP 0-9-1, for example, consumers consume from queues and thus the standard resource permissions apply.

The doc also points to https://github.com/rabbitmq/rabbitmq-auth-backend-amqp whose last release was 2 years ago.

This doesn't look so good :(

Metadata Update from @pingou:
- Issue priority set to: Next Meeting (was: Waiting on Assignee)

5 months ago

I've made some tests and it seems to work fine :-) I just merged a change in fedora-messaging to handle the "Forbidden" responses that the server might send when publishing on a forbidden topic. So before we can use this I'll need to cut a new release, rebuild the corresponding packages, and rebase the VMs of all applications that send messages so that they pick up the new version. Ideally, those applications would also handle the new exception to log and drop the message.

How do we want to set this up in Ansible? I see the following possibilities:

  • Allow all internal applications to send messages to any topic and only restrict topic for external applications (those that don't run in fedora-infra, maybe cico, koschei, etc. This is the simplest to setup but not entirely secure (compromised internal apps may send messages to any topics).

  • Build a whitelist of allowed topics for every producing application and set them in ansible, as we did with fedmsg. This is harder and more error-prone since we may miss topics. Also, topic permissions are regexp-based, which can also be a source of bugs for apps that send messages on very different topics (but for most a simple prefix should suffice). It's also the most secure.

What do you think?

I've made some tests and it seems to work fine :-) I just merged a change in fedora-messaging to handle the "Forbidden" responses that the server might send when publishing on a forbidden topic. So before we can use this I'll need to cut a new release, rebuild the corresponding packages, and rebase the VMs of all applications that send messages so that they pick up the new version. Ideally, those applications would also handle the new exception to log and drop the message.

How do we want to set this up in Ansible? I see the following possibilities:

  • Allow all internal applications to send messages to any topic and only restrict topic for external applications (those that don't run in fedora-infra, maybe cico, koschei, etc. This is the simplest to setup but not entirely secure (compromised internal apps may send messages to any topics).

  • Build a whitelist of allowed topics for every producing application and set them in ansible, as we did with fedmsg. This is harder and more error-prone since we may miss topics. Also, topic permissions are regexp-based, which can also be a source of bugs for apps that send messages on very different topics (but for most a simple prefix should suffice). It's also the most secure.

Do we have any app that send messages on very different topics?
Normally our standard is org.<origin>.<env>.<app>.<action....>
I can't think of an app sending different app topics.
So, I think we should be able to take the second, more secure, approach :)

Metadata Update from @ryanlerch:
- Issue marked as blocking: #8291

3 months ago

@abompard you were going to do this? whats the status?

Login to comment on this ticket.

Metadata
Boards 1
dev Status: Blocked