#8167 Adding topic authorization to our RabbitMQ instances
Opened 2 years ago by abompard. Modified a month ago

Community applications will want to publish messages on the bus (like election in its future CommunityShift home). Currently, any read-write account can publish to any topic, which can be a security issue.

Starting with RabbitMQ 3.7.0, topic authorization is possible, but the version we are running is 3.6.0 since that's what's in EPEL7. If we want to have topic authorizations, we need to upgrade RabbitMQ. Making an infra-specific package seems like a bad idea because of the maintenance it involves. The other way would be to upgrade the servers to RHEL8.


Turns out, rhel8 doesn't seem to have the rabbitmq server in it, only the client libraries. :(

Will have to ponder on a solution here...

Metadata Update from @kevin:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: rabbitmq

2 years ago

Metadata Update from @cverna:
- Issue tagged with: backlog

2 years ago

A Fedora host?

@abompard do you have any experience/clues on how to upgrade a rabbitmq cluster
from an OS version to another?

I don't... yet! But @jcline may know, and it's very probably in the docs. Others have had this need before us.

EPEL actually comes with 3.3 or something, we are getting it from the OpenStack channel as far as I know. From what I understand 3.7 will come with the next OpenStack release, no idea on the timeline.

Upgrade docs are https://www.rabbitmq.com/upgrade.html

OS15 comes with 3.7.x... for rhel8.

So, we need fasClient working on rhel8 and enough epel8 stuff for us to run things on rhel8 and then we can use the newer one from os15.

I believe fasClient is now working on RHEL8.

So we need to figure out the dependency list we want in EPEL8.

So, we updated staging with newer version and rhel8.

I guess the next step is to adjust the rabbitmq config and get that all working, then move to prod?

Metadata Update from @cverna:
- Issue untagged with: backlog
- Issue tagged with: high-gain, low-trouble

2 years ago

This is live in prod now also because the iad2 datacenter prod has the updated version.

Metadata Update from @smooge:
- Issue tagged with: dev

a year ago

hello, what's the status of this? Is this planned work?

I would love to see it fixed to see localization events part of our event system (see: https://pagure.io/fedora-infrastructure/issue/8291).

For reference, the topic authorization doc is here: https://www.rabbitmq.com/access-control.html#topic-authorisation

Reading this doc I can see:

Topic authorisation targets protocols like STOMP and MQTT, which are structured around topics and use topic exchanges under the hood.
[..]
The concept of topic authorisation only really makes sense for the topic-oriented protocols such as MQTT and STOMP. In AMQP 0-9-1, for example, consumers consume from queues and thus the standard resource permissions apply.

The doc also points to https://github.com/rabbitmq/rabbitmq-auth-backend-amqp whose last release was 2 years ago.

This doesn't look so good :(

Metadata Update from @pingou:
- Issue priority set to: Next Meeting (was: Waiting on Assignee)

8 months ago

I've made some tests and it seems to work fine :-) I just merged a change in fedora-messaging to handle the "Forbidden" responses that the server might send when publishing on a forbidden topic. So before we can use this I'll need to cut a new release, rebuild the corresponding packages, and rebase the VMs of all applications that send messages so that they pick up the new version. Ideally, those applications would also handle the new exception to log and drop the message.

How do we want to set this up in Ansible? I see the following possibilities:

  • Allow all internal applications to send messages to any topic and only restrict topic for external applications (those that don't run in fedora-infra, maybe cico, koschei, etc. This is the simplest to setup but not entirely secure (compromised internal apps may send messages to any topics).

  • Build a whitelist of allowed topics for every producing application and set them in ansible, as we did with fedmsg. This is harder and more error-prone since we may miss topics. Also, topic permissions are regexp-based, which can also be a source of bugs for apps that send messages on very different topics (but for most a simple prefix should suffice). It's also the most secure.

What do you think?

I've made some tests and it seems to work fine :-) I just merged a change in fedora-messaging to handle the "Forbidden" responses that the server might send when publishing on a forbidden topic. So before we can use this I'll need to cut a new release, rebuild the corresponding packages, and rebase the VMs of all applications that send messages so that they pick up the new version. Ideally, those applications would also handle the new exception to log and drop the message.

How do we want to set this up in Ansible? I see the following possibilities:

  • Allow all internal applications to send messages to any topic and only restrict topic for external applications (those that don't run in fedora-infra, maybe cico, koschei, etc. This is the simplest to setup but not entirely secure (compromised internal apps may send messages to any topics).

  • Build a whitelist of allowed topics for every producing application and set them in ansible, as we did with fedmsg. This is harder and more error-prone since we may miss topics. Also, topic permissions are regexp-based, which can also be a source of bugs for apps that send messages on very different topics (but for most a simple prefix should suffice). It's also the most secure.

Do we have any app that send messages on very different topics?
Normally our standard is org.<origin>.<env>.<app>.<action....>
I can't think of an app sending different app topics.
So, I think we should be able to take the second, more secure, approach :)

Metadata Update from @ryanlerch:
- Issue marked as blocking: #8291

6 months ago

@abompard you were going to do this? whats the status?

Updating here with status:

Upstream rabbitmq has taken the patch from @abompard that was needed and merged it. They have not yet done a release.

https://github.com/ansible-collections/community.rabbitmq/pull/73

However, ansible 2.9 will likely never get this change (it eols at the end of the year). We will likely switch batcave01 to ansible-core/collections later this year, after that we can get this fix and move this forward again.

So, now we just need someone to package up ansible-collections-community-rabbitmq so we can use it. ;)

Any takers?

I think @petebuffon was willing to take it.

If not, I can be second line volunteer :)

I'll take a crack at it. @lenkaseg if you wanna work on it too then let's do it.

I'm still learning up on ansible-collections. Could this be as easy as ansible-galaxy collection install community.rabbitmq, seen here?

If not then it would be helpful to get a link to some other collections Fedora Infra has packaged to get started.

I'm trying to find if there is somewhere to look how to package a collection in Fedora-infra...

Could this be as easy as ansible-galaxy collection install community.rabbitmq, seen here?

I have never done it before, but this seems more like what would be done after packaging from batcave (?). But how to package...I don't know where to look to see where the others packacked collections are.

So I found this ansible-collection-community-mysql repo and there is this spec file:
https://src.fedoraproject.org/rpms/ansible-collection-community-mysql/blob/f34/f/ansible-collection-community-mysql.spec

Probably for the rabbitmq we should do the same thing. Create a package with a similar spec file, using the same ansible macros.

(would find it yesterday probably, but at 11PM I mistook pagure.io for src.fpo since it has the same design and wondered why I cannot find any collections there :) )

@lenkaseg exactly what I was going to suggest.

After we have a spec/package we could just directly use it ourselves, and/or we could take it through the process to be reviewed and officially added to Fedora and then EPEL. :)

Probibly we want to use it directly once we have it, but also in the background move forward to make it an official package.

Hi @petebuffon!
Any progress on the collection packaging? Can I be of some help?

@lenkaseg No progress yet. I've been sick the past few days, but I'm ready to work on the collection packaging. Want to chat about it over on IRC?

@lenkaseg check out the new repo I made, ansible-collection-community-rabbitmq. Want to give it a look and let me know what you think?

@lenkaseg check out the new repo I made, ansible-collection-community-rabbitmq. Want to give it a look and let me know what you think?

Cool!
In the spec file where there is the Source url:
https://pagure.io/ansible-collection-community-rabbitmq/blob/master/f/ansible-collection-community-rabbitmq.spec#_12
I see that in the mysql collection the address is in the same format:
https://github.com/ansible-collections/community.rabbitmq/archive/%{version}/%{name}-%{version}.tar.gz,
but when I check in the rabbitmq github repo, the tar.gz file seems to be here:
https://github.com/ansible-collections/community.rabbitmq/archive/refs/tags/1.1.0.tar.gz
I wonder why.

@lenkaseg check out the new repo I made, ansible-collection-community-rabbitmq. Want to give it a look and let me know what you think?

Cool!
In the spec file where there is the Source url:
https://pagure.io/ansible-collection-community-rabbitmq/blob/master/f/ansible-collection-community-rabbitmq.spec#_12
I see that in the mysql collection the address is in the same format:
https://github.com/ansible-collections/community.rabbitmq/archive/%{version}/%{name}-%{version}.tar.gz,
but when I check in the rabbitmq github repo, the tar.gz file seems to be here:
https://github.com/ansible-collections/community.rabbitmq/archive/refs/tags/1.1.0.tar.gz
I wonder why.

I just followed the same path as the mysql collection. Both are valid and go to the see tar file.

The spec/package are ready. What is the next step for building on batcave01? And then I'll move forward with submitting to Fedora and Epel as well.

The spec/package are ready. What is the next step for building on batcave01? And then I'll move forward with submitting to Fedora and Epel as well.

I'm not sure we want to update this during freeze, so I would say go ahead and submit it for review and we can get it added to Fedora/EPEL, if it's not done after freeze is over we can build it in infra tags, and if it is we can just use the EPEL one. ;) Please feel free to cc me on the bug and I can review...

Login to comment on this ticket.

Metadata
Boards 1
dev Status: Blocked