#11144 Create monitoring tool for rabbitmq certificates
Opened a year ago by zlopez. Modified a month ago

Describe what you would like us to do:


After the conversation with @arrfab and the unexpected expiration of ODCS fedora messaging certificates it would be nice to create a tool (probably cron job) that will watch the expiration dates for rabbitmq certs and send notification about the expiration. It would be also nice if the tool would send the notification to anybody responsible for the relates service as well. I'm not sure if we have information who's that.

When do you need this to be done by? (YYYY/MM/DD)


This is not urgent, but would really help to prevent another incident with expired certificates


Metadata Update from @zlopez:
- Issue untagged with: ops
- Issue tagged with: dev

a year ago

@zlopez could you assigne to me this ticket
I can take a look .

Metadata Update from @zlopez:
- Issue assigned to seddik

a year ago

@seddik Done, feel free to play with it.

Adding more info for anybody, who would like to work on this ticket. The certs are in ansible private repository (more details about them could be find in How-To for fedora-messaging certs), so to get to them you need to be in sysadmin-main.

@kevin @abompard Is there a way to get to public certs without being in sysadmin-main? Do we expose them somewhere?

@zlopez thank you for your answer ,,
it would be easy for me to check certs expiration !, otherwise i don't know which cluster / host will be verified .
Could you share this info ?

thanks

Is there a way to get to public certs without being in sysadmin-main? Do we expose them somewhere?

I suppose you could test the TLS cert offered by the AMQP connection, no? It would be on port 5671 on rabbitmq.fp.o.

So, currently I don't think there's any way. I guess we could publish/sync somewhere the public certs? Perhaps to just a dir under https://infrastructure.fedoraproject.org ?

The AMQP connection will have the actual server cert, but not any of the client ones (which I think this is about?)

The AMQP connection will have the actual server cert, but not any of the client ones (which I think this is about?)

Oh, right, I misunderstood.

So, currently I don't think there's any way. I guess we could publish/sync somewhere the public certs? Perhaps to just a dir under https://infrastructure.fedoraproject.org ?

Yeah that would work.

@kevin any feedback ?? finally you will expose the certs ??
thanks

This is still on my radar and I hope to do this soon (next week)...

I also talked with @t0xic0der about our cert setup in general and he's going to work on this area as well. I've asked him to see if he can collaborate with you on some of this and move it forward better/faster rather than being blocked on me. ;)

@kevin i totally agree :)
@t0xic0der let me know when you're ready :wave:

@seddik, nice to make your acquaintance!

Let's communicate over at https://chat.fedoraproject.org/#/room/#admin:fedoraproject.org. For DMs, you can reach out to me on Telegram using my username "@t0xic0der".

This is now complete.

The project repository is now available here. It is now available for installation on PyPI and packaged in the official RPM repositories for Fedora 37, Fedora 38, Rawhide and most importantly, EPEL 9.

I have opened some issue tickets to further progress with improving the notification service project, fixing possible bugs etc. and am constantly collaborating with @seddik on this.

@zlopez, what do you suggest we do next?

Metadata Update from @zlopez:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

11 months ago

Metadata Update from @zlopez:
- Issue status updated to: Open (was: Closed)

6 months ago

Metadata Update from @zlopez:
- Assignee reset

6 months ago

This still needs to be deployed in fedora infrastructure.

@zlopez which machine is storing rabbitMQ ca files ??
Also there are many input requirements needed for the tool, you can refer to this standard file config https://gitlab.com/gridhead/firmitas/-/blob/main/firmitas/conf/standard.py?ref_type=heads

Example of ticket that will be generated by tool => https://pagure.io/firmitas-notifier/issue/20
I could work on the install of the tool on the target host.

@seddik The RabbitMQ certificates are in ansible-private repository on batcave.

Feel free to take this one if you have spare time :-)

I don't know how without access to ansible private repo :(

loop @kevin

@seddik I think only sysadmin-main members has access, but you should be able to request what you need here and just deploy the firmitas in openshift

Yes i will look into sure !

@kevin is there any reason to install the tool on openshift ?? I'm just curious to know what's the main advantage for that :)
At least , could you print somewhere the list of all certificate (path of crt files) we need to check ??

thanks

https://www.zabbix.com/integrations/rabbitmq might be worth looking at using Zabbix to perform this monitoring. Check to see if this rabbitmq integration could be used or if we need to create some custom.

@dkirwan thank you for the suggestion, but I would say it should be discussed with the members ;)

So, the certs for this are in the ansible-private git repo on batcave01, and also distributed across the universe to clients.

I don't think we can query rabbitmq, because these certs are using for authentication, so we would have no way of asking it 'hey, if we were xyz, would you accept our cert' ? :)

I think it might be ok if we exported just the certs (since clients need the private key also to use them). If we did that we could export them to somewhere web accessable on batcave01 and query it from a openshift pod version of this?

Agreed to expose certificates somewhere, but currently Firmitas can ony read only crt files from local directory path .

@kevin what do you think ? do you agree to add this option ? query remote crt files from web?

Sorry this dropped off my radar.

I guess it's waiting on me syncing the crt's somewhere that can be read so they can be monitored?

Metadata Update from @zlopez:
- Issue assigned to t0xic0der

a month ago

Login to comment on this ticket.

Metadata
Boards 1
dev Status: Backlog