#9662 Stop staging from repeating prod fedmsg/fedora-messages
Closed: Fixed a month ago by kevin. Opened 2 months ago by ankursinha.

Describe what you would like us to do:

Please disable the staging fedmsg bot in #fedora-neuro. (We're getting duplicate notifications)


When do you need this to be done by? (YYYY/MM/DD)

Whenever convenient please, not very urgent.

Yes, every channel is getting these. We need to figure out where it's getting crossed and fix it.

Metadata Update from @smooge:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: medium-gain, medium-trouble, ops

2 months ago

The problem is the fedmsg policy on busgateway01.stg.

However, I have no idea why it's now wrong... it has to be some change in ansible recently. ;(

This seems to be fixed in #fedora-admin, do you still have the issue in -neuro?

Seems to be fixed there too. Thanks! Closing.

Metadata Update from @ankursinha:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 months ago

Metadata Update from @ankursinha:
- Issue status updated to: Open (was: Closed)

2 months ago

Yeah, we need to figure out why the config is going wrong here and sending all the prod messages to staging. ;(

ok. I think I have this fixed. It was a fun road. :)

In phx2 datacenter we had a firewall rule on all prod machines that would drop everything from stg. In iad2, we didnt do this because there's firewall between the various vlans and we didn't think we needed it. However, there's some things (like the fedmsg ports) that are open accross all vlans, so fedmsg is able to send / connect accross stg and prod.
Additionally, since time began fedmsg in stg actually had all the prod servers in it too. They just didn't work because of the firewall rule. Once that was gone everything actually connected and messages started flowing.
Additionally, the fedmsg/base role was disabled for a number of staging hosts because of the sysadmin group being not there. (This was pre noggin deployment). Which meant these hosts just always kept the initial fedmsg config and running the playbooks did nothing to change it.

I fixed the ansible fedmsg filter plugin to only put staging hosts in stg policy.py files.
I fixed playbooks to run fedmsg/base on staging hosts.
I ran all the playbooks to make sure everything synced up.

I think things are back to ok now.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

a month ago

Issue status updated to: Open (was: Closed)

a month ago

Issue status updated to: Closed (was: Open)
Issue close_status updated to: Fixed

a month ago

Login to comment on this ticket.

Boards 1
ops Status: Done