If the reason for introducing pgbdr was to reduce outages, then it's doing the opposite. Old db.stg used to be mostly stable. But now I have a feeling that most of the time I try to do anything with our stg instance, pgbdr is either down or in some degraded state.
That seems like a bit of hyperbole... it was down yesterday and today (It's the same issue, I need to find and fix the root cause), but to my knowledge it hasn't been down at all before that for quite a long time.
It should be back up now and I will look into the issue to prevent it from happening again...
down again now
So, the issue here seems to have happened when there was some kind of network hiccup and there was a failover from 01 to 02 that lasted only a few seconds. It then failed back. However, transactions that were in progress got messed up and then everything was out of sync.
I have changed the keepalived scripts to restart on becoming backup server. This means in the above case that the failover would have stopped postgres and anything in progress and then restarted it.
Patrick also cleaned up the conflict caused by the above so everything should be back to normal now as far as I know.
:baseball:
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)