#10693 Track down mirrormanager database problem
Closed: Fixed 2 years ago by kevin. Opened 2 years ago by smooge.

Describe what you would like us to do:


Mirrormanager has been reporting for a couple of week the following tracebacks:

 Message type:       ERROR
    Location:           /usr/lib/python2.7/site-packages/flask/app.py:1423
    Module:             app
    Function:           log_exception
    Time:               2022-05-15 21:52:52,529
...
OperationalError: (OperationalError) could not connect to server: No route to host
        Is the server running on host "db01" (10.3.163.41) and accepting
        TCP/IP connections on port 5432?
 None None

We can get 2-4 per day of this. Need to determine if this is a problem with db01 or a problem with mirrormanager and if it is a
1. Fixable
2. Ignorable.

When do you need this to be done by? (YYYY/MM/DD)



Metadata Update from @phsmoura:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: medium-gain, medium-trouble, ops

2 years ago

So, I see two of these yesterday, but... I completely hosed up all our IAD2 networks yesterday, so 99.99999% likely it was that.

                                                 │6986     2022-03-09 nobody@fedorapr (3.9K) (1/1) MirrorManager2 error
                                                 │6987     2022-03-09 nobody@fedorapr (3.9K) (1/1) MirrorManager2 error
                                                 │7053     2022-03-31 nobody@fedorapr (3.8K) (1/1) MirrorManager2 error
                                                 │7056     2022-03-31 nobody@fedorapr (3.9K) (1/1) MirrorManager2 error
                                                 │7057     2022-03-31 nobody@fedorapr (3.9K) (1/1) MirrorManager2 error
                                                 │7085     2022-04-27 nobody@fedorapr (4.1K) (1/1) MirrorManager2 error
                                                 │7102     2022-05-15 nobody@fedorapr (4.1K) (1/1) MirrorManager2 error
                                                 │7103     2022-05-15 nobody@fedorapr (3.9K) (1/1) MirrorManager2 error

03-31 was a mass upgrade/reboot...

the 04-27 one seems to be dns...

Metadata Update from @kevin:
- Issue untagged with: medium-gain, medium-trouble, ops
- Issue priority set to: Needs Review (was: Waiting on Assignee)

2 years ago

So, I see two of these yesterday, but... I completely hosed up all our IAD2 networks yesterday, so 99.99999% likely it was that.

                                                 │6986     2022-03-09 nobody@fedorapr (3.9K) (1/1) MirrorManager2 error
                                                 │6987     2022-03-09 nobody@fedorapr (3.9K) (1/1) MirrorManager2 error
                                                 │7053     2022-03-31 nobody@fedorapr (3.8K) (1/1) MirrorManager2 error
                                                 │7056     2022-03-31 nobody@fedorapr (3.9K) (1/1) MirrorManager2 error
                                                 │7057     2022-03-31 nobody@fedorapr (3.9K) (1/1) MirrorManager2 error
                                                 │7085     2022-04-27 nobody@fedorapr (4.1K) (1/1) MirrorManager2 error
                                                 │7102     2022-05-15 nobody@fedorapr (4.1K) (1/1) MirrorManager2 error
                                                 │7103     2022-05-15 nobody@fedorapr (3.9K) (1/1) MirrorManager2 error

03-31 was a mass upgrade/reboot...

the 04-27 one seems to be dns...

Metadata Update from @kevin:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: medium-gain, medium-trouble, ops

2 years ago

Ok thanks for tracking that down. I wasn't sure what might have caused the issues so wanted to get it logged before I forgot.

I'm gonna close this one now... but if you see more or a pattern, please reopen...

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog