#9512 various Fedora servers are not logging correctly.
Opened 4 months ago by smooge. Modified 11 days ago

Describe what you would like us to do:


In trying to debug problems this week, I have found that both log01 and the server itself do not have all the logs that should have been reported. In most cases they were in the journal but had been dropped

Dec 02 00:25:10 mailman01.iad2.fedoraproject.org rsyslogd[1016]: imjournal: begin to drop messages due to rate-limiting
Dec 02 00:30:13 mailman01.iad2.fedoraproject.org rsyslogd[1016]: imjournal: 25919 messages lost due to rate-limiting
Dec 02 00:34:21 mailman01.iad2.fedoraproject.org rsyslogd[1016]: imjournal: begin to drop messages due to rate-limiting
Dec 02 00:40:14 mailman01.iad2.fedoraproject.org rsyslogd[1016]: imjournal: 29598 messages lost due to rate-limiting
Dec 02 00:45:03 mailman01.iad2.fedoraproject.org rsyslogd[1016]: imjournal: begin to drop messages due to rate-limiting
Dec 02 00:48:04 mailman01.iad2.fedoraproject.org rsyslogd[1016]: imjournal: journal reloaded... [v8.24.0-57.el7_9 try http://www.rsyslog.com/e/0 ]
Dec 02 00:48:04 mailman01.iad2.fedoraproject.org rsyslogd[1016]: imjournal: journal reloaded... [v8.24.0-57.el7_9 try http://www.rsyslog.com/e/0 ]

When do you need this to be done by? (YYYY/MM/DD)


We need to track down why the journal and rsyslogd are throttling each other.

https://access.redhat.com/solutions/1417483

may be helpful


Metadata Update from @smooge:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: medium-gain, medium-trouble, ops

4 months ago

Metadata Update from @smooge:
- Issue untagged with: medium-trouble
- Issue tagged with: high-trouble

4 months ago

Metadata Update from @smooge:
- Issue untagged with: medium-gain
- Issue tagged with: high-gain

4 months ago

In general we also should try and make our servers log less stuff. koji is particularly bad...

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog