In trying to debug problems this week, I have found that both log01 and the server itself do not have all the logs that should have been reported. In most cases they were in the journal but had been dropped
Dec 02 00:25:10 mailman01.iad2.fedoraproject.org rsyslogd[1016]: imjournal: begin to drop messages due to rate-limiting Dec 02 00:30:13 mailman01.iad2.fedoraproject.org rsyslogd[1016]: imjournal: 25919 messages lost due to rate-limiting Dec 02 00:34:21 mailman01.iad2.fedoraproject.org rsyslogd[1016]: imjournal: begin to drop messages due to rate-limiting Dec 02 00:40:14 mailman01.iad2.fedoraproject.org rsyslogd[1016]: imjournal: 29598 messages lost due to rate-limiting Dec 02 00:45:03 mailman01.iad2.fedoraproject.org rsyslogd[1016]: imjournal: begin to drop messages due to rate-limiting Dec 02 00:48:04 mailman01.iad2.fedoraproject.org rsyslogd[1016]: imjournal: journal reloaded... [v8.24.0-57.el7_9 try http://www.rsyslog.com/e/0 ] Dec 02 00:48:04 mailman01.iad2.fedoraproject.org rsyslogd[1016]: imjournal: journal reloaded... [v8.24.0-57.el7_9 try http://www.rsyslog.com/e/0 ]
We need to track down why the journal and rsyslogd are throttling each other.
https://access.redhat.com/solutions/1417483
may be helpful
Metadata Update from @smooge: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: medium-gain, medium-trouble, ops
Metadata Update from @smooge: - Issue untagged with: medium-trouble - Issue tagged with: high-trouble
Metadata Update from @smooge: - Issue untagged with: medium-gain - Issue tagged with: high-gain
In general we also should try and make our servers log less stuff. koji is particularly bad...
Login to comment on this ticket.