It doesn't seem to be rotating it's httpd logs at least.
Someone needs to investigate and fix it.
Metadata Update from @phsmoura: - Issue assigned to phsmoura - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: Needs investigation, medium-gain, medium-trouble, ops
Still investigating. As a work around, for now, I added logrotate to the root crontab
ok, so I added a -v to the service and let it run and:
... Mar 22 00:00:00 proxy31.fedoraproject.org logrotate[3068400]: rotating pattern: /var/log/httpd/*log after 1 days (7 rotations) Mar 22 00:00:00 proxy31.fedoraproject.org logrotate[3068400]: empty log files are rotated, old logs are removed Mar 22 00:00:00 proxy31.fedoraproject.org logrotate[3068400]: No logs found. Rotation not needed.
So, I think whats happening there is that the cron one you added runs at 01:00... and the logrotate is set to run 'daily', but it checks /var/lib/logrotate/logrotate.status to see the last time it was rotated and it's 23hours, not yet a day, so it says... nope, not yet.
So, I've disabled the cron added one and left the -v on the service, so we should be able to check back on this tomorrow and my theory is that it should rotate then, or at least tell us in logs why not. ;)
My theory turned out to be...wrong. ;)
I put the cron back.
So, for some reason it's saying 'no logs found'... why can't it see the httpd/*log files?
Could it be SELinux issue?
No denials and the labels on the files seem right. ;(
I also had trouble to find the root cause, worked on that before going on PTO (about 2 weeks ago) and here are a few things that I checked:
/etc/logrotate.d/httpd
/etc/logrotate.conf
logrotate
The only error I found was an overlaping alias in httpd.conf pointed by error.log, but it doesn't seem to be related.
httpd.conf
error.log
Our infra is coded right? Would be a bad idea to provision that machine again?
Yeah, we could just redeploy it. But that should probibly wait until after freeze.
I'm pretty stumped as to what could be going on.
I wonder if it could be a noaudit selinux thing... and doing a 'fixfiles onboot' and rebooting it (forcing a selinux relabel) is worth trying?
Shouldn't the SELinux errors be visible somewhere even without audit turned on?
Not always. selinux has what it calls 'don't audit' rules, that don't log ever, even when they are hit. This is usually for things that have a lot of denials...
You can turn on logging of noaudit rules too, but it really spews a lot. It's usually easier to just relabel and see if that fixes it.
Then I agree that relabel sounds like a good option. Should we wait for end of freeze?
Yep.
@kevin I think with the freeze over, sssd giving issues, and logrotate being an issue, let just do a clean rebuild and if the issue persists we can take a deeper look.
Proxy31 was redeployed and logrotate is up and running again
Metadata Update from @phsmoura: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.