We are now centralizing all our selinux AVC messages to our log02 host.
We would like to start processing these logs and report on things.
Logs are in /var/log/merged/audit.log daily.
We want to report on two groups (which may mean two passes of logs and/or two emails):
For each host that has denials we want a total number and then a pass through 'audit2why' for each of them. If there's some easy way to collapse the very similar denials and simply report that it happened 50 times, that would be great.
The script should output an email daily that sends to sysadmin-logs-members (or some other suitable group). We may want to look at just the enforcing hosts to start with.
If the script turns out nicely it may be worth making a project for and releasing so others could use and enjoy it.
would aureport be useful for this task?
Possibly.
It doesn't really seem to be very detailed. It's good for a summary of all denied things... but less so per host or telling you why something was denied. ie, it can easily give you:
{{{ Avc Object Summary Report ================================= total obj ================================= 491 system_u:object_r:auditd_log_t:s0 222 unconfined_u:object_r:user_tmp_t:s0 181 system_u:object_r:nfs_t:s0 ... }}}
but doesn't tell you much about what machines need fixing and why those are happening. We could use this for the base summary and grep out each case to see what hosts/what do do about it? Or just start from scratch. ;)
attachment avc.py
For the brave soul that wants to try, point AUDIT_LOG to the log file first;)
https://fedorahosted.org/fedora-infrastructure/browser/scripts/selinux/selinux-overlord.py
I wrote this tool a while ago. It uses func and can perform mass selinux-policy upgrades, query the enforcing/permissive/disabled status, fetching all enforced denials, and dump the full audit logs.
well, the overlord is interesting, but not really what we were looking for here... we more want to have a nice report on the avc's and what to do to fix them...
jacibato: interesting script, but also not sure it's quite what we want.
Seth noticed that 'audit2allow' handles uniq-ifying things for us, so that might be something to go for.
$ cat /var/log/merged/audit.log | sed 's/^.*tag_audit_log: //' | audit2allow | wc -l 243
$ wc -l /var/log/merged/audit.log 71889 /var/log/merged/audit.log
A pretty good reduction. Of course some of those issues apply to multiple machines and/or happen multiple times a day.
attachment avc2.py
It might be good to note the selinux mode of the hosts affected... thats now in infra-hosts:
http://infrastructure.fedoraproject.org/infra/hosts/app01.dev.fedoraproject.org/autoinfo/selinux_current_mode
Otherwise it's starting to look pretty nice. ;)
attachment avc3.py
Updated the script to look up selinux mode... there are quite a few hostnames from the logfile that do not match the hostnames on infra-hosts.
Yeah, thats due to the vpn vs non vpn I fear. You can sometimes get around this by just looking at the base hostname instead of fqdn.
Any thoughts on how we can deal with the:
libsepol.context_from_record: invalid security context: "user_u:system_r:httpd_t:s0" libsepol.context_from_record: could not create context structure libsepol.context_from_string: could not create context structure
stuff? Is that due to something we don't have installed on log02? Or should we just filter it out?
Would you like to make a puppet patch to add this as a script and cron job? Note that we possibly will want to wait until after the freeze to apply it.
great work though. Thanks!
Regarding the hostnames, I can't make a wild guess where to point these three - admin.fedoraproject.org, archives.fedoraproject.org and infrastructure.fedoraproject.org.[[BR]]
Also, ask01.fedoraproject.org and pkgs.fedoraproject.org have staging(stg) counterparts. Just need your confirmation as I'm leaning to point these to the regular hosts and not the stg ones.[[BR]]
About that invalid security context, I'm also getting the same error on my machine with apache running on it. My only guess was that of a semodule not being loaded so I checked for that. However, all of the targeted semodules are loaded so that could not be it. Correct me if I'm wrong with my assumption though. If you got any pointer on how to approach this, I'd appreciate it a lot.[[BR]]
Anyway, I can cross-reference the error against the log to see which node(s) it's coming from and include that with the report if that helps.[[BR]]
Also, any preference as to what time of day should the email be sent? I'll work on the patch as soon as we get these deatails ironed out. Thanks again.
archive.fedoraproject.org => secondary01.phx2.fedoraproject.org
admin.fedoraproject.org => proxy01.phx2.fedoraproject.org
infrastructure.fedoraproject.org => lockbox01.phx2.fedoraproject.org
yes, ask01 and pkgs01 are the the production instances... ask01.stg and pkgs01.stg are the stg ones.
Yeah, I can ask around about that invalid thing...
I think it might make sense to send the email after logrotate time? Just post process it after logrotate and then we have a full days worth of audit logs...
attachment 1.patch
Added initial patch through scripts module, let me know if it needs to go somewhere else. Anyway, I'll just repatch it again the moment we have something for the invalid thing.
Ok, commited. ;)
Thanks for all your work on this.
I guess we will see how useful this is and adjust from there.
Login to comment on this ticket.