Most (but not all) of our hosts should log syslog to log02. They should appear under /var/log/hosts/<hostname>/YYYY/MM/DD/ via rsyslog.
We should check and confirm:
Current list of all hosts in puppet:
{{{
app01.dev.fedoraproject.org app01.phx2.fedoraproject.org app01.stg.phx2.fedoraproject.org app02.phx2.fedoraproject.org app02.stg.phx2.fedoraproject.org app03.phx2.fedoraproject.org app04.phx2.fedoraproject.org app05.fedoraproject.org app07.phx2.fedoraproject.org app6.fedoraproject.org backup01.phx2.fedoraproject.org backup02.fedoraproject.org bapp01.phx2.fedoraproject.org bastion01.phx2.fedoraproject.org bastion02.phx2.fedoraproject.org bastion-comm01.qa.fedoraproject.org bnfs01.phx2.fedoraproject.org bodhost01.fedoraproject.org bvirthost01.phx2.fedoraproject.org bvirthost02.phx2.fedoraproject.org bvirthost03.phx2.fedoraproject.org bxen03.phx2.fedoraproject.org bxen04.phx2.fedoraproject.org cnode01.fedoraproject.org collab1.fedoraproject.org collab2.fedoraproject.org compose-x86-01.phx2.fedoraproject.org db01.stg.phx2.fedoraproject.org db02.phx2.fedoraproject.org db04.phx2.fedoraproject.org db05.phx2.fedoraproject.org dhcp01.phx2.fedoraproject.org dhcp02.c.fedoraproject.org download01.phx2.fedoraproject.org download02.phx2.fedoraproject.org download03.phx2.fedoraproject.org download04.phx2.fedoraproject.org download05.phx2.fedoraproject.org download-rdu01.fedoraproject.org download-rdu02.fedoraproject.org download-rdu03.fedoraproject.org fakefas01.fedoraproject.org fas01.phx2.fedoraproject.org fas01.stg.phx2.fedoraproject.org fas02.phx2.fedoraproject.org fas03.phx2.fedoraproject.org hosted1.fedoraproject.org hosted2.fedoraproject.org ibiblio01.fedoraproject.org insight01.dev.fedoraproject.org insight01.phx2.fedoraproject.org insight01.stg.phx2.fedoraproject.org internetx01.fedoraproject.org koji01.phx2.fedoraproject.org koji01.stg.phx2.fedoraproject.org koji02.phx2.fedoraproject.org kojipkgs01.phx2.fedoraproject.org lockbox01.phx2.fedoraproject.org log02.phx2.fedoraproject.org memcached03.phx2.fedoraproject.org memcached04.phx2.fedoraproject.org nfs01.phx2.fedoraproject.org noc01.phx2.fedoraproject.org noc01.stg.phx2.fedoraproject.org noc02.fedoraproject.org ns02.fedoraproject.org ns03.phx2.fedoraproject.org ns04.phx2.fedoraproject.org ns05.fedoraproject.org ns1.fedoraproject.org osuosl1.fedoraproject.org people02.fedoraproject.org pkgs01.phx2.fedoraproject.org pkgs01.stg.phx2.fedoraproject.org ppc05.phx2.fedoraproject.org ppc06.phx2.fedoraproject.org ppc07.phx2.fedoraproject.org ppc08.phx2.fedoraproject.org ppc09.phx2.fedoraproject.org ppc10.phx2.fedoraproject.org ppc12.phx2.fedoraproject.org proxy01.phx2.fedoraproject.org proxy01.stg.phx2.fedoraproject.org proxy02.fedoraproject.org proxy04.fedoraproject.org proxy07.fedoraproject.org proxy3.fedoraproject.org proxy5.fedoraproject.org proxy6.fedoraproject.org publictest01.fedoraproject.org publictest02.fedoraproject.org publictest04.fedoraproject.org releng01.phx2.fedoraproject.org releng01.stg.phx2.fedoraproject.org releng02.phx2.fedoraproject.org relepel01.phx2.fedoraproject.org retrace01.fedoraproject.org secondary01.phx2.fedoraproject.org serverbeach1.fedoraproject.org serverbeach2.fedoraproject.org serverbeach3.fedoraproject.org serverbeach4.fedoraproject.org serverbeach5.fedoraproject.org sign-bridge01.phx2.fedoraproject.org sign-vault01.phx2.fedoraproject.org smolt01.phx2.fedoraproject.org smtp-mm01.fedoraproject.org smtp-mm02.fedoraproject.org smtp-mm03.fedoraproject.org telia1.fedoraproject.org torrent01.fedoraproject.org tummy1.fedoraproject.org value01.phx2.fedoraproject.org value01.stg.phx2.fedoraproject.org value02.phx2.fedoraproject.org virthost01.phx2.fedoraproject.org virthost02.phx2.fedoraproject.org virthost03.phx2.fedoraproject.org virthost13.phx2.fedoraproject.org virthost-comm01.qa.fedoraproject.org x86-01.phx2.fedoraproject.org x86-02.phx2.fedoraproject.org x86-03.phx2.fedoraproject.org x86-04.phx2.fedoraproject.org x86-05.phx2.fedoraproject.org x86-06.phx2.fedoraproject.org x86-07.phx2.fedoraproject.org x86-09.phx2.fedoraproject.org x86-10.phx2.fedoraproject.org x86-11.phx2.fedoraproject.org x86-12.phx2.fedoraproject.org x86-13.phx2.fedoraproject.org x86-14.phx2.fedoraproject.org x86-15.phx2.fedoraproject.org x86-16.phx2.fedoraproject.org x86-17.phx2.fedoraproject.org x86-18.phx2.fedoraproject.org xen03.phx2.fedoraproject.org xen04.phx2.fedoraproject.org xen05.phx2.fedoraproject.org xen09.phx2.fedoraproject.org xen15.phx2.fedoraproject.org
}}}
(list can be regenerated on demand).
Note that this ticket could be worked on by several apprentice folks, as long as they coordinate which hosts they are checking.
Just an update here. Currently apprentice folks are not allowed to login to log02. Discussion about this is taking place on the mailing list and I will update this ticket when more info is known.
Hi Kevin and lbazan, I would like to work on this ticket with you. I am thinking that I could take the bottom 1/2 of these machines and check from the machine side rather than waiting for access to log02. I can begin by poking around and annotatitng the ticket if I find a machine that is logging somewhere else.
I created this script:
SERVER=$1 YEAR=date +%Y MONTH=date +%m DAY=date +%d
date +%Y
date +%m
date +%d
if [ -d /var/log/hosts/$SERVER/$YEAR/$MONTH/$DAY ] then echo "OK - Path found successfully" else echo "CRITICAL - Path not found" fi
I think we can use it for monitor this path.
What you think about it?
Can I reassign for me?
I've thought of possibly another way to do this. We have a script that parses logs and does a report. We may be able to add a module there for 'note when there are 0 lines from hosts on a list' or the like.
Let me investigate that for a bit.
I have checked a few of these machines and believe that they are not set up to log in the format requested. They are in fact set up to log to log2 via tcp, but the default line is as follows: ssh people02.fedoraproject.org grep "log02" /etc/rsyslog.conf cron.;kern.;authpriv.;local7.;*.info @@log02:514
To get these boxes to log in the correct format using rsyslogd, we will have to add lines to rsyslog.conf that look like the following:
$template DynFile,"/var/log/syslog/system-%HOSTNAME%-%$YEAR%-%$MONTH%-%$DAY%-messages.log" :fromhost-ip,!isequal,"127.0.0.1" -?DynFile & ~
I think that we can roll this out pretty easily with puppet, but I am not an expert at puppet and would need to do some reading or get help.
The other thing that I noticed is that I don't have access to all the machines in the second half of this list. I was able to check the following boxes, and they all had the same default log config to log2:
proxy02.fedoraproject.org y Default format proxy04.fedoraproject.org y Default format proxy07.fedoraproject.org y Default format proxy3.fedoraproject.org y Default format proxy5.fedoraproject.org y Default format proxy6.fedoraproject.org y Default format publictest01.fedoraproject.org y Default format publictest02.fedoraproject.org y Default format publictest04.fedoraproject.org y Default format serverbeach1.fedoraproject.org y Default format smtp-mm01.fedoraproject.org y Default format smtp-mm02.fedoraproject.org y Default format smtp-mm03.fedoraproject.org y Default format torrent01.fedoraproject.org y Default format
I am assuming that the rest are set up like these, so if everyone agrees, I think that we can set up a puppet config for rsyslog that logs in the format that we need. We can then check our formats and file locations on log2 when we get access.
Also, one last thought. We will also need to check iptables on log2 to make sure that tcp 514 is open to these boxes. I am assuming that it is since this where the current rsyslog.conf configs are telling rsyslogd to log.
ok, log02 access should be open for apprentice folks.
I also did a cleanup of directories, so /var/log/hosts/ should be currently logging hosts only.
so, I think the next step here is to check those /var/log/hosts/ directories against a current puppet list and see whats missing. The monitoring or checking for log lines isn't going to work out because we have some hosts that don't log anything when they don't have any activity to log (the mgmt* ones are like this).
I can attach a current puppet hosts list.
A current list can be found at:
http://infrastructure.fedoraproject.org/infra/hosts/
app01.dev.fedoraproject.org ... Missing[[BR]] app05.fedoraproject.org ... Missing[[BR]] app6.fedoraproject.org ... Missing[[BR]] backup01.phx2.fedoraproject.org ... Missing[[BR]] backup02.fedoraproject.org ... Missing[[BR]] backup03.phx2.fedoraproject.org ... Missing[[BR]] bodhost01.fedoraproject.org ... Missing[[BR]] collab1.fedoraproject.org ... Missing[[BR]] collab2.fedoraproject.org ... Missing[[BR]] download-rdu01.fedoraproject.org ... Missing[[BR]] download-rdu02.fedoraproject.org ... Missing[[BR]] download-rdu03.fedoraproject.org ... Missing[[BR]] fakefas01.fedoraproject.org ... Missing[[BR]] hosted1.fedoraproject.org ... Missing[[BR]] hosted2.fedoraproject.org ... Missing[[BR]] ibiblio01.fedoraproject.org ... Missing[[BR]] ibiblio02.fedoraproject.org ... Missing[[BR]] insight01.dev.fedoraproject.org ... Missing[[BR]] insight02.dev.fedoraproject.org ... Missing[[BR]] internetx01.fedoraproject.org ... Missing[[BR]] kojipkgs01.phx2.fedoraproject.org ... Missing[[BR]] lockbox01.phx2.fedoraproject.org ... Missing[[BR]] log02.phx2.fedoraproject.org ... Missing[[BR]] noc02.fedoraproject.org ... Missing[[BR]] ns02.fedoraproject.org ... Missing[[BR]] ns05.fedoraproject.org ... Missing[[BR]] ns1.fedoraproject.org ... Missing[[BR]] osuosl01.fedoraproject.org ... Missing[[BR]] paste01.dev.fedoraproject.org ... Missing[[BR]] people02.fedoraproject.org ... Missing[[BR]] pkgs01.phx2.fedoraproject.org ... Missing[[BR]] proxy02.fedoraproject.org ... Missing[[BR]] proxy04.fedoraproject.org ... Missing[[BR]] proxy07.fedoraproject.org ... Missing[[BR]] proxy3.fedoraproject.org ... Missing[[BR]] proxy5.fedoraproject.org ... Missing[[BR]] proxy6.fedoraproject.org ... Missing[[BR]] publictest01.fedoraproject.org ... Missing[[BR]] publictest02.fedoraproject.org ... Missing[[BR]] publictest04.fedoraproject.org ... Missing[[BR]] retrace01.fedoraproject.org ... Missing[[BR]] secondary01.phx2.fedoraproject.org ... Missing[[BR]] serverbeach06.fedoraproject.org ... Missing[[BR]] serverbeach07.fedoraproject.org ... Missing[[BR]] serverbeach08.fedoraproject.org ... Missing[[BR]] serverbeach09.fedoraproject.org ... Missing[[BR]] serverbeach1.fedoraproject.org ... Missing[[BR]] serverbeach2.fedoraproject.org ... Missing[[BR]] serverbeach3.fedoraproject.org ... Missing[[BR]] serverbeach4.fedoraproject.org ... Missing[[BR]] serverbeach5.fedoraproject.org ... Missing[[BR]] serverbeach10.fedoraproject.org ... Missing[[BR]] sign-bridge01.phx2.fedoraproject.org ... Missing[[BR]] sign-vault01.phx2.fedoraproject.org ... Missing[[BR]] smtp-mm01.fedoraproject.org ... Missing[[BR]] smtp-mm02.fedoraproject.org ... Missing[[BR]] smtp-mm03.fedoraproject.org ... Missing[[BR]] telia1.fedoraproject.org ... Missing[[BR]] torrent01.fedoraproject.org ... Missing[[BR]] torrent02.fedoraproject.org ... Missing[[BR]] tummy1.fedoraproject.org ... Missing[[BR]] x86-11.phx2.fedoraproject.org ... Missing[[BR]] ------------------------------------------[[BR]] Total hosts found by puppet = 158[[BR]] Total hosts with missing parent dir = 62
Modified the script to check for <hostname>.vpn.fedoraproject.org in case <hostname>.fedoraproject.org is not a match.[[BR]] Here is the output of the script:[[BR]][[BR]] app01.dev.fedoraproject.org ... Missing[[BR]] app6.fedoraproject.org ... Missing[[BR]] backup01.phx2.fedoraproject.org ... Missing[[BR]] backup03.phx2.fedoraproject.org ... Missing[[BR]] collab1.fedoraproject.org ... Missing[[BR]] collab2.fedoraproject.org ... Missing[[BR]] fakefas01.fedoraproject.org ... Missing[[BR]] hosted1.fedoraproject.org ... Missing[[BR]] hosted2.fedoraproject.org ... Missing[[BR]] insight01.dev.fedoraproject.org ... Missing[[BR]] insight02.dev.fedoraproject.org ... Missing[[BR]] kojipkgs01.phx2.fedoraproject.org ... Missing[[BR]] lockbox01.phx2.fedoraproject.org ... Missing[[BR]] log02.phx2.fedoraproject.org ... Missing[[BR]] ns1.fedoraproject.org ... Missing[[BR]] paste01.dev.fedoraproject.org ... Missing[[BR]] pkgs01.phx2.fedoraproject.org ... Missing[[BR]] proxy3.fedoraproject.org ... Missing[[BR]] proxy5.fedoraproject.org ... Missing[[BR]] proxy6.fedoraproject.org ... Missing[[BR]] publictest01.fedoraproject.org ... Missing[[BR]] publictest02.fedoraproject.org ... Missing[[BR]] publictest04.fedoraproject.org ... Missing[[BR]] retrace01.fedoraproject.org ... Missing[[BR]] secondary01.phx2.fedoraproject.org ... Missing[[BR]] serverbeach1.fedoraproject.org ... Missing[[BR]] serverbeach2.fedoraproject.org ... Missing[[BR]] serverbeach3.fedoraproject.org ... Missing[[BR]] serverbeach4.fedoraproject.org ... Missing[[BR]] serverbeach5.fedoraproject.org ... Missing[[BR]] sign-bridge01.phx2.fedoraproject.org ... Missing[[BR]] sign-vault01.phx2.fedoraproject.org ... Missing[[BR]] telia1.fedoraproject.org ... Missing[[BR]] torrent01.fedoraproject.org ... Missing[[BR]] torrent02.fedoraproject.org ... Missing[[BR]] tummy1.fedoraproject.org ... Missing[[BR]] x86-11.phx2.fedoraproject.org ... Missing[[BR]] ------------------------------------------[[BR]] Total hosts found by puppet = 158[[BR]] Total hosts using VPN for logging = 25[[BR]] Total hosts with missing parent dir = 37[[BR]]
Upon manual inspection of the 37 machines reported missing by the script, I found that 20 of these missing have a dir with slight deviation from the name:[[BR]][[BR]]
app6.fedoraproject.org => app06.vpn.fedoraproject.org[[BR]] collab1.fedoraproject.org => collab01.vpn.fedoraproject.org[[BR]] collab2.fedoraproject.org => collab02.vpn.fedoraproject.org[[BR]] hosted1.fedoraproject.org => hosted01.vpn.fedoraproject.org[[BR]] hosted2.fedoraproject.org => hosted02.vpn.fedoraproject.org[[BR]] log02.phx2.fedoraproject.org => log02[[BR]] ns1.fedoraproject.org => ns01.vpn.fedoraproject.org[[BR]] proxy3.fedoraproject.org => proxy03.vpn.fedoraproject.org[[BR]] proxy5.fedoraproject.org => proxy05.vpn.fedoraproject.org[[BR]] proxy6.fedoraproject.org => proxy06.vpn.fedoraproject.org[[BR]] retrace01.fedoraproject.org => retrace01.qa.fedoraproject.org[[BR]] secondary01.phx2.fedoraproject.org => secondary01[[BR]] serverbeach1.fedoraproject.org => serverbeach01.vpn.fedoraproject.org[[BR]] serverbeach2.fedoraproject.org => serverbeach02.vpn.fedoraproject.org[[BR]] serverbeach3.fedoraproject.org => serverbeach03.vpn.fedoraproject.org[[BR]] serverbeach4.fedoraproject.org => serverbeach04.vpn.fedoraproject.org[[BR]] serverbeach5.fedoraproject.org => serverbeach05.vpn.fedoraproject.org[[BR]] telia1.fedoraproject.org => telia01.vpn.fedoraproject.org[[BR]] tummy1.fedoraproject.org => tummy01.vpn.fedoraproject.org[[BR]] x86-11.phx2.fedoraproject.org => x86-11.stg.phx2.fedoraproject.org[[BR]]
Also The following machines looked totally missing:[[BR]]
app01.dev.fedoraproject.org[[BR]] backup01.phx2.fedoraproject.org[[BR]] backup03.phx2.fedoraproject.org[[BR]] fakefas01.fedoraproject.org[[BR]] insight01.dev.fedoraproject.org[[BR]] insight02.dev.fedoraproject.org[[BR]] kojipkgs01.phx2.fedoraproject.org[[BR]] lockbox01.phx2.fedoraproject.org[[BR]] paste01.dev.fedoraproject.org[[BR]] pkgs01.phx2.fedoraproject.org[[BR]] publictest01.fedoraproject.org[[BR]] publictest02.fedoraproject.org[[BR]] publictest04.fedoraproject.org[[BR]] sign-bridge01.phx2.fedoraproject.org[[BR]] sign-vault01.phx2.fedoraproject.org[[BR]] torrent01.fedoraproject.org[[BR]] torrent02.fedoraproject.org[[BR]]
Great. ;)
A few misc notes:
lockbox01 logs as infrastructure.fedoraproject.org pkgs01 logs as pkgs.fedoraproject.org
Looks like the 20 above we need to rename/sync names up with.
These are the hosts currently missing from log02:
backup03[[BR]] darkserver01[[BR]] fakefas01[[BR]] hosted04[[BR]] insight02[[BR]] lockbox01 (is present as infrastructure.fedoraproject.org)[[BR]] publictest01[[BR]] publictest04[[BR]] sign-bridge02
Great! Thanks.
I have filed ticket #3193 to fix these. ;)
We can close this ticket now. Thanks for working on it!
Login to comment on this ticket.