#10120 Investigate forwarding centralised logs to splunk
Closed: Fixed 9 months ago by aheath1992. Opened 2 years ago by mobrien.

Describe what you would like us to do:


We may need to send the centralised Fedora logs to an instance of Splunk in the future for security auditing purposes and need some investigation into what is needed to make this work.

cc: @lgriffin


Thanks for logging this Mark, @mattdm pinging as an FYI here

Thanks! Doing this will help Red Hat product security help us in the event of ... some sort of event.

Metadata Update from @humaton:
- Issue tagged with: high-gain, medium-trouble

2 years ago

Metadata Update from @mohanboddu:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: ops

2 years ago

Metadata Update from @mohanboddu:
- Issue untagged with: medium-trouble
- Issue tagged with: low-trouble

2 years ago

So, I never quite heard back from internal folks on this, but I might have missed that they asked me to file a internal ticket on it.

I have now done so, so we will see what comes of it.

@kevin is there any documentation about this process ? (sending Fedora logs to splunk )

There's no externally reachable / public docs no. ;(

I'll try and see where this is soon.

ok i see

Le jeu. 24 mars 2022 =C3=A0 20:40, Kevin Fenzi pagure@pagure.io a =C3=A9c=
rit :

kevin added a new comment to an issue you are following:
``
There's no externally reachable / public docs no. ;(

I'll try and see where this is soon.
``

To reply, visit the link below or just reply to this email
https://pagure.io/fedora-infrastructure/issue/10120

@mobrien @kevin is this ticket still needed if so what is needed?

Metadata Update from @aheath1992:
- Issue assigned to aheath1992

2 years ago

@aheath1992 got the dns alias from splunk admins, splunk-syslog.corp.redhat.com:514
we should be able to telnet the target after accepting ACL Network Rule from REDHAT IT guys .

We are on the good way :)

@kevin @seddik

After speaking with the Splunk admins we are able to send traffic over to Red Hat Splunk via TLS, this will require a couple of things for us to set his up.

  1. We need to install the rsyslog-gnutls package. The rsyslog-gnutls package contains the rsyslog plugins that provide the ability to receive syslog messages via upcoming syslog-transport-tls IETF standard protocol.

  2. We need to add the cert-RH-IT-pki-ca-chain to log01 keychain so that log01 will trust splunk-syslog.corp.redhat.com with the TLS connection.

  3. We need to update the rsyslog configs to reflect the change to TLS layer forwarding of logs.

@aheath1992
I'm working on PR, and need some more infos.
where can i fetch cert-RH-IT-pki-ca-chain ?

Thanks

@kevin @aheath1992
workaround here => https://pagure.io/fedora-infra/ansible/pull-request/1265
i know it's not completed ... but we can start from this version and update config if needed.

[backlog refinement]
PR is still in our scope, but let's wait after F38 freeze end.

So, I made a tweak to make it happy (we needed to install the cert and use that).

However, it doesn't seem to be sending to splunk. I don't see any packets via tcpdump or in ss. ;(

I also tried to use logger to log some test message. The udp one appears to go but doesn't show up and the tcp one just hangs. ;(

So, this needs some more investigating.

Got connection time out !!
Network issue ?? as you said @kevin , this needs more investigation...

Also, it might be easy if we can troubleshoot with Red Hat's IT Gyus :)

Loop @aheath1992

[backlog refinement]
Still needs investigation, we just haven't got back to it

Have enabled log forwarding to Red Hat Splunk under the index="fedoraproject". At this time the only access is via Red Hat rover groups, so users Red Hat Internal and need to be part of the fedora-infrastructure-splunk at this time. We can sync with the Red Hat if we need to open this to other community members.

Metadata Update from @aheath1992:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

9 months ago

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog
Attachments 1