Event Forwarding syntax

[L0rdV0ld3m0rt]

Version

2.4.80

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

12

RAM

16

Storage for /

500GB

Storage for /nsm

500GB

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

I'm trying to use Security Onion as a log ingestion point to forward on network traffic/alerts to our SIEM, but I can't find information on the syntax for the config files anywhere except this single example below. Is there any documentation for this specifically? I want to start by just forwarding everything, and then work on tuning it down if I need to. The syslog plugin below seems like could be an option, but I feel like shouldn't be necessary, just need to know the proper syntax for that config file. Any help would be really appreciated.

https://docs.securityonion.net/en/latest/logstash.html
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-syslog.html

output {
if [event][module] == "zeek" and [pipeline] == "dns" {
udp {
id => "cloned_events_out"
host => "192.168.x.x"
port => 1001
codec => "json_lines"
}
}
}

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

If you are going to forward all of Zeek, you would just use if [event][module] == zeek so something like below.

output {
  if [event][module] == "zeek" {
  syslog {
    id => "my_plugin_id"
    host => "192.168.x.x"
    port => 1001
    }
  }
}

[cm-ops]

If you are planning on editing the data sent, I would recommend setting up a custom pipeline, for example, custom0 and editing from there. You would create a custom output to send to that pipeline like below.

https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html

output {
  pipeline {
    send_to => ["custom0"]
  }
}

Then create an input with a filter and output like below (example sends to a file using file output plugin). The filter plugin is using prune and whitelist to send only those fields.

input {
  pipeline {
    address => custom0
  }
}

filter {
  if "suricata" in [event][module] {
    prune {
      whitelist_names => [ "message", "tags" ]
      id => "suri_prune"
      add_tag => ["prune"]
    }
  }
}

output {
  if "prune" in [tags] {
    file {
      path => "/var/log/logstash/test"
    }
  }
}

[L0rdV0ld3m0rt]

Good morning, thank you for the thorough reply. Apologies as I'm brand new to this so I don't really have a framework for the data flow. I spent last couple days reading through documentation but am still confused. What file do I add the output to pipeline script to? Is the second script still in the .conf file at "/opt/so/saltstack/local/salt/logstash/pipelines/config/custom"?

The data I'm trying to remove is between the red brackets in the screenshot below. None of this is in the original Suricata logs. Are these part of the syslog header output, or are those generated from logstash?

[cm-ops]

In /opt/so/saltstack/local/salt/logstash/pipelines/config/custom/ create file named *.conf, for example, 9901_output.conf. Put that output in that file.

The second config is a prune logstash plugin https://www.elastic.co/guide/en/logstash/current/plugins-filters-prune.html it uses whitelist_names to only send the message and tags fileds, nothing else.

The output in the second config needs to be changed to whatever output you are sending - http, udp, tcp, syslog, etc. https://www.elastic.co/guide/en/logstash/current/output-plugins.html

[L0rdV0ld3m0rt]

Thanks, I've been using the config file in the custom directory which has been working. I have my conf file specified in the logstash > defined pipelines > search. I tried doing a config file in custom with only the output to pipeline you have listed above, but when I tried to add it to the "custom0" pipeline in the screenshots below. It didn't error but it also didn't send anything.

Where does the second half of the code go? In the same config file, the logstash.yml? I read through all the KBs below and couldn't figure out where to add the code for the prune logstash plugin. It gave examples but never mentions where to apply it

https://www.elastic.co/guide/en/logstash/current/plugins-filters-prune.html
https://www.elastic.co/guide/en/logstash/current/output-plugins.html
https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html
https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html#plugins-filters-elasticsearch-hosts
https://docs.securityonion.net/en/2.4/elasticsearch.html
https://docs.securityonion.net/en/2.4/logstash.html

[cm-ops]

The first output to custom0 pipeline would go in either you manager or search pipeline as custom/RA_2_pipe.conf. I usually put it in the manager pipeline and follow the SO naming, for example 1234_RA_2_pipe.conf. Then number it and place it before the output to Redis.

The longer config goes in custom0 pipeline as custom/RAforwarder_pruned.conf.

[L0rdV0ld3m0rt]

I never could manage to get the config to work using multiple pipelines, so I was able to get really close using the configs in the attached images. Unfortunately, the first output added what looks like improperly formatted syslog headers, so our SIEM still would not parse them correctly. The only thing I want to send is the Suricate "EVE" formatted message, starting at the "timestamp" field. I also tried filtering the output by the tag, and everything was correct, except for some reason it then appended the "tags" field at the end, again not allowing it to be parsed correctly.

I also Thought I could try something like this, but not sure if you can filter from within the syslog/output
remove_field => [ "%{host}", "LOGSTASH[-]:" ] …. And read the below about “whitelisting”, so maybe need to blacklist the fields we don’t want so the tag doesn’t persist…

https:/www.elastic.co/guide/en/logstash/current/plugins-filters-prune.html

[L0rdV0ld3m0rt]

input { file { path => "/nsm/suricata/eve*.json" start_position => "end" sincedb_path => "/dev/null" } } output { tcp { host => "10.*.*.*" id => "RA_SO_LOG" port => 30051 codec => line { format => "%{message}"} } }

I also tested the attached config and grabbed a capture of the actual traffic being sent from the server. As you can see some of the logs are being sent correctly (beginning with "timestamp" yay!), however now we're getting these random logs that begin with "ts". Some of them appear to be local Security Onion or Logstash app logs, and some look like they may be events with no alerts. The bizarre thing is that they are definitely NOT in the /nsm/suricata directory so I'm super confused where they are coming from. Any suggestions on how to get rid of them? Thanks again for the help

[cm-ops]

Are you using Suricata as metadata too?

[L0rdV0ld3m0rt]

Sorry, honestly I don't know. Unfortunately most of the documentation seems to presuppose a detailed understanding of how to filter and parse all the data so most of it's over my head and I can't seem to find the supporting information I need (and I've spent literally days reading and trying to track it down). I do know that in my /nsm/suricata directory the ONLY logs match the EVE format like such, "{"timestamp":"2024-09-13T18:42:56.767923+0000","flow_id":528852817273834,"in_iface":"bond0","event_type":"alert","vlan":[975],"src_ip": etc...." so I don't understand where the extra logs are coming from or why they're being included.

My SIEM has a parser for Suricata/EVE, so at this point, I'm not trying to get into any kind of advanced filtering, so not sure why it's been so difficult, I just want to forward on ONLY the alerts exactly as they are in the nsm directory with nothing else added.

https://docs.suricata.io/en/latest/output/eve/eve-json-format.html

[L0rdV0ld3m0rt]

I think I got it! adding the format message in the output looks like it removed those extra headers! Thanks for the help

filter { if [event][module] == "suricata" { prune { whitelist_names => [ "message" ] add_tag => [ "Msg_filter" ] } } } output { if "Msg_filter" in [tags] { tcp { codec => line { format => "%{message}" } id => "SO_LOG" host => "10.x.x.x" port => 30051 } } }