Alerts stop working after a day

[L0rdV0ld3m0rt]

Version

2.4.150

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

6

RAM

32

Storage for /

500GB

Storage for /nsm

500GB

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

I have a new Security Onion install on a physical server. After initial setup everything works great for about a day and then it randomly stops generating alerts in the console. I initially thought it was because the server was under-spec'd, so I added more RAM and now it seems to still be sending alerts to our SIEM, but they're not showing in our console so I can't tune them. I'm not finding any useful errors, and when I have, the forums and KBs seem to be silent about how to remedy them, or offer a high level suggestion that I have no idea how to implement. Everything on the grid says it's healthy and running, but there is high capture loss in the application. Looking at the Bond interface there is a large amount of traffic but not seeing many drops comparatively. Elastic agent is healthy, have tried adjusting lsheap, have looked at hot/cold settings, soup and so-test have been run and are happy. Restarting the server does not resolve the problem, and the only fix I've found is a full reinstall of SO, and again it only works for a day. This was a straightforward out of the box standalone install, only change was moving the nsm partition to a secondary drive but have seen no issues there.

I'm sure there's got to be some some simple config fix for this, whether with queuing or with packet capture. I'm not a Security Onion expert, just trying to set up a simple server to alert on network traffic. Been spending months on this and at my wits end. Any help would be very greatly appreciated.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[L0rdV0ld3m0rt]

I've also tried increasing the settings mentioned here, though there's no direction on what they mean by "adjust". I can't tune the NIDS ruleset because I'm not getting alerts. Went into InfluxDB console and there's a "low pcap retention" alert, but can't find any info on how to correct this.

[L0rdV0ld3m0rt]

Tried removing a couple scanning servers with a BPF, but it broke suricata. I couldn't find any better documentation anywhere to validate those are configured properly. tried adjusting max pending packets and af-packet ring size, though the documentation doesn't specify a proper value for this. ended up having to reinstall SO yet again. Worked for about an hour before it stopped generating alerts again. Some of the documentation suggested issues with Tap/Span port upstream, but is only ingesting 1Gbps on a 10Gbps port, so curious what the suggestions would be to remedy this?

[cm-ops]

A couple of things to check. The screenshot above shows the PCAP BPF, what is in Suricata BPF? Do you see a current eve log in /nsm/suricata? Is there a current Suricata index in Elasticsearch?

Capture loss without packet loss is usually attributed to something upstream, TAP/SPAN traffic. https://docs.securityonion.net/en/2.4/zeek.html#packet-loss-and-capture-loss

[L0rdV0ld3m0rt]

Good morning, thanks for the reply. When I looked at the bond0 span interface I saw almost no dropped packets so assumed issue couldn't be upstream. I noticed though that the 10Gb nic speed was set at 1000 even though both ends were 10Gbps ports. Went into the switch config and found the SFP that was plugged into was only 1Gb :(. Just ordered a new SFP and hoping this will remedy the capture loss. Is the screenshot above the proper way to do the BPF filters? I couldn't find much information on web to validate (SO documentation didn't actually show an example), but what I had above ended up faulting the associated modules.

[L0rdV0ld3m0rt]

[L0rdV0ld3m0rt]

Yeah, there is. config is (ip redacted):

`filter {
if [event][module] == "suricata" {
prune {
whitelist_names => [ "message" ]
add_tag => [ "Msg_filter" ]
}
}
}
output {

if "Msg_filter" in [tags] {
tcp {
codec => line { format => "%{message}" }
id => "SO_LOG"
host => "10.x.x.x"
port => xxxxx
}
}
} `

[L0rdV0ld3m0rt]

I also tried adding the config to the search pipeline. The forwarding then works, but alerts cease to generate in the console - basically the same as when I add the config to the manager pipeline. when I add it to custom, local alerts work but it doesn't forward to our SIEM.

[cm-ops]

Try using pipeline-to-pipeline if you are modifying. This is using custom0 in SOC.

In /opt/so/saltstack/local/salt/logstash/pipelines/config/custom/ create an output Logstash plugin, for example 9901_output.conf with a config like:

output {
  pipeline {
    send_to => ["custom0"]
  }
}

Add an input to your config:

input {
  pipeline {
    address => custom0
  }
}

In SOC, in logstash > defined_pipelines add your custom config to custom0 and your output config to manager. Then add the custom pipeline to your role in logstash > assigned_pipelines.

Finally, restart Logstash.

[L0rdV0ld3m0rt]

Thanks Chris, I tried this and now neither console or SIEM is getting alerts. Been trying to read up on the logstash documentation, but is not super helpful since Sec Onion is built differently. Even looking at the yaml file, the paths listed don't even exist on the machine. I'm not familiar with setting up these pipelines, so I'm sure I did something wrong here. Screenshots above

[L0rdV0ld3m0rt]

I saw the article below and the suggestion was to "You will need to add a custom pipeline and clone the IDS events to send to both SOC and your external system" - which sounds like it would work perfectly, but the links are expired and I am not sure how to use this tool and then how to reingest it. The config below is my attempt at the Manager config, with the original config above in Custom0, but it doesn't seem to be working...I'm a complete noob at this so struggling to understand how the pipelines work together and where each config item is supposed to go.

`filter {
clone {
clones => ["custom0"]
add_tag=> ["cloned"]
}
}

output {
if "cloned" in [tags] {
pipeline {
send_to => ["custom0"]
}
}
else {
elasticsearch {
hosts => ["localhost:9200"]
index => "syslog-%{+YYYY.MM.dd}"
}
}
}`

#2066
https://www.elastic.co/docs/reference/logstash/plugins/plugins-filters-clone