Log ingestion rate significantly decreased after 2.4.80 update

[tsmith-spscc]

Version

2.4.80

Installation Method

Security Onion ISO image

Description

installation

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

24

RAM

192

Storage for /

1 TB

Storage for /nsm

40 TB

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

After updating from version 2.4.70 to 2.4.80, the rate of log ingestion decreased significantly. I went from ingesting roughly 800k logs every 30 minutes to around 25k logs every 30 minutes. From what I can tell, all indices are still still receiving logs, just far fewer than normal. There has been no change outside of Security Onion that would cause this decrease and the decrease was immediate after the update finished.

I found the following errors in the elastalert.log file, but I do not know if they are relevant to this issue.

2024-07-11 19:36:50,038 ERROR elastalert Error running query: RequestError(400, 'parsing_exception', 'line 1:36: no viable alternative at input \'(""=\'') 2024-07-11 19:36:50,061 ERROR elastalert Error running query: RequestError(400, 'parsing_exception', 'line 1:253: token recognition error at: \'"\\\\\\\\\\*\'') 2024-07-11 19:36:50,620 ERROR elastalert Error running query: RequestError(400, 'parsing_exception', "line 1:13: mismatched input 'exited' expecting {<EOF>, 'and', 'in', 'in~', 'like', 'like~', 'not', 'or', 'regex', 'regex~', ':', '+', '-', '*', '/', '%', '|'}") 2024-07-11 19:36:50,990 ERROR elastalert Error running query: RequestError(400, 'verification_exception', 'Found 2 problems\nline 1:11: Unknown column [cs]\nline 1:14: Cannot use field [host] type [object] only its subfields') 2024-07-11 19:36:52,081 ERROR elastalert Error running query: RequestError(400, 'verification_exception', 'Found 2 problems\nline 1:12: argument of [destination.port like~ (80, 443)] must be [string], found value [destination.port] type [long]\nline 1:12: argument of [destination.port like~ (80, 443)] must be [string], found value [destination.port] type [long]') 2024-07-11 19:36:52,363 ERROR elastalert Error running query: RequestError(400, 'verification_exception', 'Found 1 problem\nline 1:11: Unknown column [properties.message]') 2024-07-11 19:36:52,829 ERROR elastalert Error running query: RequestError(400, 'parsing_exception', 'line 1:203: no viable alternative at input \'(""POST\'') 2024-07-11 19:36:53,854 ERROR elastalert Error running query: RequestError(400, 'parsing_exception', 'line 1:28: token recognition error at: \'"\\\\Windows\\\\__1\\d\'') 2024-07-11 19:36:54,176 ERROR elastalert Error running query: RequestError(400, 'verification_exception', 'Found 2 problems\nline 1:11: Unknown column [Status]\nline 1:33: Unknown column [userAgent]') 2024-07-11 19:36:54,305 ERROR elastalert Error running query: RequestError(400, 'parsing_exception', 'line 1:28: token recognition error at: \'"\\\\mojo\\.\'')

It seems like I spend more time trying to fix Security Onion than I do using it for it's intended purpose. It's getting really frustrating. Any assistance is appreciated. Thanks.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[defensivedepth]

How many Elastic Agents do you have deployed? Also, I would suggest comparing the before and after to confirm what types of logs are no longer being ingested.

[tsmith-spscc]

All of my nodes were requesting a reboot after the 2.4.80 update. So, last Thursday I rebooted them. Once again, after the reboot Elasticsearch went into a pending state and Kibana will not load (404 error). This is the exact same thing that happened after the 2.4.70 update. See my previous post on this issue: #13128

So now I'm stuck trying to troubleshoot that issue before I can get back to trying to figure out why SO decided to significantly reduce its log ingestion. I wager I'll probably just have to blow this instance of SO away and start over. Again. Very frustrating.

[TOoSmOotH]

Looking at the issue you linked I can see where we responded asking for logs and you did not provide any. So my guess is there is an underlying issue that has nothing to do with upgrades and everything to do with your current setup. The most probable issue based on this is a failed shard in your cluster that is causing this to happen but I can't be definitive because you have not provided any information. I get that you are frustrated but I don't see what we can do to help until you provide more information.

[tsmith-spscc]

I'm just going to re-install once again. I'm tired of hunting down errors and looking through logs that don't provide useful information. I don't have enough time in the day to constantly try to make sure SO stays running error free along with everything else I've got to do.

[TOoSmOotH]

The logs provide plenty of useful information otherwise we would not ask for them. 95% of the time the issue is something with Elasticsearch which we don't write or maintain. We try to limit the interaction between the user and the complexities of Elasticsearch but sometimes you have to get in there and fix things. If you think a full reinstall is a better use for your precious time vs looking at a log and cluster status then by all means go for it. We will go ahead and close down this issue.

[tsmith-spscc]

It absolutely is a better use of time. I don't trust that "fixing" the issue will stabilize SO in the long run, especially since I had a nearly identical issue when updating from 2.4.60 to 2.4.70. Re-installing isn't what I want to spend my time on, but it is what it is. I don't mind spending time under the hood fixing things, but when more of my time is used fixing things than actually using the product for it's intended purpose, then yes, it's frustrating. Thanks for understanding.