IDH node runaway telegraf and agentbeat/filebeat

[RagnarSTS]

Version

2.4.190

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

4

RAM

1.5G

Storage for /

15

Storage for /nsm

200

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Occasional issues with an IDH node that doesn't seem to complete tasks quick enough. They keep building on top of eachother and causing ridiculous IO read and cpu consumption numbers.
Looking at logs is generally useless as the lag is wild. The problem sometimes doesn't happen for weeks, but then it returns without warning and I frequently don't notice for several days. If it eats a little more power that is acceptable, but my concern is that it isn't actually responding and reporting appropriatelly with canary.
I've run canary many times before with wildly tiny resources. The problem here is heavy handed oversight.
Any suggestions for reducing load applied by the master?

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[RagnarSTS]

Over the last few months I've toyed with this off and on. I think the only reliable fix is a bit more resources applied as the default polling mechanisms just plain overload a system built to the spec on the 2.4 hardware architecture specs.

After everything spins up, adding quite modest fake services http, portscan, mysql, and default ssh - sits at around 3.25G ram utilization.

Vanilla canary is fine with 1G and these services...but SO is big boned.

Recommend updating minimum IDH node spec to 4 cores and 4G ram. Currently it says 2 cores and 1G.

[reyesj2]

Please keep in mind that those specs are absolute bare minimum requirements

I imagine that quite a bit of the resource usage you're seeing on the idh node comes from just the elastic agent running to pickup and forward logs

From your screenshot I see telegraf, that is for metric collection. If you'd like you could disable telegraf from running via SOC -> Administration -> Configuration -> telegraf -> enabled (make sure you have advanced config option toggled) and set to false.