Receivers and ManagerSearch

[kvkamin]

Version

2.4.30

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

20

RAM

64gb

Storage for /

320GB

Storage for /nsm

8TB

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

This is to document my experience with using Receiver nodes. I wanted this HA feature that allows for downtime on Master/MasterSearch without losing data. It occurred to me that if I offloaded the logstash processing from the ManagerSearch to the receiver nodes, there would be a performance increase on the ManagerSearch node, and indeed it worked. Logstash load on my system over time was about 1/3 of system load.

I after deploying a receiver node, I had to allow elastic_agent_data from anywhere (roaming agents) and redis traffic from ManagerSearch to receiver, redis is a PULL technology. Seems like this could be done during integration time.

All works great.

So here is a small frustration. Under SO menu/Elastic Fleet/Settings/Outputs/grid-logstash receiver nodes are published to elastic agents. And this happens surprisingly fast, change receivers and the fielded agents obey this in mere moments. What I want to do is only publish the receiver nodes here, but by default, the salt engine replaces the list of receivers with the receivers plus the managersearch node. (assuming here that this is true for manager nodes). I want to leave all logstash processing to the receiver nodes and exclude the managersearch node. So where in config menu/salt files can I control the published list of logstash receiver nodes, that overwrite the Elastic Fleet settings?

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[kvkamin]

Would removing the logstash roll from the ManagerSearch effectively do the same thing? If salt is configure to publish all logstash nodes, simply remove it from the Manager. Is this possible?

[kvkamin]

Ahh, (staring at security onion architecture drawing) Logstash is still the front end to storage, so in case of ManagerSearch, logstash still receives the data from receiver redis. The logstash load is remarkably lower than before receiver. Example, system load was ~30 and has dropped to ~8

[kvkamin]

Another salt issue with configuration of Fleet is similar to OUTPUT round robin receivers, is the list of Fleet server hosts on the same page. Salt insists on populating it with:
https://FQDN:8220
https://HOSTNAME:8220
https://FQDN:8220

The issue is that the second entry with just HOSTNAME creates a LOT of client side errors as the client cannot resolve the HOST because there is no search directive in the client's DNS configuration with the search directive of the securityonion system. They do resolve after trying the FQDN.

This does not cause a functional issue, just bad form to fill client logs with unnecessary errors.

[TOoSmOotH]

There are core issues here that cause this "bad form". The first being we can't count on people to use DNS in the first place. We use the hostname for grid members since they have the hosts file entry for the hostname of the manager. In 2.3 we used to get around the DNS issue with hard coding IP addresses but then you can't change IPs without causing tons of issues.

In a perfect world we would use separate input policies for the sensors and grid infrastructure vs agents connecting from external servers/workstations. This is where core issue #2 becomes involved. The basic license by Elastic limits us to a single input policy for all agents.

We have to make things work under those above constraints. We are looking at ways to filter out the error messages as a possible short term solution.

[kvkamin]

Thanks.

[TOoSmOotH]

We have an issue in to disable incoming connections to the manager but keep logstash running when using receiver nodes. #12033

[defensivedepth]

Also, you can disable auto-configuration of the Logstash Outputs & Fleet Host URLs by enabling Advanced options under the SOC Configuration and navigating under Elastic Fleet. Disabling this sets those Elastic Fleet config options to not be automatically updated. Any changes you made in the Elastic Fleet configuration settings for those options would not get overridden.

[kvkamin]

That will do! Thanks.

[kvkamin]

This has been answered, thanks. Just want re-iterate how remarkable the change in system LOAD, and cpu use, when the LOGSTASH workload was moved to the receiver. Both the elastic and logstash on the LOAD on the ManagerSearch dropped from mid 30's down to 8 this morning. Simply by redirecting Elastic Agent outputs to receiver. Could be the characteristics of my hardware, or a concern I had that the receiver logstash is not doing the same functions to the streams. But the data looks good so far. The receiver is running on a Proxmox VM. 4 cores, 16gb. Bumped java from 500 to 4192. I failed to mention earlier that when I integrated Elastic Defend on about 400 workstation, logstash on ManagerSearch crashed twice, so I changed the java 1000mb to 4192. No issues since.