Heavy nodes not getting data from elastic agents

[Sahale]

Version

2.4.20

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

32

Storage for /

200GB

Storage for /nsm

200GB

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Hey!

Architecture:

• Manager;
• Fleet;
• 2x Heavy nodes.

Problem: agents not sending logs to heavy nodes.
What did I do:

1. After installation I allowed access from test PC to Fleet node (5055/tcp, 8220/tcp, 8443/tcp) and Heavy node (5055/tcp);
2. Downloaded elastic agent and installed it on test PC;
3. In Fleet web-interface I added both heavy nodes to hosts in "grid-logstash" output and applied settings;
4. Checked Kibana - no logs from test PC;
5. Downloaded logs from PC via Fleet agents panel - agent can not access manager (it's ok) and can not access both heavy nodes with SSL handshake error (it's not ok):

{"@timestamp":"2023-11-10T13:40:43.6332585Z","agent":{"id":"fbe9cc8d-a744-44f8-91b8-45e7c4e6a10b","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":863,"name":"LogstashClient.cpp"}}},"message":"LogstashClient.cpp:863 SSL handshake with Logstash server at heavynode-2:5055 encountered an error: connect error","process":{"pid":8816,"thread":{"id":11280}}}

6. Checked SSL settings for logstash input - no certificate is used by default. Tried to find out how to set up one without success. At least, for 2.4;
7. After next highstate both heavy nodes were deleted from logstash. Looks like it managed by /opt/so/saltstack/default/salt/elasticfleet/tools/sbin_jinja/so-elastic-fleet-outputs-update.

Finally, questions:

1. Are heavy nodes supported now?
2. Is there any instruction to add SSL certificate to logstash 5055/tcp? Or maybe it should be automated somehow? It also should be added in Fleet web-interface for agents to accept them as I understand.
3. How to configure SO not to delete heavy nodes from "grid-logstash" output?

Thanks a lot!

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[TOoSmOotH]

Heavy nodes are supported however you cannot have local elastic agents send data to it as the node is not part of the cluster. Elastic Agent requires all agents to send data to the same cluster. There are 2 agents running on the heavy node. The one that pulls all the logs from zeek etc is running in standalone mode. You could configure standalone agents on site to report to the heavy node but this would require you to manage updates etc manually.

[Sahale]

Ok, thanks!
Administration and support of such standalone agents on sites with a lot of PC's will be really difficult.
Will such architecture be covered in future but with centralized control? Or maybe some workaround?

[TOoSmOotH]

You can put a receiver node on site for the local agents to send to but the agent data will go to the main cluster.

[Sahale]

Main reason of using heavy nodes it to save logs locally on every site without sending them outbound.
Looks like it's more simple to use very powerful standalone nodes in every office in our case.