How do receiver nodes work? The answer

[TOoSmOotH]

There seems to have been some confusion about the role of receiver nodes and how they work. I am hoping to clear this up by explaining the inner workings of them so you can properly utilize them in your infrastructure.

Receiver nodes were designed with 2 purposes in mind:

• Reduce the load on the manager
• Offer pipeline redundancy

Receiver nodes still need to be close to the search nodes because when you add a new receiver node to the grid, the search nodes add the redis service as an input in their configs automatically. What that means is if you drop a receiver at a remote site ALL of your search nodes will be trying to access that redis queue remotely. You do not save any bandwidth by placing a receiver node at a remote site.

How do receiver nodes work with Elastic Agent?

There are 2 components that encompass managing Elastic Agents. The first being fleet which handles things like updating the agents and scheduling searches etc. The other is where the log output goes to which in out case is logstash running on the manager or receiver node. Due to limitations in Elastic licensing we can only have a single output policy. What this means is when you add a receiver or a fleet node it gets added to a list that is distributed to the agents. The agents go down that list and stop after a successful connection. The only way to direct agents to specific receivers is to use firewall rules to block agents to certain receivers. Again keep in mind that there is no bandwidth savings here because the search nodes still need to empty the redis queue on the receivers.

Why are search nodes pulling from the redis queue on receiver nodes?

The best way to articulate this is by giving an example. If you don't have any receivers and the manager goes down, the search nodes do not index anything because they cannot connect to redis. The agents can not connect to logstash so the pipeline starts backing up on the agents. In this same scenario with a receiver node the agents would not be able to talk to logstash on the manager and then would try to connect to the receiver node. Once connected they would send their logs to the receiver like nothing was wrong. The search nodes connect to both the manager and receiver nodes redis queue to pull events. If the manager goes down it complains but keeps pulling the log events from the queue on the receiver. Doing it this way allows for scaling of the pipeline. More receivers + more search nodes = more EPS ingestion of events.

[TOoSmOotH]

I did forget to add that if you don't set dmz_nodes for your remote receivers your sensors will send to all available receivers as well. This will create an issue as well.

[argwfm]

@TOoSmOotH, thank you! Encountered some other issues w/ receivers under v2.4.30; notably relating to resolvability #11909 (comment) and load balancing. Was able to get forward nodes to actively load balance between the manager and receivers (default only sends to one) by adding loadbalance: true to the Fleet grid-logstash output advanced YAML configuration, which is persisting across salt cycles.

https://www.elastic.co/guide/en/fleet/current/logstash-output.html#output-logstash-performance-tuning-settings

[Bal33p]

@TOoSmOotH, Thanks for taking the time to explain this. It is helpful.

[dougburks]

I've worked this into the documentation:
https://docs.securityonion.net/en/2.4/architecture.html#receiver-node