Unable to connect Reciever to manger Search node Error Fully-formed clusters do not attempt to discover other nodes. Is this possible?

[Bal33p]

Version

2.4.30

Installation Method

Network installation on Red Hat derivative like Oracle, Rocky, Alma, etc.

Description

configuration

Installation Type

Distributed

Location

cloud

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

64

Storage for /

250

Storage for /nsm

30tb

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Can I use a Receiver with just a manager search node or do I also need a standalone search node as well?

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[Bal33p]

none of the integrations are starting up on the Receiver node
Ror example, logstash has the following error

      ID: so-logstash
Function: docker_container.running
  Result: False
 Comment: Failed to pull orseconip01:5000/security-onion-solutions/so-logstash:2.4.30: Error 404: manifest for orseconip01:5000/security-onion-solutions/so-logstash:2.4.30 not found: manifest unknown: manifest unknown
 Started: 14:28:34.400876
Duration: 75.52 ms
 Changes:  

Manager Search node has the following error in the logs also
securityonion.log:[2023-12-11T13:31:22,887][WARN ][org.elasticsearch.cluster.coordination.Coordinator] This node is a fully-formed single-node cluster with cluster UUID [XXXXXXXXXXX], but it is configured as if to discover other nodes and form a multi-node cluster via the [discovery.seed_hosts=[orseconip01, orseconrp01]] setting. Fully-formed clusters do not attempt to discover other nodes, and nodes with different cluster UUIDs cannot belong to the same cluster. The cluster UUID persists across restarts and can only be changed by deleting the contents of the node's data path(s). Remove the discovery configuration to suppress this message.

[argwfm]

Based on doc's, doesn't look like it... receivers are intended for distributed architectures with standalone search nodes.
https://docs.securityonion.net/en/2.4/architecture.html?highlight=architecture#receiver-node

[Bal33p]

Yeah, I saw that as well, which is why I am hoping to get a confirmation on that from the SO guys. I don't understand why I would use a distributed deployment with the manager search if I can't use these types of nodes to bolt-on as I build out the deployment over time
Everything shows Healthy in my grid and it has configured the agent update. The high state on the Manager search node comes back clean. it is just the Receiver that won't start the logstash and redis services all 4 services fail due to lack of manifest.
The install for the Receiver node took several different attempts but it was finally installed.

Any info from the SO guys or a clear explanation from anyone would be greatly appreciated

[Bal33p]

Any chance i could get a yes no on this topic from @defensivedepth or one of the other SO guys?

[Bal33p]

According to Elasic.co i shoudl be able ot. use a receiver node with a manger search node. so if this is true anyone have any insite on the above error?

Manager Search node has the following error in the logs also
securityonion.log:[2023-12-11T13:31:22,887][WARN ][org.elasticsearch.cluster.coordination.Coordinator] This node is a fully-formed single-node cluster with cluster UUID [XXXXXXXXXXX], but it is configured as if to discover other nodes and form a multi-node cluster via the [discovery.seed_hosts=[orseconip01, orseconrp01]] setting. Fully-formed clusters do not attempt to discover other nodes, and nodes with different cluster UUIDs cannot belong to the same cluster. The cluster UUID persists across restarts and can only be changed by deleting the contents of the node's data path(s). Remove the discovery configuration to suppress this message.

Yes, you can use an Elasticsearch manager search node with a receiver node. In Elasticsearch, every node in the cluster can handle HTTP and transport traffic by default 1. A collection of connected nodes is called a cluster. If you are running a single node of Elasticsearch, then you have a cluster of one node 1. Requests like search requests or bulk-indexing requests may involve data held on different data nodes. A search request, for example, is executed in two phases which are coordinated by the node which receives the client request — the coordinating node 1. The receiver node runs Logstash and Redis and allows for events to continue to be processed by search nodes in the event the manager node is offline 2. When a receiver node joins the grid, Elastic Agent on all nodes adds this new address as a load balanced Logstash output. The search nodes add this new node as another Logstash input 2.

[defensivedepth]

@Bal33p Per #1720 :

Tagging Individuals
Please do not tag an individual in a discussion unless that individual has already volunteered to help you in that discussion.

Feel free to bump, just not tag.

As to the question:

Yes, Receiver nodes should work with Manager + Search nodes. We are still working out some edge cases related to Receiver nodes on 2.4.

[Bal33p]

I apologize; I will not call out again.
I have added a receiver node to my Distributed ManagerSearch Node. However, Logstash, Redis, and so on will not start.
With the following error
ID: so-logstash
Function: docker_container.running
Result: False
Comment: Failed to pull orseconip01:5000/security-onion-solutions/so-logstash:2.4.30: Error 404: manifest for orseconip01:5000/security-onion-solutions/so-logstash:2.4.30 not found: manifest unknown: manifest unknown
Started: 14:28:34.400876
Duration: 75.52 ms
Changes:

Any idea what I can do to resolve this issue?
We have a remote site we need to add to our environment. We need to bolt on a receiver search and forward node to this manager search node.

Any help would be greatly appreciated.

[defensivedepth]

I see that your other Discussion is being answered as well. Feel free to tag me on this one if you still need some followup