Missing so-elasticsearch and so-elastalert from manager node

[Fox-E-1337]

Version
2.4.111

Installation Method
Security Onion ISO image

Description
other (please provide detail below)

Installation Type
Distributed

Location
on-prem with Internet access

Hardware Specs
Exceeds minimum requirements

CPU
12

RAM
64

Storage for /
500GB

Storage for /nsm
200GB

Status
Manager node is in fault showing missing so-elasticsearch and so-elastalert

Salt Status
up:
- forward_sensor
- 1_searchnode
- 2_searchnode
- secon_manager

Logs
Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail
After mistakenly pasting wrong CIDR for firewall config for my manager node I have lost so-elasticsearch and so-elastalert on the manager node. When looking at the securityonion.log on the manager node I see the error below

[WARN ][org.elasticsearch.cluster.coordination.Coordinator] failed to validate incoming join request from node [{search1}{qDiOHCLrT46Rs3XMAnT8Ng}{e5nmUmKvSnC8Pxz-5IFXxQ}{search1}{search1}{10.10.10.16:9300}{dhistw}{8
.14.3}{7000099-8505000}{xpack.installed=true, ml.config_version=12.0.0, transform.config_version=10.0.0}]
org.elasticsearch.transport.RemoteTransportException: [search1][172.17.1.22:9300][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: This node previously joined a cluster with UUID [nTec_lDtRPKYdOL4sdBJFQ] and is now trying to join a different cluster with UUID [kUjZ1tG_RnCTibLZK-qX5Q]. This is forbidden and usually indicates an incorrect discovery or cluster bootstrapping configuration. Note that the cluster UUID persists across restarts and can only be changed by deleting the contents of the node's data paths [] which will also remove any data held by this node.
at org.elasticsearch.cluster.coordination.JoinValidationService.lambda$new$2(JoinValidationService.java:140) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:578) ~[?:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$3.onResponse(SecurityServerTransportInterceptor.java:631) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$3.onResponse(SecurityServerTransportInterceptor.java:620) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:682) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:314) ~[?:?]
at org.elasticsearch.xpack.security.transport.ServerTransportFilter.lambda$inbound$1(ServerTransportFilter.java:114) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:201) ~[?:?]
at org.elasticsearch.xpack.security.transport.ServerTransportFilter.authenticate(ServerTransportFilter.java:127) ~[?:?]
at org.elasticsearch.xpack.security.transport.ServerTransportFilter.inbound(ServerTransportFilter.java:105) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:642) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.transport.InboundHandler.doHandleRequest(InboundHandler.java:288) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:301) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.14.3.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1570) ~[?:?]

I can't find where to delete the data described in the bootstrapping a cluster in the elastic documentation. Unsure if this would even restore the missing containers.

Any help would be appreciated

[cm-ops]

This node previously joined a cluster with UUID [nTec_lDtRPKYdOL4sdBJFQ] and is now trying to join a different cluster with UUID [kUjZ1tG_RnCTibLZK-qX5Q].

On your manager and search nodes, what does this command show? sudo so-elasticsearch-query /

[Fox-E-1337]

Manager:
sudo so-elasticsearch-query /
{
"name" : "secon_manager",
"cluster_name" : "securityonion",
"cluster_uuid" : "kUjZ1tG_RnCTibLZK-qX5Q",
"version" : {
"number" : "8.14.3",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "d55f984299e0e88dee72ebd8255f7ff130859ad0",
"build_date" : "2024-07-07T22:04:49.882652950Z",
"build_snapshot" : false,
"lucene_version" : "9.10.0",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}

Search:
{
"name" : "1_searchnode",
"cluster_name" : "securityonion",
"cluster_uuid" : "nTec_lDtRPKYdOL4sdBJFQ",
"version" : {
"number" : "8.14.3",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "d55f984299e0e88dee72ebd8255f7ff130859ad0",
"build_date" : "2024-07-07T22:04:49.882652950Z",
"build_snapshot" : false,
"lucene_version" : "9.10.0",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}

[Fox-E-1337]

I see the two have different UUIDs assume they should be the same? I would think the Grid would show red though.

[Fox-E-1337]

We ended up rebuilding the VMs and starting over. We did not apply a signed cert from our CA in nginx left everything defaulted and it appears to be working.