Distributed Environment with so-elastalert Missing

[chantellmocha]

Version

2.4.160

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Distributed

Location

cloud

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

64 GB

Storage for /

270.5 GB

Storage for /nsm

12883.9 GB

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

We have a distributed environment, including 2 search nodes and 7 forward nodes. There is a mix of on-premise and cloud forward nodes; search nodes and our manager node are hosted in AWS. When we added a second search node, I noticed some things were off with it. After a reboot, so-elastalert was missing.

I've looked at all the existing posts linked below, and could not find a fruitful solution.

#13908
#6966
#13817
#12334

The so-elastalert Docker container also cannot be found. Manual reinstallation of the Docker container via docker pull fails. We've already gone through all similar posts and haven't found a solution.

The errors we get on the Elastic side are tied to unallocated shards. How can we rectify this?

Errors when running "sudo salt-call state.highstate":

[ERROR ] Command 'so-elasticsearch-wait' failed with return code: 1 [ERROR ] stdout: Waiting for value 'green open' at 'https://localhost:9200/_cat/indices/.kibana*' (1/300) Server is not ready Server still not ready after 300 attempts; giving up. [ERROR ] retcode: 1 [ERROR ] {'pid': 3895139, 'retcode': 1, 'stdout': "Waiting for value 'green open' at 'https://localhost:9200/_cat/indices/.kibana*' (1/300)\nServer is not ready\nWaiting for value 'green open' at 'https://localhost:9200/_cat/indices/.kibana*' (2/300)\nServer is not ready\nWaiting for value 'green open' at 'https://localhost:9200/_cat/indices/.kibana*' (3/300)\nServer is not ready Server still not ready after 300 attempts; giving up.", 'stderr': ''

[root@angband onion]# so-elasticsearch-query _cat/shards | grep -i una
.ds-ilm-history-7-2025.08.08-000040                                    0 r UNASSIGNED                                            
.ds-.kibana-event-log-ds-2025.08.23-000066                             0 r UNASSIGNED                                            
.ds-.kibana-event-log-ds-2025.07.26-000058                             0 r UNASSIGNED                                            
.ds-.kibana-event-log-ds-2025.08.16-000064                             0 r UNASSIGNED                                            
.security-profile-8                                                    0 r UNASSIGNED                                            
.ds-ilm-history-7-2025.08.22-000044                                    0 r UNASSIGNED                                            
.ds-.kibana-event-log-ds-2025.07.19-000056                             0 r UNASSIGNED                                            
.ds-ilm-history-7-2025.08.01-000038                                    0 r UNASSIGNED                                            
.ds-ilm-history-7-2025.07.25-000036                                    0 r UNASSIGNED                                            
so-detectionhistory                                                    0 r UNASSIGNED                                            
.ds-.kibana-event-log-ds-2025.08.09-000062                             0 r UNASSIGNED                                            
.ds-ilm-history-7-2025.07.18-000034                                    0 r UNASSIGNED                                            
.ds-.kibana-event-log-ds-2025.08.02-000060                             0 r UNASSIGNED                                            
.ds-ilm-history-7-2025.08.15-000042                                    0 r UNASSIGNED    

[root@angband onion]# salt \* cmd.run 'df -h /nsm' 
angband-forward-6_sensor:
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/sdc        3.7T  3.3T  411G  89% /nsm
angband-search-2_searchnode:
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/nvme1n1     12T   96G   12T   1% /nsm
angband_manager:
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/nvme1n1     12T  148G   12T   2% /nsm
angband-forward-7_sensor:
    Filesystem           Size  Used Avail Use% Mounted on
    /dev/mapper/nsm-nsm   22T   20T  2.5T  89% /nsm
angband-forward-1_sensor:
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/sdb        7.3T  6.5T  820G  89% /nsm
angband-forward-5_sensor:
    Filesystem           Size  Used Avail Use% Mounted on
    /dev/mapper/nsm-nsm  2.2T  2.0T  246G  89% /nsm
angband-forward-2_sensor:
    Filesystem              Size  Used Avail Use% Mounted on
    /dev/mapper/system-nsm  177G  157G   20G  89% /nsm
angband-forward-3_sensor:
    Minion did not return. [No response]
    The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:
    
    salt-run jobs.lookup_jid 20250827001057502827
angband-forward-4_sensor:
    Minion did not return. [No response]
    The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:
    
    salt-run jobs.lookup_jid 20250827001057502827
angband-search-1_searchnode:
    Minion did not return. [No response]
    The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:
    
    salt-run jobs.lookup_jid 20250827001057502827
ERROR: Minions returned with non-zero exit code

[root@angband onion]# so-elasticsearch-query _cluster/health?pretty
{
  "cluster_name" : "securityonion",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 203,
  "active_shards" : 343,
  "relocating_shards" : 2,
  "initializing_shards" : 2,
  "unassigned_shards" : 14,
  "unassigned_primary_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 95.54317548746518
}

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[jdonovan-itcnp]

@chantellmocha I seem to be having a very similar issue. Please post here if you find a solution.

Thanks!

[reyesj2]

Are you able to verify that the searchnode can communicate with the manager on the required ports? https://docs.securityonion.net/en/2.4/firewall.html#firewall

For the replias you can set the replias to 0 to get the cluster green and then try running the highstate command on the searchnode
an example of setting the replias to 0 would be so-elasticsearch-query ilm-history-7/_settings -d '{"number_of_replicas":0}' -XPUT

[chantellmocha]

Thank you for the reply! The search nodes both can communicate with the manager on the required ports.

I ran both the provided command and a salt highstate on the manager node, and the cluster is showing yellow health with the below stats.

[root@angband onion]# so-elasticsearch-query _cluster/health?pretty
{
  "cluster_name" : "securityonion",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 202,
  "active_shards" : 334,
  "relocating_shards" : 2,
  "initializing_shards" : 2,
  "unassigned_shards" : 21,
  "unassigned_primary_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 93.55742296918767
}

[reyesj2]

The search nodes both can communicate with the manager on the required ports.

How did you test this? I would typically try running nc -zv <manager ip> <port> from the searchnode to test each ports connectivity. Aside from that can you check the searchnodes minion log file for any errors connecting to the manager? That log file is /opt/so/log/salt/minion

cluster is showing yellow health

Did you run that for each replica or just the 1 example I posted above? Here is a oneliner that should check for unassigned shards and set the replicas to 0

so-elasticsearch-query _cat/shards | grep UNAS | awk '{print $1}' | xargs -I % so-elasticsearch-query %/_settings -d'{"index":{"number_of_replicas":0}}' -XPUT

[chantellmocha]

Hi again - I ran nc and had positive results for all ports with the exception of 5056 (Logstash-to-Logstash for Elastic Agent data ingest), which timed out. I redacted the manager IP but put the command outputs below.

[onion@angband-search-1 ~]$ nc -zv [mgr_ip] 443
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:443.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-1 ~]$ nc -zv [mgr_ip] 5000
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:5000.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-1 ~]$ nc -zv [mgr_ip] 8086
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:8086.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-1 ~]$ nc -zv [mgr_ip] 4505
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:4505.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-1 ~]$ nc -zv [mgr_ip] 4506
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:4506.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-1 ~]$ nc -zv [mgr_ip] 8220
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:8220.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-1 ~]$ nc -zv [mgr_ip] 8443
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:8443.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-1 ~]$ nc -zv [mgr_ip] 5055
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:5055.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-1 ~]$ nc -zv [mgr_ip] 9200
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:9200.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-1 ~]$ nc -zv [mgr_ip] 9300
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:9300.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-1 ~]$ nc -zv [mgr_ip] 9696
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:9696.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-1 ~]$ nc -zv [mgr_ip] 9200
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:9200.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-1 ~]$ nc -zv [mgr_ip] 5056
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: TIMEOUT.

[onion@angband-search-2 ~]$ nc -zv [mgr_ip] 443
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:443.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-2 ~]$ nc -zv [mgr_ip] 5000
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:5000.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-2 ~]$ nc -zv [mgr_ip] 8086
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:8086.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-2 ~]$ nc -zv [mgr_ip] 4505
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:4505.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-2 ~]$ nc -zv [mgr_ip] 4506
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:4506.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-2 ~]$ nc -zv [mgr_ip] 8220
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:8220.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-2 ~]$ nc -zv [mgr_ip] 8443
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:8443.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-2 ~]$ nc -zv [mgr_ip] 5055
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:5055.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-2 ~]$ nc -zv [mgr_ip] 9200
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:9200.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-2 ~]$ nc -zv [mgr_ip] 9300
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:9300.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-2 ~]$ nc -zv [mgr_ip] 9696
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:9696.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-2 ~]$ nc -zv [mgr_ip] 9200
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to [mgr_ip]:9200.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
[onion@angband-search-2 ~]$ nc -zv [mgr_ip] 5056
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: TIMEOUT.

I searched the minion log files on both search nodes for errors and didn't see anything present.

Also - I definitely only ran the oneliner and then a highstate. Thank you for the new command, I ran it and ran another highstate. We still have a good chunk of unassigned shards that I need to figure out what to do to address.

[onion@angband ~]$ sudo so-elasticsearch-query _cluster/health?pretty
'{
  "cluster_name" : "securityonion",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 202,
  "active_shards" : 333,
  "relocating_shards" : 2,
  "initializing_shards" : 2,
  "unassigned_shards" : 22,
  "unassigned_primary_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 93.27731092436974
}