so-elastalert is missing after restart of the server

[sanzit99]

Version

2.4.141

Installation Method

Security Onion ISO image

Description

installation

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

18 core 2.60 GHz single socket

RAM

126 GB

Storage for /

314 GB

Storage for /nsm

12263 GB

Network Traffic Collection

tap

Network Traffic Speeds

Less than 1Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
801c81de4c3d so-live:5000/security-onion-solutions/so-strelka-backend:2.4.141 "strelka-backend" About an hour ago Up About an hour so-strelka-backend
5a47ab6e2598 so-live:5000/security-onion-solutions/so-elasticsearch:8.17.3 "/bin/tini -- /usr/l…" About an hour ago Up About an hour 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp so-elasticsearch
b595722b360c so-live:5000/security-onion-solutions/so-zeek:2.4.141 "/usr/local/sbin/zee…" 7 days ago Up About an hour (healthy) so-zeek
ac98a8078115 so-live:5000/security-onion-solutions/so-strelka-filestream:2.4.141 "strelka-filestream" 7 days ago Up About an hour so-strelka-filestream
2bb35e163784 so-live:5000/security-onion-solutions/so-idstools:2.4.141 "./entrypoint.sh" 8 weeks ago Up About an hour so-idstools
2dfc6be072db so-live:5000/security-onion-solutions/so-suricata:2.4.141 "/usr/local/sbin/so-…" 8 weeks ago Up About an hour so-suricata
ed67e6b83745 so-live:5000/security-onion-solutions/so-logstash:2.4.141 "/usr/local/bin/dock…" 2 months ago Up About an hour 0.0.0.0:3765->3765/tcp, 0.0.0.0:5044->5044/tcp, 0.0.0.0:5055-5056->5055-5056/tcp, 0.0.0.0:5644->5644/tcp, 0.0.0.0:6050-6053->6050-6053/tcp, 0.0.0.0:9600->9600/tcp so-logstash
bb135d949cee so-live:5000/security-onion-solutions/so-elastic-agent:2.4.141 "/usr/bin/tini -- /u…" 2 months ago Up About an hour 0.0.0.0:8220->8220/tcp so-elastic-fleet
49701d0c6212 so-live:5000/security-onion-solutions/so-kibana:2.4.141 "/usr/local/bin/so-k…" 2 months ago Up About an hour 0.0.0.0:5601->5601/tcp so-kibana
7ebbbde39bbf so-live:5000/security-onion-solutions/so-soc:2.4.141 "/opt/sensoroni/sens…" 2 months ago Up About an hour 0.0.0.0:9822->9822/tcp so-soc
326aaa7a1337 so-live:5000/security-onion-solutions/so-strelka-manager:2.4.141 "strelka-manager" 2 months ago Up About an hour so-strelka-manager
cb71acd3eac7 so-live:5000/security-onion-solutions/so-strelka-frontend:2.4.141 "strelka-frontend" 2 months ago Up About an hour 0.0.0.0:57314->57314/tcp so-strelka-frontend
29b4d4bb4d83 so-live:5000/security-onion-solutions/so-redis:2.4.141 "redis-server --save…" 2 months ago Up About an hour 0.0.0.0:6381->6379/tcp so-strelka-gatekeeper
fc2755edae91 so-live:5000/security-onion-solutions/so-redis:2.4.141 "redis-server --save…" 2 months ago Up About an hour 0.0.0.0:6380->6379/tcp so-strelka-coordinator
3d8b2eb14d56 so-live:5000/security-onion-solutions/so-elastic-fleet-package-registry:2.4.141 "./package-registry" 2 months ago Up About an hour (healthy) 0.0.0.0:8080->8080/tcp so-elastic-fleet-package-registry
94e6957dfb0b so-live:5000/security-onion-solutions/so-redis:2.4.141 "redis-server /usr/l…" 2 months ago Up About an hour 0.0.0.0:6379->6379/tcp, 0.0.0.0:9696->9696/tcp so-redis
eaecaace5ad2 so-live:5000/security-onion-solutions/so-telegraf:2.4.141 "/entrypoint.sh tele…" 2 months ago Up About an hour so-telegraf
30b5fa7e06e1 so-live:5000/security-onion-solutions/so-soc:2.4.141 "/opt/sensoroni/sens…" 2 months ago Up About an hour so-sensoroni
8569498d1c97 so-live:5000/security-onion-solutions/so-nginx:2.4.141 "/docker-entrypoint.…" 2 months ago Up About an hour (healthy) 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:7788-7789->7788-7789/tcp, 0.0.0.0:8443->8443/tcp so-nginx
86a868189b62 so-live:5000/security-onion-solutions/so-influxdb:2.4.141 "/redirect_to_file.s…" 2 months ago Up About an hour (healthy) 0.0.0.0:8086->8086/tcp so-influxdb
2db51f5543a1 so-live:5000/security-onion-solutions/so-kratos:2.4.141 "/start-kratos.sh" 2 months ago Up About an hour 0.0.0.0:4433-4434->4433-4434/tcp so-kratos
af2bb8b2cef3 ghcr.io/security-onion-solutions/registry:2.8.3 "/entrypoint.sh /etc…" 2 months ago Up About an hour 0.0.0.0:5000->5000/tcp so-dockerregistry

In the Grid:
so-elastalert - missing

Dashboard error:
Kibana dashboard showing 404 error.
Fleet agent is also showing 404 error.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[rh-ops]

Run the following command and relay any fail messages it returns:

sudo salt-call state.apply elastalert -linfo

And also run these commands and provide the outputs:

sudo so-elasticsearch-query _cluster/health?pretty

sudo so-elasticsearch-shards-list | grep UNAS

sudo so-elasticsearch-query _cluster/allocation/explain?pretty

sudo so-elasticsearch-query _cat/allocation?v

[sanzit99]

[user@so-live ~]$ sudo salt-call state.apply elastalert -linfo

local:
Data failed to compile:

The function "state.highstate" is running as PID 2712973 and was started at 2025, Jul 03 06:32:01.205671 with jid 20250703063201205671

[user@so-live ~]$ sudo so-elasticsearch-query _cluster/health?pretty
{
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "current license is non-compliant for [security]",
"license.expired.feature" : "security"
}
],
"type" : "security_exception",
"reason" : "current license is non-compliant for [security]",
"license.expired.feature" : "security"
},
"status" : 403
}

[user@so-live ~]$ sudo so-elasticsearch-shards-list | grep UNAS

[user@so-live ~]$ sudo so-elasticsearch-query _cluster/allocation/explain?pretty
{
"note" : "No shard was specified in the explain API request, so this response explains a randomly chosen unassigned shard. There may be other unassigned shards in this cluster which cannot be assigned for different reasons. It may not be possible to assign this shard until one of the other shards is assigned correctly. To explain the allocation of other shards (whether assigned or unassigned) you must specify the target shard in the request to this API. See https://www.elastic.co/guide/en/elasticsearch/reference/8.17/cluster-allocation-explain.html for more information.",
"index" : ".ds-logs-suricata.alerts-so-2025.05.19-000047",
"shard" : 0,
"primary" : true,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "CLUSTER_RECOVERED",
"at" : "2025-07-02T15:51:52.461Z",
"last_allocation_status" : "no_valid_shard_copy"
},
"can_allocate" : "no_valid_shard_copy",
"allocate_explanation" : "Elasticsearch can't allocate this shard because there are no copies of its data in the cluster. Elasticsearch will allocate this shard when a node holding a good copy of its data joins the cluster. If no such node is available, restore this index from a recent snapshot.",
"node_allocation_decisions" : [
{
"node_id" : "YscQgk0zRUWLNDKxepaqDA",
"node_name" : "so-live",
"transport_address" : "192.168.1.5:9300",
"node_attributes" : {
"transform.config_version" : "10.0.0",
"xpack.installed" : "true",
"ml.config_version" : "12.0.0"
},
"roles" : [
"data",
"data_hot",
"ingest",
"master",
"remote_cluster_client",
"transform"
],
"node_decision" : "no",
"store" : {
"found" : false
}
}
]
}

[user@so-live ~]$ sudo so-elasticsearch-query _cat/allocation?v
{"error":{"root_cause":[{"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security"}],"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security"},"status":403}[user@so-live ~]$

[InfosecGoon]

It looks like you've turned on some Elastic Security features that you aren't licensed for, so the cluster isn't responding properly. This may well be what's causing the Elastalert issue, because it verifies the cluster health as part of the container startup process.

Did you turn on or enable anything in Kibana that you can recall?

[sanzit99]

[sanzit99]

[kspringer-maf]

Running sudo so-checkin from the node that isn't working has saved me many times when the state.highstate command leaves things broken.