so-elastalert is missing all GET _cat/indices/.kibana* are yellow

[byronsimseku]

Version

2.4.160

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

6

RAM

16GB

Storage for /

500GB

Storage for /nsm

327GB

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

I have a master node, 3 storage nodes, and 2 sensor nodes. so-elastalert is missing from my master node. When I do "GET _cat/indices/.kibana*" from the kibana dev tools, this is the output:

yellow open .ds-.kibana-event-log-8.10.4-2024.10.01-000012 2iEtuyfpQ824dPKUiJxLRw 1 1 0 0 250b 250b 250b
yellow open .kibana_task_manager_8.10.4_001 x56oIGj8QgqMQOe5eJoJDw 1 1 40 6 87.2kb 87.2kb 87.2kb
yellow open .kibana_analytics_8.10.4_001 zkkFmpQhQPaOEBUcwBu6uQ 1 1 22325 1833 13.3mb 13.3mb 13.3mb
yellow open .ds-.kibana-event-log-ds-2025.09.10-000051 nNidKlpcT42QmihNhaBYTw 1 1 0 0 249b 249b 249b
yellow open .kibana-observability-ai-assistant-conversations-000001 rbCxx91-QHivqDjfh69hgg 1 1 0 0 250b 250b 250b
yellow open .kibana_ingest_8.10.4_001 EPL6ImOZSWKS0_JlVxkZDw 1 1 19670 2635 458.4mb 458.4mb 458.4mb
yellow open .kibana_usage_counters_8.17.3_001 cQ6-hlebRseVA-0UaoY2_w 1 1 240 22 88.9kb 88.9kb 88.9kb
yellow open .kibana_8.10.4_001 CHyT_tInSNurvbqH1WNK3w 1 1 494 15 143.3kb 143.3kb 143.3kb
yellow open .kibana_security_session_1 q5wF65kySEWcbjb1_lzAvA 1 1 1 0 6.8kb 6.8kb 6.8kb
yellow open .kibana_security_solution_8.10.4_001 rErtCuBCSue7WotAorfRDw 1 1 4014 113 9.8mb 9.8mb 9.8mb
yellow open .kibana_alerting_cases_8.10.4_001 J6PXa9BCSUe6kq_3C51nlg 1 1 1 0 6.9kb 6.9kb 6.9kb

As things are trying to startup, I can see that it's waiting for these to be "green open", or that's what I'm assuming.

Salt failures:

      ID: wait_for_elasticsearch
Function: cmd.run
    Name: so-elasticsearch-wait
  Result: False
 Comment: Command "so-elasticsearch-wait" run
 Started: 15:46:47.129091
Duration: 309875.858 ms
 Changes:

---

      ID: so-elastalert
Function: docker_container.running
  Result: False
 Comment: One or more requisite failed: elastalert.enabled.wait_for_elasticsearch
 Started: 15:51:57.017565
Duration: 0.005 ms
 Changes:

---

      ID: wait_for_elasticsearch_elasticfleet
Function: cmd.run
    Name: so-elasticsearch-wait
  Result: False
 Comment: Command "so-elasticsearch-wait" run
 Started: 15:51:58.675625
Duration: 309450.082 ms
 Changes:

There may be other things in the logs, but I'm not sure which ones might be interesting.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

What does this command return? sudo so-elasticsearch-query _health_report?pretty