Security Onion 2.4.12 - Missing Shards in Elasticsearch (Cannot Access Hunt, Alerts, Dashboard)

[SankaGamage]

We are experiencing an issue in Security Onion 2.4.12 (latest version) where we can see logs in Kibana, but we cannot access Hunt, Alerts, or the Dashboard. The following error appears repeatedly in the logs.

LOG:-
[2025-02-24T11:01:30,434][WARN ][rest.suppressed ] path: /.ds-logs-/_eql/search, params: {ignore_unavailable=true, index=.ds-logs-}, status: 503
org.elasticsearch.action.search.SearchPhaseExecutionException: start
at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.onPhaseFailure(CanMatchPreFilterSearchPhase.java:416) ~[elasticsearch-8.14.3.jar:?]
...
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.ds-logs-elastic_agent-default-2025.01.24-000001][0],
[.ds-logs-elastic_agent.filebeat-default-2025.01.24-000001][0], [.ds-logs-elastic_agent.fleet_server-default-2025.01.24-000001][0], [.ds-logs-elasticsearch.server-default-2025.01.24-000001][0],
[.ds-logs-soc-so-2025.01.24-000001][0], [.ds-logs-system.syslog-default-2025.01.24-000001][0], [.ds-logs-zeek-so-2025.01.24-000001][0]].

Consider using allow_partial_search_results setting to bypass this error.

Environment Details

• Security Onion Version: 2.4.12 (Latest)
• Deployment Type: Distributed (Forward, Search, and Receiver Nodes)

Question

• How can we fix this issue and restore access to Hunt, Alerts, and the Dashboard?

Any help or guidance would be greatly appreciated! 🚀

[dougburks]

Based on your comment at #14053 (reply in thread), if you have destroyed Elasticsearch indices from the filesystem without using the Elastic API then you've created lots of problems on your system. You may be able to recover your system without a full reinstall but we can't provide any support for that.

[SankaGamage]

Based on your comment at #14053 (reply in thread), if you have destroyed Elasticsearch indices from the filesystem without using the Elastic API then you've created lots of problems on your system. You may be able to recover your system without a full reinstall but we can't provide any support for that.

I didn’t delete any Elasticsearch indices manually. I believe I misconfigured something, so I restored a snapshot of a previous state. After restoring, I started seeing this issue. Could you please guide me on how to resolve it?

[dougburks]

Looking at your screenshot, there is an exclamation point next to Grid. Does the Grid page show Elasticsearch as Pending?
https://docs.securityonion.net/en/2.4/elasticsearch.html#status-pending