ElasticSearch on manager is in Fault state

[Ximalas]

I've struggled for a few days with ElasticSearch on our manager node. It won't leave the Fault state. Today, I've restarted the whole fleet due to OS updates, and I also ran sudo soup twice on the manager.

I've identified a few unassigned indices:

$ sudo so-elasticsearch-shards-list | awk '/UNASSIGNED/{print $1}'

.ds-logs-elastic_agent-default-2024.10.15-000001
.ds-logs-elastic_agent.fleet_server-default-2024.10.15-000001
.ds-logs-elasticsearch.server-default-2024.10.15-000001
.ds-logs-system.syslog-default-2024.10.15-000001
.ds-logs-soc-so-2024.10.15-000001

A subsequent attempt at removing these indices seems to succeed, but to no avail:

for index in `sudo so-elasticsearch-shards-list | awk '/UNASSIGNED/{print $1}'`; do
  echo "${index}:";
  sudo so-elasticsearch-query $index/_settings -d '{"number_of_replicas":0}' -XPUT
  echo; echo
done

.ds-logs-elastic_agent-default-2024.10.15-000001:
{"acknowledged":true}

.ds-logs-elastic_agent.fleet_server-default-2024.10.15-000001:
{"acknowledged":true}

.ds-logs-elasticsearch.server-default-2024.10.15-000001:
{"acknowledged":true}

.ds-logs-system.syslog-default-2024.10.15-000001:
{"acknowledged":true}

.ds-logs-soc-so-2024.10.15-000001:
{"acknowledged":true}

In /opt/so/log/elasticsearch/securityonion.log, I see error messages such as:

[2024-10-15T12:27:16,685][WARN ][rest.suppressed          ] path: /.ds-logs-*/_eql/search, params: {ignore_unavailable=true, index=.ds-logs-*}, status: 503
org.elasticsearch.action.search.SearchPhaseExecutionException: start
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.onPhaseFailure(CanMatchPreFilterSearchPhase.java:416) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase$1.onFailure(CanMatchPreFilterSearchPhase.java:405) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.14.3.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1570) ~[?:?]
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.ds-logs-elastic_agent-default-2024.10.15-000001][0], [.ds-logs-elastic_agent.filebeat-default-2024.10.15-000001][0], [.ds-logs-elastic_agent.fleet_server-default-2024.10.15-000001][0], [.ds-logs-elasticsearch.server-default-2024.10.15-000001][0], [.ds-logs-system.syslog-default-2024.10.15-000001][0]]. Consider using `allow_partial_search_results` setting to bypass this error.
        at org.elasticsearch.action.search.SearchPhase.doCheckNoMissingShards(SearchPhase.java:61) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.checkNoMissingShards(CanMatchPreFilterSearchPhase.java:196) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.run(CanMatchPreFilterSearchPhase.java:131) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase$1.doRun(CanMatchPreFilterSearchPhase.java:410) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.14.3.jar:?]
        ... 6 more
[2024-10-15T12:27:17,705][WARN ][org.elasticsearch.xpack.eql.plugin.RestEqlSearchAction] Request failed with status [SERVICE_UNAVAILABLE]:

What can I do to get ElasticSearch operational again, short of reinstalling everything?

I've also noticed the new release notes asking us to delete any incorrect indices via the command line. One piece of advice, don't assume everyone is an expert, well, not initially, so from now on, please provide a working example or at least be more descriptive about what a data stream is and how they are named.

[Ximalas]

pcap > config > maxdirectoryfiles was set too high, causing issues on the search node, yes, singular. Reducing the value to 144000 and waiting for the stenographer to pick up this value and act on it, made all the difference.

[dougburks]

I've also noticed the new release notes asking us to delete any incorrect indices via the command line. One piece of advice, don't assume everyone is an expert, well, not initially, so from now on, please provide a working example or at least be more descriptive about what a data stream is and how they are named.

Noted, we will work to improve this.

pcap > config > maxdirectoryfiles was set too high, causing issues on the search node, yes, singular. Reducing the value to 144000 and waiting for the stenographer to pick up this value and act on it, made all the difference.

I'm glad you were able to resolve this issue, but I'm not sure I understand. How was stenographer impacting your search node? Search nodes shouldn't run stenographer:
https://docs.securityonion.net/en/2.4/architecture.html#search-node

Unless perhaps your search node is actually a heavy node (not recommended)?
https://docs.securityonion.net/en/2.4/architecture.html#heavy-node

[Ximalas]

Our manager was unhappy with its ElasticSearch and the /nsm filesystem got amber. Our search node wasn't exactly unhappy, but its /nsm filesystem got blue. And the Redis queues on both the manager and the receiver node filled up. Our maxdirectoryfiles got increased over the summer, ending up as high as 161000 files. I figured I should bring this one down 2000 files at a time, but that took too long. I then remembered that 30000 files is about 21.4 days, so 150000 files should make roughly 107 days. Even that turned out to be too high, so we're running at 144000 files and maybe we need to lower it further. We might have underestimated the storage for the manager and search node. It's no big deal. I'm content with ≥ 90 days of traffic on the forward node and I'm learning something every day.

[Ximalas]

Today I cleared everything stored in ElasticSearch by running so-elastic-clear. I later realised I should be adjusting elasticsearch > index_settings > global_overrides > policy > phases > delete > min_age. I set this one to a value close to the PCAP retention days, e.g. 100d. We'll see how this turns out in about four months.

[dougburks]

I've also noticed the new release notes asking us to delete any incorrect indices via the command line. One piece of advice, don't assume everyone is an expert, well, not initially, so from now on, please provide a working example or at least be more descriptive about what a data stream is and how they are named.

We've updated the Release Notes:
https://docs.securityonion.net/en/2.4/release-notes.html#known-issues

Please take a look and let us know whether or not that helps.

[Ximalas]

I saw it earlier today. It's wonderfully written, thank you so much. My issue was something different, running out of disk space on both search and manager, although the latter was hit harder. Adjusting the retention period (delete) for ElasticSearch became key in solving my issue.