Elasticsearch Status: Fault

[innovate-support]

Version

2.4.100

Installation Method

Network installation on Debian

Description

other (please provide detail below)

Installation Type

Distributed

Location

cloud

Hardware Specs

Meets minimum requirements

CPU

8

RAM

16

Storage for /

330

Storage for /nsm

330

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Yesterday my Elasticsearch Status suddenly said Fault. I investigated the indexes a bit using some of my notes that fixed some index issues in the past, but none of my previous 'fixes' worked this time. So I just rebooted the server and everything came back up and worked fine. But then this morning it went into Fault again.

When I run this command sudo so-elasticsearch-indices-list | grep -vE "green|health" it says
red open .ds-logs-suricata.alerts-so-2024.09.11-000037 kDtXYcl7QUaAr_LKjQnLkQ 1 0

When I run so-elasticsearch-query _cat/shards | grep UNASSIGNED it says
.ds-logs-suricata.alerts-so-2024.09.11-000037 0 p UNASSIGNED

I already tried changing the Watermark settings last week, which fixed the Pending issues I had last week, but this is a new problem. It's not Pending anymore, it's Faulting. Is there any script or something that I can run to clean this up? I'm willing to delete things if needed, but I don't want to rebuild unless it's a last resort since I have some Forward nodes and numerous endpoints sending their Elastic data into this server.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[innovate-support]

Here's some addition log info that looks like it may be helpful to someone who understands what it means.

root@soman:/opt/so/log/elasticsearch# tail securityonion.log
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1570) ~[?:?]
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.ds-logs-suricata.alerts-so-2024.09.11-000037][0]]. Consider using allow_partial_search_results setting to bypass this error.
at org.elasticsearch.action.search.SearchPhase.doCheckNoMissingShards(SearchPhase.java:61) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.checkNoMissingShards(CanMatchPreFilterSearchPhase.java:196) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.run(CanMatchPreFilterSearchPhase.java:131) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase$1.doRun(CanMatchPreFilterSearchPhase.java:410) ~[elasticsearch-8.14.3.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.14.3.jar:?]
... 6 more

[innovate-support]

After asking ChatGPT about the details of this, it's telling me that the issue is related to disk space. The node "soman" has exceeded the high watermark setting (cluster.routing.allocation.disk.watermark.high=80%), and it requires at least 61 GB of free space, but only 57.2 GB is available.

So I'm dropping the 'high' watermark down to 70%. We'll see if that fixes it.

[innovate-support]

I'm a bit confused about the Watermark settings vs the ILM settings. I think ILM is what used to be called Curator.

[innovate-support]

After many hours of fixing broken indexes, and adjusting the Elasticsearch Global ILM Policy's, I've got things working again. Hopefully the ILM and Watermark settings don't let this happen again.

[witekenea]

hello there, i have got the same issue as you provided above. can you tell me how you solved this issue?

[innovate-support]

Mine failed again a day or so after 'fixing' it. I tried many things without success. I even completely deleted SO off my box and re-installed. That didn't work either. Elasticsearch would never go green and function.
Here's how I re-installed.
git clone -b 2.4/main https://github.com/Security-Onion-Solutions/securityonion

My box was a Debian 12 VM hosted on Linode, which worked great for about a year and then suddenly failed shortly after upgrading to 2.4.100. No trouble with prior versions. So, I've gone with the official ISO as a new VM on my own VMware physical hardware. I haven't completed my setup yet, but if I have similar issues I will post back here.