"Elasticsearch Status:Fault" after upgrade to 2.4.160

[geistchevalier]

Version

2.4.160

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

6

RAM

64

Storage for /

500GB

Storage for /nsm

2TB

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

As per title of this discussion

Grid status:

so-status :

                         Security Onion Status
                         Container │ Status  │                 Details
───────────────────────────────────┼─────────┼─────────────────────────
                 so-dockerregistry │ running │           Up 17 minutes
                     so-elastalert │ running │           Up 12 minutes
                  so-elastic-fleet │ running │           Up 12 minutes
 so-elastic-fleet-package-registry │ running │ Up 13 minutes (healthy)
                  so-elasticsearch │ running │           Up 14 minutes
                       so-idstools │ running │           Up 14 minutes
                       so-influxdb │ running │ Up 14 minutes (healthy)
                         so-kibana │ running │           Up 13 minutes
                         so-kratos │ running │           Up 17 minutes
                       so-logstash │ running │           Up 13 minutes
                          so-nginx │ running │ Up 14 minutes (healthy)
                          so-redis │ running │           Up 13 minutes
                      so-sensoroni │ running │           Up 14 minutes
                            so-soc │ running │           Up 14 minutes
                          so-steno │ running │           Up 13 minutes
                so-strelka-backend │ running │           Up 12 minutes
            so-strelka-coordinator │ running │           Up 12 minutes
             so-strelka-filestream │ running │           Up 12 minutes
               so-strelka-frontend │ running │           Up 12 minutes
             so-strelka-gatekeeper │ running │           Up 12 minutes
                so-strelka-manager │ running │           Up 12 minutes
                       so-suricata │ running │           Up 13 minutes
                       so-telegraf │ running │           Up 14 minutes
                           so-zeek │ running │ Up 13 minutes (healthy)

/opt/so/log/elasticsearch/securityonion.log (the following repeats on and on)

...
[2025-06-29T00:29:40,036][WARN ][rest.suppressed          ] path: /.ds-logs-*/_eql/search, params: {ignore_unavailable=true, index=.ds-logs-*}, status: 503
org.elasticsearch.action.search.SearchPhaseExecutionException: start
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.onPhaseFailure(CanMatchPreFilterSearchPhase.java:422) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase$1.onFailure(CanMatchPreFilterSearchPhase.java:411) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:29) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:34) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-8.17.3.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1575) ~[?:?]
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.ds-logs-soc-so-2025.06.12-000031][0]]. Consider using `allow_partial_search_results` setting to bypass this error.
        at org.elasticsearch.action.search.SearchPhase.doCheckNoMissingShards(SearchPhase.java:69) ~[elasticsearch-8.17.3.jar:?]
[2025-06-29T00:29:40,030][WARN ][rest.suppressed          ] path: /.ds-logs-*/_eql/search, params: {ignore_unavailable=true, index=.ds-logs-*}, status: 503
org.elasticsearch.action.search.SearchPhaseExecutionException: start
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.onPhaseFailure(CanMatchPreFilterSearchPhase.java:422) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase$1.onFailure(CanMatchPreFilterSearchPhase.java:411) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:29) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:34) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-8.17.3.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1575) ~[?:?]
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.ds-logs-soc-so-2025.06.12-000031][0]]. Consider using `allow_partial_search_results` setting to bypass this error.
        at org.elasticsearch.action.search.SearchPhase.doCheckNoMissingShards(SearchPhase.java:69) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.checkNoMissingShards(CanMatchPreFilterSearchPhase.java:202) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.runCoordinatorRewritePhase(CanMatchPreFilterSearchPhase.java:189) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.run(CanMatchPreFilterSearchPhase.java:144) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase$1.doRun(CanMatchPreFilterSearchPhase.java:416) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-8.17.3.jar:?]
        ... 6 more
[2025-06-29T00:29:40,036][WARN ][org.elasticsearch.xpack.eql.plugin.RestEqlSearchAction] Request failed with status [SERVICE_UNAVAILABLE]:
org.elasticsearch.action.search.SearchPhaseExecutionException: start
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.onPhaseFailure(CanMatchPreFilterSearchPhase.java:422) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase$1.onFailure(CanMatchPreFilterSearchPhase.java:411) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:29) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:34) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-8.17.3.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1575) ~[?:?]
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.ds-logs-soc-so-2025.06.12-000031][0]]. Consider using `allow_partial_search_results` setting to bypass this error.
        at org.elasticsearch.action.search.SearchPhase.doCheckNoMissingShards(SearchPhase.java:69) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.checkNoMissingShards(CanMatchPreFilterSearchPhase.java:202) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.runCoordinatorRewritePhase(CanMatchPreFilterSearchPhase.java:189) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.run(CanMatchPreFilterSearchPhase.java:144) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase$1.doRun(CanMatchPreFilterSearchPhase.java:416) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-8.17.3.jar:?]
        ... 6 more      at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.checkNoMissingShards(CanMatchPreFilterSearchPhase.java:202) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.runCoordinatorRewritePhase(CanMatchPreFilterSearchPhase.java:189) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.run(CanMatchPreFilterSearchPhase.java:144) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase$1.doRun(CanMatchPreFilterSearchPhase.java:416) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-8.17.3.jar:?]
        ... 6 more
...

looking at unassigned index I get the following:

[admin@securityonion ~]$ sudo so-elasticsearch-query _cat/shards | grep UNASSIGNED
.ds-logs-soc-so-2025.06.12-000031                                         0 p UNASSIGNED

trying to delete gives acknowledged: true, but it still exists when querying for unassigned

[admin@securityonion ~]$ sudo so-elasticsearch-query .ds-logs-soc-so-2025.06.12-000031/_settings -d '{"number_of_replicas":0}' -XPUT
{"acknowledged":true}
[admin@securityonion ~]$ sudo so-elasticsearch-query _cat/shards | grep UNASSIGNED
.ds-logs-soc-so-2025.06.12-000031  

trying to delete via ElasticSearch index management gives the following response

I'm not sure how to proceed or fix this.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[rh-ops]

Looks like you're trying to remove a primary shard for SOC. However, it is the write index and cannot be deleted until it is rolled over and a new write index is created. To do that, use the following commands.

sudo so-elasticsearch-query logs-soc-so/_rollover -XPOST

then

so-elasticsearch-query <index_name> -XDELETE

[geistchevalier]

That fixed it, thank you.

[samuel809]

you can also delete non writable indexes via gui.
i sort them by storage size.

consider reducing shard size via ilm policy to have some flexibility deleting previous indices.

[SanibelJack]

Which one are non writeable? I'm fairly new at this