Alerts not acknowledged (Error 400)/Partial shards failure #14581

dnade · 2025-04-30T19:00:56Z

dnade
Apr 30, 2025

Version

2.4.140

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

32

RAM

MemTotal: 197432352 kB

Storage for /

293G

Storage for /nsm

47T

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

Recently when I go to the 'alerts' section and try to acknowledge any alerts, I get a 'Request failed with status code 400'. when looking at /opt/so/log/soc/sensoroni-server.log after getting the error I see:

{"fields":{"acknowledge":true,"escalate":false,"eventFilter":{"count":2,"event.module":"suricata","event.severity_label":"high","rule.name":"ET RETIRED [ANY.RUN] Lu0bot-Style DNS Query in DNS Lookup M5","rule.uuid":"2048328"},"requestId":"0058a7b8-14dd-4239-afc1-7526cc892fa4","searchFilter":"(*) AND tags:alert AND NOT event.acknowledged:true AND NOT event.escalated:true | groupby rule.name event.module* event.severity_label rule.uuid"},"level":"info","timestamp":"2025-04-30T18:38:22.990865343Z","message":"Acknowledging event"}
{"fields":{"contentLength":0,"elapsedMs":0,"impl":"/build/server/nodehandler.go:67:github.com/security-onion-solutions/securityonion-soc/server.(*NodeHandler).postNode","remoteAddr":"172.17.1.1:40034","requestId":"13168ecb-a0be-45a0-999e-1260f484d29d","requestMethod":"POST","requestPath":"/api/node","requestQuery":{},"requestorId":"00000000-0000-0000-0000-000000000000","sourceIp":"192.168.1.19","statusCode":200},"level":"info","timestamp":"2025-04-30T18:38:23.132891771Z","message":"Handled request"}
{"fields":{"contentLength":0,"elapsedMs":0,"impl":"/build/server/nodehandler.go:67:github.com/security-onion-solutions/securityonion-soc/server.(*NodeHandler).postNode","remoteAddr":"172.17.1.1:40038","requestId":"c318436b-1c57-4902-94de-7ca69a687ab5","requestMethod":"POST","requestPath":"/api/node","requestQuery":{},"requestorId":"00000000-0000-0000-0000-000000000000","sourceIp":"192.168.1.19","statusCode":200},"level":"info","timestamp":"2025-04-30T18:38:24.135367664Z","message":"Handled request"}
{"fields":{"clientHost":"https://krc-so-02:9200/","error":"search_phase_execution_exception: Partial shards failure -\u003e {\n  \"error\" : {\n    \"root_cause\" : [\n      {\n        \"type\" : \"too_many_scroll_contexts_exception\",\n        \"reason\" : \"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.\"\n      },\n      {\n        \"type\" : \"too_many_scroll_contexts_exception\",\n        \"reason\" : \"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.\"\n      }\n    ],\n    \"type\" : \"search_phase_execution_exception\",\n    \"reason\" : \"Partial shards failure\",\n    \"phase\" : \"query\",\n    \"grouped\" : true,\n    \"failed_shards\" : [\n      {\n        \"shard\" : 0,\n        \"index\" : \".ds-logs-system.security-default-2025.04.29-000105\",\n        \"node\" : \"7FFTNFawT3GCMoRIoyE0qQ\",\n        \"reason\" : {\n          \"type\" : \"too_many_scroll_contexts_exception\",\n          \"reason\" : \"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.\"\n        }\n      },\n      {\n        \"shard\" : 463,\n        \"index\" : \".ds-logs-kratos-so-2025.04.24-000015\",\n        \"node\" : \"7FFTNFawT3GCMoRIoyE0qQ\",\n        \"reason\" : {\n          \"type\" : \"too_many_scroll_contexts_exception\",\n          \"reason\" : \"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.\"\n        }\n      }\n    ]\n  },\n  \"status\" : 429\n}\n","requestId":"0058a7b8-14dd-4239-afc1-7526cc892fa4"},"level":"error","timestamp":"2025-04-30T18:38:24.356557343Z","message":"Encountered error while updating elasticsearch"}
{"fields":{"error":"search_phase_execution_exception: Partial shards failure -\u003e {\n  \"error\" : {\n    \"root_cause\" : [\n      {\n        \"type\" : \"too_many_scroll_contexts_exception\",\n        \"reason\" : \"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.\"\n      },\n      {\n        \"type\" : \"too_many_scroll_contexts_exception\",\n        \"reason\" : \"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.\"\n      }\n    ],\n    \"type\" : \"search_phase_execution_exception\",\n    \"reason\" : \"Partial shards failure\",\n    \"phase\" : \"query\",\n    \"grouped\" : true,\n    \"failed_shards\" : [\n      {\n        \"shard\" : 0,\n        \"index\" : \".ds-logs-system.security-default-2025.04.29-000105\",\n        \"node\" : \"7FFTNFawT3GCMoRIoyE0qQ\",\n        \"reason\" : {\n          \"type\" : \"too_many_scroll_contexts_exception\",\n          \"reason\" : \"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.\"\n        }\n      },\n      {\n        \"shard\" : 463,\n        \"index\" : \".ds-logs-kratos-so-2025.04.24-000015\",\n        \"node\" : \"7FFTNFawT3GCMoRIoyE0qQ\",\n        \"reason\" : {\n          \"type\" : \"too_many_scroll_contexts_exception\",\n          \"reason\" : \"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.\"\n        }\n      }\n    ]\n  },\n  \"status\" : 429\n}\n","requestId":"0058a7b8-14dd-4239-afc1-7526cc892fa4","requestorId":"76507a1a-92ba-456b-ab6f-6123aa84a7d8"},"level":"warn","timestamp":"2025-04-30T18:38:24.356664963Z","message":"Request did not complete successfully"}
{"fields":{"contentLength":35,"elapsedMs":1373,"impl":"/build/server/eventhandler.go:122:github.com/security-onion-solutions/securityonion-soc/server.(*EventHandler).postAck","remoteAddr":"172.17.1.1:40022","requestId":"0058a7b8-14dd-4239-afc1-7526cc892fa4","requestMethod":"POST","requestPath":"/api/events/ack","requestQuery":{},"requestorId":"76507a1a-92ba-456b-ab6f-6123aa84a7d8","sourceIp":"192.168.1.25","statusCode":400},"level":"info","timestamp":"2025-04-30T18:38:24.35676808Z","message":"Handled request"}

on /opt/so/log/elasticsearch/securityonion.log i see:

[2025-04-30T18:38:11,696][ERROR][org.elasticsearch.xpack.ilm.IndexLifecycleRunner] policy [so-suricata.alerts-logs] for index [.ds-logs-suricata.alerts-so-2025.04.29-000235] failed on step [{"phase":"hot","action":"rollover","name":"attempt-rollover"}]. Moving to ERROR step
org.elasticsearch.common.ValidationException: Validation Failed: 1: this action would add [600] shards, but this cluster currently has [1643]/[2000] maximum normal shards open; for more information, see https://www.elastic.co/guide/en/elasticsearch/reference/8.17/size-your-shards.html#troubleshooting-max-shards-open;
        at org.elasticsearch.indices.ShardLimitValidator.validateShardLimit(ShardLimitValidator.java:117) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.aggregateIndexSettings(MetadataCreateIndexService.java:1127) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequestWithV2Template(MetadataCreateIndexService.java:678) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:421) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.action.admin.indices.rollover.MetadataRolloverService.rolloverDataStream(MetadataRolloverService.java:405) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.action.admin.indices.rollover.MetadataRolloverService.rolloverClusterState(MetadataRolloverService.java:164) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction$RolloverExecutor.executeTask(TransportRolloverAction.java:542) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction$RolloverExecutor.execute(TransportRolloverAction.java:462) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1075) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1038) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:245) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1691) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.action.ActionListener.run(ActionListener.java:452) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1688) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1283) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.action.ActionListener.run(ActionListener.java:452) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1262) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023) ~[elasticsearch-8.17.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-8.17.3.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1575) ~[?:?]
[2025-04-30T18:38:11,742][ERROR][org.elasticsearch.xpack.ilm.IndexLifecycleRunner] policy [so-logs-system.security-logs] for index [.ds-logs-system.security-default-2025.04.29-000105] failed on step [{"phase":"hot","action":"rollover","name":"attempt-rollover"}]. Moving to ERROR step
org.elasticsearch.common.ValidationException: Validation Failed: 1: this action would add [600] shards, but this cluster currently has [1643]/[2000] maximum normal shards open; for more information, see https://www.elastic.co/guide/en/elasticsearch/reference/8.17/size-your-shards.html#troubleshooting-max-shards-open;

I tried temporarily increasing the shards from the Administration > Configuration page under elasticsearch > index_settings > global_overrides > index_template > template > settings > index > number_of_shards, waiting 15 and then trying the action again, but to no avail and I'm not sure it increased the correct shards anyway, as sudo so-elasticsearch-query _cluster/settings shows:

{"persistent":{"cluster":{"remote":{"xxx-so-02":{"seeds":["127.0.0.1:9300"]}},"max_shards_per_node":"2000"}},"transient":{}}[

Also, salt call shows error:

[daven@krc-so-02 elasticsearch]$ sudo salt-call state.highstate
local:
    Data failed to compile:
----------
    The function "state.highstate" is running as PID 353240 and was started at 2025, Apr 30 18:43:31.951025 with jid 20250430184331951025

Any suggestions or clues? If not is there a way i can reinstall without losing all my nsm data and fleet agents? Thanks so much in advance!!!

Guidelines

I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

Answered by dnade

May 2, 2025

Thanks for your response, Chris. regarding the shard increase attempt, I was trying to do something along the lines of this:

https://www.elastic.co/guide/en/elasticsearch/reference/8.17/size-your-shards.html#troubleshooting-max-shards-open

... that did not work. Given your input, I tried reducing the number of shards. I tried adjusting the following:

heap allocation : 30000m -> 31000m
index shards: 600 -> 500
altered rollover times hot=30d, warm=30d, cold=30d, delete=270d

rebooted. Still had the same problem. After more research into the first solution (temporary shard increase) I ran the command:

sudo so-elasticsearch-query _cluster/settings --request PUT --data '{"persistent": {"clust…

View full answer

cm-ops · 2025-05-01T19:12:41Z

cm-ops
May 1, 2025
Maintainer

Elastic does not recommend increasing maximum shards per node.

It looks like you set the amound of shards to 600 and the index cannot rollover becuase you don't have the shard space.

[2025-04-30T18:38:11,696][ERROR][org.elasticsearch.xpack.ilm.IndexLifecycleRunner] policy [so-suricata.alerts-logs] for index [.ds-logs-suricata.alerts-so-2025.04.29-000235] failed on step [{"phase":"hot","action":"rollover","name":"attempt-rollover"}]. Moving to ERROR step
org.elasticsearch.common.ValidationException: Validation Failed: 1: this action would add [600] shards, but this cluster currently has [1643]/[2000] maximum normal shards open; for more information, see https://www.elastic.co/guide/en/elasticsearch/reference/8.17/size-your-shards.html#troubleshooting-max-shards-open;

[2025-04-30T18:38:11,742][ERROR][org.elasticsearch.xpack.ilm.IndexLifecycleRunner] policy [so-logs-system.security-logs] for index [.ds-logs-system.security-default-2025.04.29-000105] failed on step [{"phase":"hot","action":"rollover","name":"attempt-rollover"}]. Moving to ERROR step
org.elasticsearch.common.ValidationException: Validation Failed: 1: this action would add [600] shards, but this cluster currently has [1643]/[2000] maximum normal shards open; for more information, see https://www.elastic.co/guide/en/elasticsearch/reference/8.17/size-your-shards.html#troubleshooting-max-shards-open;

0 replies

dnade · 2025-05-02T17:55:34Z

dnade
May 2, 2025
Author

Thanks for your response, Chris. regarding the shard increase attempt, I was trying to do something along the lines of this:

https://www.elastic.co/guide/en/elasticsearch/reference/8.17/size-your-shards.html#troubleshooting-max-shards-open

... that did not work. Given your input, I tried reducing the number of shards. I tried adjusting the following:

heap allocation : 30000m -> 31000m
index shards: 600 -> 500
altered rollover times hot=30d, warm=30d, cold=30d, delete=270d

rebooted. Still had the same problem. After more research into the first solution (temporary shard increase) I ran the command:

sudo so-elasticsearch-query _cluster/settings --request PUT --data '{"persistent": {"cluster.max_shards_per_node": 2500}}'

... Still had the same issue. I found another setting that related to the other error in my original post (about the too_many_scroll_contexts_exception) and then ran:

sudo so-elasticsearch-query _cluster/settings --request PUT --data '{"persistent": {"search.max_open_scroll_context": 1000}}'

This fixed the issue!

So then I set the number of shards back to 2000 and rebooted. Unfortunately after the reboot, I still have over 2000 shards hanging out. But the original issue is still solved by increasing the number of open scroll contexts. according to the link below, the number of scroll contexts should be equal to the number of shards:

https://discuss.elastic.co/t/max-open-scroll-context-scroll-ids-and-shards/253872

Is there a way for me to safely reduce shards to get back below the 2000 mark?

1 reply

cm-ops May 5, 2025
Maintainer

You have a standalone, for the majority of your indices you only need 1 shard. By default, Zeek will write 2 shards, but if you are not monitoring a lot of traffic, you could also drop that to 1 shard as well. The datastream write index will rollover every 30 days or 50GB by default. Each time the rollover occurs, you would start a new index with 500 shards.

With 500 shards per index, at 2500 shards/node, you would only be able to have 5 indices rollover once before hitting cluster maximum (2500 shards).

dnade · 2025-05-05T15:19:26Z

dnade
May 5, 2025
Author

Thanks again, Chris. So are you saying that in the settings for Configuration > Administration > elasticsearch > index_settings > global_overrides > index_template > template > settings > index > number_of_shards should be 1 or 2? I guess I'm confused about how this setting works, I thought it would create a pool of 500 new shards to be assigned to various indices, not 500 per index! Are there ways to reduce the indexes/shards other than letting them age out of the deletion rollover?

4 replies

cm-ops May 6, 2025
Maintainer

Think of your data this way. An event is processed into a document, a collection of documents make up a shard, one or more shards make up an index. Each index has settings that set certain things when it is created, for example, how many shards. The global overrides settings touch many indices with the same settings, however, you can set individual indices settings to be different.

You would have to chage the index settings and reindex the data in order to reduce the amount of shards. There is also the shrink ILM action which could reduce the amount of primary shards when the data transitions to a different ILM policy phase.

dnade May 6, 2025
Author

Thanks for the explanation, Chris. Very helpful visualization! Last couple of questions and I'll close this out...

1.) You said that all except zeek on a standalone node would probably be fine with a single shard per index. So is a good global_overrides policy to set "1" and to go into the advanced side and ensure zeek is still 2?
2.) With the above index settings changed, I would run a re-index on each index that could use consolidation like below?

[daven@krc-so-02 ~]$ sudo so-elasticsearch-shards-list | grep ".ds-logs-elasticsearch*"
.ds-logs-elasticsearch.server-default-2025.02.23-000011                      0   p STARTED    1278150  155.6mb  155.6mb 192.168.1.19 krc-so-02
.ds-logs-elasticsearch.server-default-2024.09.26-000006                      0   p STARTED     537537  184.5mb  184.5mb 192.168.1.19 krc-so-02
.ds-logs-elasticsearch.server-default-2024.07.28-000004                      0   p STARTED     242411  143.2mb  143.2mb 192.168.1.19 krc-so-02
.ds-logs-elasticsearch.server-default-2025.03.25-000012                      0   p STARTED    2314386  398.1mb  398.1mb 192.168.1.19 krc-so-02
.ds-logs-elasticsearch.server-default-2024.06.28-000003                      0   p STARTED      13891   20.2mb   20.2mb 192.168.1.19 krc-so-02
.ds-logs-elasticsearch.server-default-2025.04.24-000015                      0   p STARTED    3208362  453.4mb  453.4mb 192.168.1.19 krc-so-02
.ds-logs-elasticsearch.server-default-2024.10.26-000007                      0   p STARTED     270581  108.4mb  108.4mb 192.168.1.19 krc-so-02
.ds-logs-elasticsearch.server-default-2025.01.24-000010                      0   p STARTED     648245  224.2mb  224.2mb 192.168.1.19 krc-so-02
.ds-logs-elasticsearch.server-default-2024.12.25-000009                      0   p STARTED     280165  151.8mb  151.8mb 192.168.1.19 krc-so-02
.ds-logs-elasticsearch.server-default-2024.08.27-000005                      0   p STARTED     127571  122.7mb  122.7mb 192.168.1.19 krc-so-02
.ds-logs-elasticsearch.server-default-2024.11.25-000008                      0   p STARTED     126746  124.4mb  124.4mb 192.168.1.19 krc-so-02

cm-ops May 8, 2025
Maintainer

For 1, yes that is usually sufficient. With the datastream the index will rollover to a new one when it reaches 50GB or 30 days by default. For Zeek, if you see your index is not rolling over every day or multiple times a day, one shard might be sufficient as well. If it is rolling over many times a day, you could look at increasing the shards.

For 2, those indices look like they have one shard already, you should not need to do anything with them.

dnade May 9, 2025
Author

Great, thanks so much!!! Really appreciate all the guidance. I'll close this out.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Alerts not acknowledged (Error 400)/Partial shards failure #14581

Uh oh!

{{title}}

Uh oh!

Replies: 3 comments 5 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Alerts not acknowledged (Error 400)/Partial shards failure #14581

Uh oh!

dnade Apr 30, 2025

Version

Installation Method

Description

Installation Type

Location

Hardware Specs

CPU

RAM

Storage for /

Storage for /nsm

Network Traffic Collection

Network Traffic Speeds

Status

Salt Status

Logs

Detail

Guidelines

Replies: 3 comments · 5 replies

Uh oh!

cm-ops May 1, 2025 Maintainer

Uh oh!

Uh oh!

dnade May 2, 2025 Author

Uh oh!

cm-ops May 5, 2025 Maintainer

Uh oh!

dnade May 5, 2025 Author

Uh oh!

cm-ops May 6, 2025 Maintainer

Uh oh!

dnade May 6, 2025 Author

Uh oh!

cm-ops May 8, 2025 Maintainer

Uh oh!

dnade May 9, 2025 Author

dnade
Apr 30, 2025

Replies: 3 comments 5 replies

cm-ops
May 1, 2025
Maintainer

dnade
May 2, 2025
Author

cm-ops May 5, 2025
Maintainer

dnade
May 5, 2025
Author

cm-ops May 6, 2025
Maintainer

dnade May 6, 2025
Author

cm-ops May 8, 2025
Maintainer

dnade May 9, 2025
Author