Elastic Search Query Fault - High Disk Watermark

[murph146]

Version

2.4.141

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

12

RAM

40

Storage for /

200

Storage for /nsm

200

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

This is a fairily new standard install. All of sudden started getting an "elastic search query fault." Looking at Elastic Search logs i see the following error which i see mentioned in previous versions, but not sure I understand the fix and whether not this is a bug or an issue related to the install. A complete reboot seemed to resolve the error for about 12 hours, but then it showed up again this morning preceded with a High Redis Memory Usage alert.

[2025-04-13T02:25:35,228][WARN ][org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor] high disk watermark [85%] exceeded on [nbr3iUv-TDafv0livRjDBQ][seconion][/usr/share/elasticsearch/data] free: 29.9gb[14.9%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
[2025-04-13T02:26:05,213][INFO ][org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor] high disk watermark [85%] no longer exceeded on [nbr3iUv-TDafv0livRjDBQ][seconion][/usr/share/elasticsearch/data] free: 29.9gb[15%], but low disk watermark [80%] is still exceeded

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

Adjust your Elasticsearch ILM settings to delete sooner. What probably occured is the so-elasticsearch-indices-delete script ran and brought you below the watermark threshold. https://docs.securityonion.net/en/2.4/elasticsearch.html#ilm

Depending on how much data you are monitoring sending, you would need to be aggressive with a 200GB /nsm partition and a Standalone node.

[murph146]

Thank you for the reply regarding the issue I'm having. Is it correct that to use the ILM policies you have to disable index_clean? Root Partition Usage: 22.1% of 213.6 GB NSM Partition Usage: 89% of 214GB Current values set in the config are: Cold – 60d Delete – changed it to 90d in effort to try and resolve the issue, but believe it was 365d by default Warm – 30d The entire config is all vanilla no custom modifications at the moment. When I initially deployed 2.4.x I thought for sure a disk size of 200GB OS and 200GB NSM would be sufficient based on requirements below. Any recommendation on size for standard install would be appreciated. Standalone Deployments•<https://docs.securityonion.net/en/2.4/hardware.html#standalone-deployments> In a standalone deployment, the manager components and the sensor components all run on a single box so your hardware requirements will reflect that. You’ll need at minimum 16GB RAM, 4 CPU cores, and 200GB storage. At the bare minimum of 16GB RAM, you will need swap space to avoid issues. We recommend a minimum of 24GB of RAM if you plan on monitoring even a small amount of network traffic. More network traffic means higher hardware requirements. This deployment type is recommended for evaluation purposes, POCs (proof-of-concept) and small to medium size single sensor deployments. Although you can deploy Security Onion in this manner, it is recommended that you separate the backend components and sensor components. * CPU: Used to parse incoming events, index incoming events, search metatadata, capture PCAP, analyze packets, and run the frontend components. As data and event consumption increases, a greater amount of CPU will be required. * RAM: Used for Logstash<https://docs.securityonion.net/en/2.4/logstash.html#logstash>, Elasticsearch<https://docs.securityonion.net/en/2.4/elasticsearch.html#elasticsearch>, disk cache for Lucene, Suricata<https://docs.securityonion.net/en/2.4/suricata.html#suricata>, Zeek<https://docs.securityonion.net/en/2.4/zeek.html#zeek>, etc. The amount of available RAM will directly impact search speeds and reliability, as well as ability to process and capture traffic. * Disk: Used for storage of indexed metadata. A larger amount of storage allows for a longer retention period. It is typically recommended to retain no more than 30 days of hot Elasticsearch<https://docs.securityonion.net/en/2.4/elasticsearch.html#elasticsearch> indices. ***@***.*** Sent from Outlook<http://aka.ms/weboutlook> From: Chris Morgret ***@***.***> Sent: Monday, April 14, 2025 9:18 AM To: Security-Onion-Solutions/securityonion ***@***.***> Cc: murph146 ***@***.***>; Author ***@***.***> Subject: Re: [Security-Onion-Solutions/securityonion] Elastic Search Query Fault - High Disk Watermark (Discussion #14536) Adjust your Elasticsearch ILM settings to delete sooner. What probably occured is the so-elasticsearch-indices-delete script ran and brought you below the watermark threshold. https://docs.securityonion.net/en/2.4/elasticsearch.html#ilm Depending on how much data you are monitoring sending, you would need to be aggressive with a 200GB /nsm partition and a Standalone node. — Reply to this email directly, view it on GitHub<#14536 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AARTPNHJOS3IRDHTCN7EGV32ZOYR7AVCNFSM6AAAAAB3BGYVYKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEOBSHEZDINA>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

[cm-ops]

There are a couple things to consider. First, how long do you want to retain your data? Second, what is the throughput (average) you are seeing on the monitoring interface? Also, if you are bringing in additional log sources, what is the average size of an index before it rolls over?

[murph146]

At this point, I'd just like to keep things stable and would be happy with 30 days max of retention.

The average throughput on the monitoring interface seems to be around 50 Mb/s or less and I'm not current ingesting any other log sources.

My previous appliance was running 2.3.260 and seemed to be extremely stable and didn't require much maintenance so would like to get the new 2.4.141 appliance there.

Since my initial post i noticed the ElasticSearch has reported a fault again after being up for about 5 days. The Hunt, Cases, Detection all report ElasticSearch has encountered a failure within the cluster.

Thank you in advance!

[cm-ops]

What do you have for indices? sudo so-index-list

[murph146]

Hey Chris, Attached is an output of sudo so-index-list. Let me know if you prefer a different format. One of the variances I see is “red open .ds-logs-suricata.alerts-so-2025.04.27-000058 D1g5xuNBTUSLmC5TgFGxJw 1 0” All others show health is “Green” and status is “Open” Thank you! health status index uuid pri rep docs.count docs.deleted store.size pri.store.size dataset.size green open .ds-.fleet-actions-results-2025.03.12-000001 VC6O8kJCQjWslS01SSQR-Q 1 0 570 0 324.3kb 324.3kb 324.3kb green open .ds-.fleet-actions-results-2025.04.17-000002 9iZGNmxcQ0SROQIQUZV2Yw 1 0 1377 0 426.4kb 426.4kb 426.4kb green open .ds-logs-detections.alerts-so-2025.02.14-000001 JUJSqFBwQZe2QND9kdKfWw 1 0 2 0 126.6kb 126.6kb 126.6kb green open .ds-logs-detections.alerts-so-2025.02.16-000002 k44pVvreT7WdI7TXfJxEEQ 1 0 2 0 123.8kb 123.8kb 123.8kb green open .ds-logs-detections.alerts-so-2025.02.18-000003 oA03-OEBQNOhLliHC34bRA 1 0 1 0 61.3kb 61.3kb 61.3kb green open .ds-logs-detections.alerts-so-2025.03.07-000004 BI4X96zTSzmbjUZeGVsYGQ 1 0 3 0 77.2kb 77.2kb 77.2kb green open .ds-logs-detections.alerts-so-2025.03.12-000005 61QGNQJnSq6MPPB8x852EQ 1 0 4 0 134.4kb 134.4kb 134.4kb green open .ds-logs-detections.alerts-so-2025.03.13-000006 0k38MjkxSgexR6rwGhNxPA 1 0 3 0 77.2kb 77.2kb 77.2kb green open .ds-logs-detections.alerts-so-2025.03.15-000007 8MAc4c1hTjm38jBz3GyTcw 1 0 3 0 77.3kb 77.3kb 77.3kb green open .ds-logs-detections.alerts-so-2025.03.16-000008 lIfBw4E9TsSIxU3Pt-3YNQ 1 0 3 0 77.2kb 77.2kb 77.2kb green open .ds-logs-detections.alerts-so-2025.03.21-000009 TZVRxtduSIONTf1_zCcVmw 1 0 1 0 59.8kb 59.8kb 59.8kb green open .ds-logs-detections.alerts-so-2025.03.22-000010 RDraK45uSgeXfw4GFbtz7w 1 0 1 0 36.7kb 36.7kb 36.7kb green open .ds-logs-detections.alerts-so-2025.03.25-000011 dxxUbUotSUKsip9xpgdh4A 1 0 3 0 81.3kb 81.3kb 81.3kb green open .ds-logs-detections.alerts-so-2025.03.31-000012 c_KFxQImTQeY6TPLjSIPhA 1 0 3 0 81.2kb 81.2kb 81.2kb green open .ds-logs-detections.alerts-so-2025.04.25-000013 NrfFpNXpTnOKF9eiLJZCFQ 1 0 0 0 249b 249b 249b green open .ds-logs-elastic_agent-default-2025.01.10-000002 k7NOa81NQkiNfVHT3kM_6w 1 0 1196 0 918.4kb 918.4kb 918.4kb green open .ds-logs-elastic_agent-default-2025.02.14-000004 SEOsJy0-QJGAvxNtDSi6SA 1 0 174 0 618kb 618kb 618kb green open .ds-logs-elastic_agent-default-2025.02.14-000005 PAeBQqA4S3OJDS6JszdQjw 1 0 8 0 62.2kb 62.2kb 62.2kb green open .ds-logs-elastic_agent-default-2025.02.14-000006 UIUki2weScmIleXWkarrKA 1 0 4827 0 2.4mb 2.4mb 2.4mb green open .ds-logs-elastic_agent-default-2025.03.16-000007 ugCX4C8ZSf-BeJnrr01UqA 1 0 1159 0 572.8kb 572.8kb 572.8kb green open .ds-logs-elastic_agent-default-2025.04.17-000008 QCmIHWXbTxuho-7aOHXvgA 1 0 803 0 221.5kb 221.5kb 221.5kb green open .ds-logs-elastic_agent.filebeat-default-2025.01.11-000001 Wr1NPeThS5mrQQTRqLHOwA 1 0 1298 0 1.1mb 1.1mb 1.1mb green open .ds-logs-elastic_agent.filebeat-default-2025.02.14-000002 T6r9UahVT6umj1vWBohNnA 1 0 98 0 269.4kb 269.4kb 269.4kb green open .ds-logs-elastic_agent.filebeat-default-2025.02.14-000003 yBvVEAsXTPGt7KeC-qmSRg 1 0 7 0 127.9kb 127.9kb 127.9kb green open .ds-logs-elastic_agent.filebeat-default-2025.02.14-000004 gg7tgRUlRZOU9dZQii8Unw 1 0 93657 0 128.1mb 128.1mb 128.1mb green open .ds-logs-elastic_agent.filebeat-default-2025.03.16-000005 2XHL_bFqQZWWsm0aOt0a3Q 1 0 44012 0 59mb 59mb 59mb green open .ds-logs-elastic_agent.filebeat-default-2025.04.17-000006 RmpNgaKFQ7-gDJlfnIpyfA 1 0 29775 0 51mb 51mb 51mb green open .ds-logs-elastic_agent.fleet_server-default-2025.01.10-000002 GJ5sQbtPTH6zxiD0MW2kdg 1 0 2390 0 1.7mb 1.7mb 1.7mb green open .ds-logs-elastic_agent.fleet_server-default-2025.02.14-000004 0DsYsZ_pSXSNGR-Z3xzUHQ 1 0 487 0 566.3kb 566.3kb 566.3kb green open .ds-logs-elastic_agent.fleet_server-default-2025.02.14-000005 bpcuKVtcRrmT4aVdlP_LXQ 1 0 68 0 345kb 345kb 345kb green open .ds-logs-elastic_agent.fleet_server-default-2025.02.14-000006 9dOxkBnoTd6ZQpIy2i4vTg 1 0 667123 0 247.4mb 247.4mb 247.4mb green open .ds-logs-elastic_agent.fleet_server-default-2025.03.16-000007 o30D5OewRfyguRcwhv6WNQ 1 0 421471 0 54.4mb 54.4mb 54.4mb green open .ds-logs-elastic_agent.fleet_server-default-2025.04.17-000008 jr7aDlhTQeC4XGn0v972ag 1 0 232968 0 56.7mb 56.7mb 56.7mb green open .ds-logs-elastic_agent.metricbeat-default-2025.02.14-000002 oA5WUaQ-QJSVax88iD-oGw 1 0 26 0 57.1kb 57.1kb 57.1kb green open .ds-logs-elastic_agent.metricbeat-default-2025.03.16-000004 LVUqEQYpRNCw_HDetWCPcg 1 0 0 0 249b 249b 249b green open .ds-logs-elastic_agent.osquerybeat-default-2025.02.14-000002 UCs29sFuTJ-fC8YUOn_ODQ 1 0 559 0 547kb 547kb 547kb green open .ds-logs-elastic_agent.osquerybeat-default-2025.03.16-000004 m-dp3ql5ROCN_V-7LKPb3Q 1 0 110 0 272.1kb 272.1kb 272.1kb green open .ds-logs-elastic_agent.osquerybeat-default-2025.04.17-000005 zSxN-g_hTVCHL_5je5pijQ 1 0 62 0 136.3kb 136.3kb 136.3kb green open .ds-logs-elasticsearch.server-default-2025.02.14-000002 6RFny4gDTwqbhjczE4dfOA 1 0 41849 0 25mb 25mb 25mb green open .ds-logs-elasticsearch.server-default-2025.03.16-000004 nF2G-4CeRV-V-P3o-smKrA 1 0 9424 0 3.7mb 3.7mb 3.7mb green open .ds-logs-elasticsearch.server-default-2025.04.17-000005 tZfsTKsBQcK5ZBfaXbTIig 1 0 15654 0 3.9mb 3.9mb 3.9mb green open .ds-logs-kratos-so-2025.01.10-000001 fmmPg7-ZTGaYZXTN4x18lw 1 0 562 0 2.3mb 2.3mb 2.3mb green open .ds-logs-kratos-so-2025.02.14-000002 SO1C873XTiyxWJwOQ8xEeQ 1 0 33385 0 75.9mb 75.9mb 75.9mb green open .ds-logs-kratos-so-2025.03.16-000003 34SZA4mMSbyEokpN9OjV6w 1 0 15708 0 36.2mb 36.2mb 36.2mb green open .ds-logs-kratos-so-2025.04.17-000004 ngbkwlTxSau2Fa9i8zhhqw 1 0 13367 0 27.8mb 27.8mb 27.8mb green open .ds-logs-soc-so-2025.01.10-000001 vOzelf7ARQuyEHPgRPoMyA 1 0 145063 0 71.6mb 71.6mb 71.6mb green open .ds-logs-soc-so-2025.02.14-000002 Dzz0_hobRimZ2hi0o2lRvw 1 0 6433646 0 3.5gb 3.5gb 3.5gb green open .ds-logs-soc-so-2025.03.16-000003 uWRJfpLnTS-4o_Bh5qsfQw 1 0 3321792 0 1.8gb 1.8gb 1.8gb green open .ds-logs-soc-so-2025.04.17-000004 tIDoyiRwRR6k4TSHh3tLOA 1 0 2278396 0 1.2gb 1.2gb 1.2gb green open .ds-logs-strelka-so-2025.02.16-000001 q3hAoSy8Q7GtsJWwpJwssw 1 0 732 0 4.9mb 4.9mb 4.9mb green open .ds-logs-strelka-so-2025.03.18-000002 5L3ih22MRq2CCBc3IM6tNQ 1 0 128 0 2.4mb 2.4mb 2.4mb green open .ds-logs-strelka-so-2025.04.17-000003 XDhzlqvJQzGRBdxfUEWpHw 1 0 119 0 1.5mb 1.5mb 1.5mb green open .ds-logs-suricata.alerts-so-2025.01.10-000001 grpd92F2Q2uYtddgvitbaw 1 0 4577 0 11.5mb 11.5mb 11.5mb green open .ds-logs-suricata.alerts-so-2025.02.14-000002 rCEuWsP6Qna3YX6icETFkw 1 0 2395 0 11.4mb 11.4mb 11.4mb green open .ds-logs-suricata.alerts-so-2025.02.14-000003 uP12RIIpQ6e-Tmwi8CMBtw 1 0 809 0 2.3mb 2.3mb 2.3mb green open .ds-logs-suricata.alerts-so-2025.02.16-000004 Tb554PK7TeybH9XPB9wnuQ 1 0 13695 0 33.7mb 33.7mb 33.7mb green open .ds-logs-suricata.alerts-so-2025.02.17-000005 f1XE5HIgRCOgcrR8FDgH1w 1 0 29967 0 44.9mb 44.9mb 44.9mb green open .ds-logs-suricata.alerts-so-2025.02.18-000006 7WyEx8XWQIy4mxX707JPpw 1 0 19196 0 31.9mb 31.9mb 31.9mb green open .ds-logs-suricata.alerts-so-2025.02.19-000007 CwguFjyNQmyK0X1nW5lsSQ 1 0 23453 0 42.6mb 42.6mb 42.6mb green open .ds-logs-suricata.alerts-so-2025.02.20-000008 Uo74MLaSSnW3Nk07FufrHQ 1 0 38652 0 89.1mb 89.1mb 89.1mb green open .ds-logs-suricata.alerts-so-2025.02.21-000009 LjnfkdBwQRq0KqUiU2coiw 1 0 35270 0 57.5mb 57.5mb 57.5mb green open .ds-logs-suricata.alerts-so-2025.02.22-000010 dVlu6_QQT_KGHWmWnp8vxA 1 0 57175 0 89.3mb 89.3mb 89.3mb green open .ds-logs-suricata.alerts-so-2025.02.23-000011 GZE-K2LOSg2Fv1voprSZUQ 1 0 71278 0 116mb 116mb 116mb green open .ds-logs-suricata.alerts-so-2025.02.24-000012 EJAu7quETWCo_fa41N25DQ 1 0 24958 0 39.3mb 39.3mb 39.3mb green open .ds-logs-suricata.alerts-so-2025.02.25-000013 J0EM2H54SaCHF4DlJCwoEg 1 0 16403 0 28.8mb 28.8mb 28.8mb green open .ds-logs-suricata.alerts-so-2025.02.26-000014 UAwLzcSoS4q_Bm2pJbJINw 1 0 40779 0 79.9mb 79.9mb 79.9mb green open .ds-logs-suricata.alerts-so-2025.02.27-000015 UgdSWM6vTSCkvl6JMyr3ZQ 1 0 16848 0 29mb 29mb 29mb green open .ds-logs-suricata.alerts-so-2025.02.28-000016 MhWnzI02S5ORzFXEDjH5ZQ 1 0 28259 0 50.1mb 50.1mb 50.1mb green open .ds-logs-suricata.alerts-so-2025.03.01-000017 K4X5OIZvSEqjcR7K3C6jUA 1 0 26158 0 39.9mb 39.9mb 39.9mb green open .ds-logs-suricata.alerts-so-2025.03.02-000018 _aYsE1r0TDOEfACOv8JzUg 1 0 22080 0 38.4mb 38.4mb 38.4mb green open .ds-logs-suricata.alerts-so-2025.03.03-000019 2_Q2XCZfTYyZGAzcvUGtqg 1 0 33312 0 57.5mb 57.5mb 57.5mb green open .ds-logs-suricata.alerts-so-2025.03.04-000020 7_VZHle-SHOXQC7xPasLZg 1 0 21679 0 38.4mb 38.4mb 38.4mb green open .ds-logs-suricata.alerts-so-2025.03.05-000021 6yI0MAiPQcSA1IhtPKPZkw 1 0 24989 0 60.2mb 60.2mb 60.2mb green open .ds-logs-suricata.alerts-so-2025.03.06-000022 R7HQDggwRreEwxBcwloTxg 1 0 19813 0 31.6mb 31.6mb 31.6mb green open .ds-logs-suricata.alerts-so-2025.03.07-000023 42L7CKTWQhSHhoa6QIpWWw 1 0 14338 0 28.3mb 28.3mb 28.3mb green open .ds-logs-suricata.alerts-so-2025.03.08-000024 S5G1YhrfTYmAlmwwNWIRzw 1 0 11594 0 26.9mb 26.9mb 26.9mb green open .ds-logs-suricata.alerts-so-2025.03.09-000025 tSg5SsLyRxa34s00EKRdtg 1 0 15807 0 26.5mb 26.5mb 26.5mb green open .ds-logs-suricata.alerts-so-2025.03.10-000026 QhB07ZIaQzq_JyjS_xhlmg 1 0 16125 0 36.1mb 36.1mb 36.1mb green open .ds-logs-suricata.alerts-so-2025.03.11-000027 TK_p_snGQ5eqZT1CVLsfIQ 1 0 69684 0 129.7mb 129.7mb 129.7mb green open .ds-logs-suricata.alerts-so-2025.03.12-000028 Y0zB1DczRiygTA0K5rtmiA 1 0 26992 0 56.2mb 56.2mb 56.2mb green open .ds-logs-suricata.alerts-so-2025.03.13-000029 rsECpDfdTYS7LgqYPevbbg 1 0 29522 0 62.8mb 62.8mb 62.8mb green open .ds-logs-suricata.alerts-so-2025.03.14-000030 ZL7RANjVRA6TPeGLJ5XObA 1 0 49810 0 73.3mb 73.3mb 73.3mb green open .ds-logs-suricata.alerts-so-2025.03.15-000031 -v8BzGI7T1OgOpjc5oE5ng 1 0 37729 0 56.7mb 56.7mb 56.7mb green open .ds-logs-suricata.alerts-so-2025.03.16-000032 NEBJYbxXTs-KguB_opPl0Q 1 0 24501 0 67.2mb 67.2mb 67.2mb green open .ds-logs-suricata.alerts-so-2025.03.17-000033 YrSeo7sATt-clTauon-8RQ 1 0 16346 0 39.6mb 39.6mb 39.6mb green open .ds-logs-suricata.alerts-so-2025.03.18-000034 u23_P1tuQHCWWB7rIUb9Vg 1 0 32923 0 55.2mb 55.2mb 55.2mb green open .ds-logs-suricata.alerts-so-2025.03.19-000035 pEMuj48NTimD8wH3a_8UrQ 1 0 18107 0 40.5mb 40.5mb 40.5mb green open .ds-logs-suricata.alerts-so-2025.03.20-000036 4k6hnOKHRoWAdJjensD-dQ 1 0 28061 0 62.9mb 62.9mb 62.9mb green open .ds-logs-suricata.alerts-so-2025.03.21-000037 PrgWY7LVS8SHoN-fMLS6mQ 1 0 24322 0 49.8mb 49.8mb 49.8mb green open .ds-logs-suricata.alerts-so-2025.03.22-000038 m-L_QEQxQgCOj4Tmv5jjow 1 0 14386 0 32.1mb 32.1mb 32.1mb green open .ds-logs-suricata.alerts-so-2025.03.23-000039 SbZJEkwgQ1qyn-66J8dFpw 1 0 16527 0 33mb 33mb 33mb green open .ds-logs-suricata.alerts-so-2025.03.24-000040 JHzrebrFSDSWKSxzHGRDLg 1 0 15555 0 31.9mb 31.9mb 31.9mb green open .ds-logs-suricata.alerts-so-2025.03.25-000041 kU9DvgIsSL6SZ1mTj-b6Jg 1 0 19467 0 38.7mb 38.7mb 38.7mb green open .ds-logs-suricata.alerts-so-2025.03.26-000042 R-pFgsK-TWiYdZcaR1oO9A 1 0 32187 0 54.7mb 54.7mb 54.7mb green open .ds-logs-suricata.alerts-so-2025.03.27-000043 rJqdpK2JSwCWcqCcRsfV6w 1 0 58026 0 75.6mb 75.6mb 75.6mb green open .ds-logs-suricata.alerts-so-2025.03.28-000044 f67AxAmoQOaD4ovVI-rSYw 1 0 58278 0 83.1mb 83.1mb 83.1mb green open .ds-logs-suricata.alerts-so-2025.03.29-000045 wUdec-UjT6urrtAn7hdL1w 1 0 56284 0 95mb 95mb 95mb green open .ds-logs-suricata.alerts-so-2025.03.30-000046 B6pzaxxMT5m9k7bzj2dL3w 1 0 23782 0 65.8mb 65.8mb 65.8mb green open .ds-logs-suricata.alerts-so-2025.03.31-000047 njGDeXvNSnaqOe_CTEMAeg 1 0 4736 0 11.6mb 11.6mb 11.6mb green open .ds-logs-suricata.alerts-so-2025.04.17-000048 BTSjy8RKT1S_wHgLNtwoJA 1 0 39471 0 77mb 77mb 77mb green open .ds-logs-suricata.alerts-so-2025.04.18-000049 o6mlpEGCSnebDwVhJiA0HQ 1 0 22026 0 44.8mb 44.8mb 44.8mb green open .ds-logs-suricata.alerts-so-2025.04.19-000050 zi2bE6oNRq2D7U241ipFUg 1 0 13174 0 34.3mb 34.3mb 34.3mb green open .ds-logs-suricata.alerts-so-2025.04.20-000051 -tvLe7VgSgK4iBdKdZ15LA 1 0 17171 0 32.9mb 32.9mb 32.9mb green open .ds-logs-suricata.alerts-so-2025.04.21-000052 auTF7cK5SE2D4zPrQH16gw 1 0 17338 0 34.2mb 34.2mb 34.2mb green open .ds-logs-suricata.alerts-so-2025.04.22-000053 0JUasKfIROy5yYVkqgv7Lg 1 0 18731 0 41.8mb 41.8mb 41.8mb green open .ds-logs-suricata.alerts-so-2025.04.23-000054 aS-Zoas4TYGC32exFeHn0A 1 0 26652 0 54.9mb 54.9mb 54.9mb green open .ds-logs-suricata.alerts-so-2025.04.24-000055 uRdDA-sCTlGrK6l4h6pz-g 1 0 39491 0 98.1mb 98.1mb 98.1mb green open .ds-logs-suricata.alerts-so-2025.04.25-000056 DkAyLbrfQnGNuidkb-QU-w 1 0 35861 0 67.4mb 67.4mb 67.4mb green open .ds-logs-suricata.alerts-so-2025.04.26-000057 ycUyl8ouQ4qY7ajNWyKNOg 1 0 17020 0 35.4mb 35.4mb 35.4mb red open .ds-logs-suricata.alerts-so-2025.04.27-000058 D1g5xuNBTUSLmC5TgFGxJw 1 0 green open .ds-logs-system.auth-default-2025.02.14-000002 1XE3P0ceSL-1RXm6RxD6fA 1 0 50518 0 34.1mb 34.1mb 34.1mb green open .ds-logs-system.auth-default-2025.03.16-000004 bsVwwZ0WS2SReoiPlkz7bw 1 0 26614 0 19.6mb 19.6mb 19.6mb green open .ds-logs-system.auth-default-2025.04.17-000005 yVRaHnZYSNqa-HgvrIEN-A 1 0 17226 0 14mb 14mb 14mb green open .ds-logs-system.syslog-default-2025.01.10-000001 40e5oERtQ2CHV_SWwE81Nw 1 0 48399 0 11.7mb 11.7mb 11.7mb green open .ds-logs-system.syslog-default-2025.02.14-000002 h2xBmv4uQZucLsCxgjURvw 1 0 13172225 0 2.7gb 2.7gb 2.7gb green open .ds-logs-system.syslog-default-2025.03.16-000003 rxVVkAPjQ5i8Imf6Jg4Ptw 1 0 9945581 0 2gb 2gb 2gb green open .ds-logs-system.syslog-default-2025.04.17-000004 rzfnu1xFQYWQGS7ndLakug 1 0 6713707 0 1.4gb 1.4gb 1.4gb green open .ds-logs-zeek-so-2025.01.10-000001 dWvUASTAQsm2-PcFCsJpOQ 2 0 83926 0 99.1mb 99.1mb 99.1mb green open .ds-logs-zeek-so-2025.02.14-000002 P8fhN2YeQ2qdrDQ1b7qDzw 2 0 26428986 0 27.8gb 27.8gb 27.8gb green open .ds-logs-zeek-so-2025.03.16-000003 mRy9QZfdRRCvYZtjfQYcZg 2 0 13833310 0 14.7gb 14.7gb 14.7gb green open .ds-logs-zeek-so-2025.04.17-000004 RWuwHr9tRBGyVFCTHsrMDw 2 0 9117677 0 9.8gb 9.8gb 9.8gb green open .ds-metrics-fleet_server.agent_status-default-2025.02.14-000001 NVK2fJPyRCK8DQ3RbayKmg 1 0 32775 0 846.9kb 846.9kb 846.9kb green open .ds-metrics-fleet_server.agent_status-default-2025.03.12-000002 AF_IQSIzSRGVaQBLZ3U-ZQ 1 0 28328 0 5.1mb 5.1mb 5.1mb green open .ds-metrics-fleet_server.agent_status-default-2025.04.17-000003 bx06bymGS82mcn_jbAztGw 1 0 21020 0 1.6mb 1.6mb 1.6mb green open .ds-metrics-fleet_server.agent_versions-default-2025.02.14-000001 54OYxpZxS-WpmiwlSikiSA 1 0 32807 0 822.5kb 822.5kb 822.5kb green open .ds-metrics-fleet_server.agent_versions-default-2025.03.12-000002 Ki6hm9l2RniBwILDOOY3Sg 1 0 56656 0 3mb 3mb 3mb green open .ds-metrics-fleet_server.agent_versions-default-2025.04.17-000003 JSY-xHoVQiKHDDXASWK1YQ 1 0 42040 0 2.4mb 2.4mb 2.4mb green open .internal.alerts-default.alerts-default-000001 1zOySlN2TYO8TkMZP9Ok2g 1 0 0 0 249b 249b 249b green open .internal.alerts-ml.anomaly-detection-health.alerts-default-000001 tgw0J-AlRB2EqpwQY-Zsfw 1 0 0 0 249b 249b 249b green open .internal.alerts-ml.anomaly-detection.alerts-default-000001 C0_pa9BhTpeWZNnit_GTVw 1 0 0 0 249b 249b 249b green open .internal.alerts-observability.apm.alerts-default-000001 vKE4KZpNQbmlLcmniO8vgw 1 0 0 0 249b 249b 249b green open .internal.alerts-observability.logs.alerts-default-000001 Wpw3xBtJT428N3RwWqRgiQ 1 0 0 0 249b 249b 249b green open .internal.alerts-observability.metrics.alerts-default-000001 4DlcOGGxSLWighzxIY4vPg 1 0 0 0 249b 249b 249b green open .internal.alerts-observability.slo.alerts-default-000001 7DHvp8yZS7-9ktMLM6--aA 1 0 0 0 249b 249b 249b green open .internal.alerts-observability.threshold.alerts-default-000001 cmTymdOcRLmXgFJDzv9Nfw 1 0 0 0 249b 249b 249b green open .internal.alerts-observability.uptime.alerts-default-000001 AaH8R68gQL6yqNcPZh7Wog 1 0 0 0 249b 249b 249b green open .internal.alerts-security.alerts-default-000001 QepLfdG9QRi8cryj2Fn5LA 1 0 0 0 249b 249b 249b green open .internal.alerts-stack.alerts-default-000001 rcbVCq9JQhSMvuULiXIKsg 1 0 0 0 249b 249b 249b green open .internal.alerts-transform.health.alerts-default-000001 s9rjc3xdQuaxA-2tva8pXg 1 0 0 0 249b 249b 249b green open .kibana-observability-ai-assistant-conversations-000001 xWKhckxaSCuebPjJl_91FQ 1 0 0 0 249b 249b 249b green open .logs-osquery_manager.action.responses-default feG3KU2XTYWX6Ae2iT-BWg 1 0 0 0 249b 249b 249b green open .logs-osquery_manager.actions-default uD9zluMMRUusmQfR9JPk3A 1 0 0 0 249b 249b 249b green open elastalert FE2cweYuTXGrJfjCXsr4mA 1 0 29 0 1mb 1mb 1mb green open elastalert_error wtzm2GT5QgK_S55VSWLodQ 1 0 913080 0 742.4mb 742.4mb 742.4mb green open elastalert_past fDCKDh3aSp-uMpUMPMH8nw 1 0 0 0 249b 249b 249b green open elastalert_silence 0lOxpyejRkG4dnso69aTGA 1 0 0 0 249b 249b 249b green open elastalert_status G1LE7HOaSVOUXKNp6oU6Tg 1 0 1652442 0 282.6mb 282.6mb 282.6mb green open logs-github_latest.dest_code_scanning-1 372fYJ1LQG6YkZPwnxmQqw 1 0 0 0 249b 249b 249b green open logs-github_latest.dest_dependabot-1 GVj45fykSE6p5XrLSN7PRQ 1 0 0 0 249b 249b 249b green open logs-github_latest.dest_issues-1 QHnBQGE9RBeSFs6r9a2IlA 1 0 0 0 249b 249b 249b green open logs-github_latest.dest_secret_scanning-1 tze2IbKWTf-q6jfWLe4zMQ 1 0 0 0 249b 249b 249b green open logs-microsoft_sentinel_latest.dest_incident-1 y57-uYf5SdieeHnUprqe0w 1 0 0 0 249b 249b 249b green open logs-ti_abusech_latest.dest_malware-2 tt2aW6B9Tzea-J3v0GDRFw 1 0 0 0 249b 249b 249b green open logs-ti_abusech_latest.dest_malware-3 a893MIxkSRKQWidrX2cimw 1 0 0 0 249b 249b 249b green open logs-ti_abusech_latest.dest_malwarebazaar-2 oUxIOCMERmmsjdoYJkPLQw 1 0 0 0 249b 249b 249b green open logs-ti_abusech_latest.dest_malwarebazaar-3 drQAYIz4QFWoZ1YKnaP8gg 1 0 0 0 249b 249b 249b green open logs-ti_abusech_latest.dest_threatfox-2 y-uyoEzITgaUYAlOD1POCw 1 0 0 0 249b 249b 249b green open logs-ti_abusech_latest.dest_threatfox-3 XWx9VV2GTei0pnhs_jWW5w 1 0 0 0 249b 249b 249b green open logs-ti_abusech_latest.dest_url-2 gpo2mKi4SVW6D9sPtUKo6w 1 0 0 0 249b 249b 249b green open logs-ti_abusech_latest.dest_url-3 uMp76v6wQEer9_I4L0724w 1 0 0 0 249b 249b 249b green open logs-ti_anomali_latest.intelligence-1 7QMqk3K6TKSgSNpFtuUQVQ 1 0 0 0 249b 249b 249b green open logs-ti_anomali_latest.threatstream-2 rOvEmi5-TSGUVa6KPXyw3w 1 0 0 0 249b 249b 249b green open logs-ti_anomali_latest.threatstream-3 t0IBEuwYRqWXt7iDkB_EdQ 1 0 0 0 249b 249b 249b green open logs-ti_cif3_latest.dest_feed-3 vHTutg1-SXuWHSwzldvhNw 1 0 0 0 249b 249b 249b green open logs-ti_crowdstrike_latest.dest_intel-4 ZaDMuRx7R0ylhgD3U5qHuA 1 0 0 0 249b 249b 249b green open logs-ti_crowdstrike_latest.dest_ioc-5 ViDe8NjBTzODs_4-STwsmg 1 0 0 0 249b 249b 249b green open logs-ti_custom_latest.indicator-3 aak2ZkGdQGqTOh54VEsvkQ 1 0 0 0 249b 249b 249b green open logs-ti_cybersixgill_latest.dest_threat-2 7GP1kLTvQceFvQGtR_Wm-w 1 0 0 0 249b 249b 249b green open logs-ti_cybersixgill_latest.dest_threat-3 7Cspsj1kTXeX13BzBzs5kQ 1 0 0 0 249b 249b 249b green open logs-ti_domaintools_latest.nod_feed-1 LOiNXoxaSTO3E6jztch7CA 1 0 0 0 249b 249b 249b green open logs-ti_eclecticiq_latest.threat-3 slJ2Fj1hR0uOco9R8CEeWg 1 0 0 0 249b 249b 249b green open logs-ti_eset_latest.dest_apt-3 UYhUZcEqTPW3Zj9PUHADVQ 1 0 0 0 249b 249b 249b green open logs-ti_eset_latest.dest_botnet-3 mWm_abqRTHGGvxX0GJVPCg 1 0 0 0 249b 249b 249b green open logs-ti_eset_latest.dest_cc-3 JmucqLyMQWuPPjt6GXMW_Q 1 0 0 0 249b 249b 249b green open logs-ti_eset_latest.dest_domains-3 7QqO2o30Rkq_44Lz3RJwyg 1 0 0 0 249b 249b 249b green open logs-ti_eset_latest.dest_files-3 H3mlhXPgRGqIAMe8Zsf3wg 1 0 0 0 249b 249b 249b green open logs-ti_eset_latest.dest_ip-3 yLR1Vmg6TP6NgY_aprnxgQ 1 0 0 0 249b 249b 249b green open logs-ti_eset_latest.dest_url-3 1n28uvMaR0ilCS9dW-BKYQ 1 0 0 0 249b 249b 249b green open logs-ti_maltiverse_latest.indicator-3 Z-TkqDb3Q4aJVMh0clw_OA 1 0 0 0 249b 249b 249b green open logs-ti_misp_latest.dest_threat_attributes-2 d1e6kL6QRzS0Jdl0ZGvKRg 1 0 0 0 249b 249b 249b green open logs-ti_misp_latest.dest_threat_attributes-3 gXkbL3YQQ3y41wH70uC-Hw 1 0 0 0 249b 249b 249b green open logs-ti_opencti_latest.dest_indicator-2 rtRK1imbQE6MtKeb3pkzeg 1 0 0 0 249b 249b 249b green open logs-ti_opencti_latest.dest_indicator-3 AUng5wPWTBiNNW5aKBlA7Q 1 0 0 0 249b 249b 249b green open logs-ti_otx_latest.dest_pulses_subscribed-1 jDtvOe5aQomzfg-lypc4Mw 1 0 0 0 249b 249b 249b green open logs-ti_otx_latest.dest_pulses_subscribed-2 JQx6T3ZuSiq12_-FPiMX4g 1 0 0 0 249b 249b 249b green open logs-ti_rapid7_threat_command_latest.dest_alert-1 7rD3IDP3TGyfx2PeK9rIyA 1 0 0 0 249b 249b 249b green open logs-ti_rapid7_threat_command_latest.dest_alert-2 H6WCAbGfQ5ePNnbApbmAyw 1 0 0 0 249b 249b 249b green open logs-ti_rapid7_threat_command_latest.dest_ioc-1 Tb-s7sJzQLO_95JwMcjnGA 1 0 0 0 249b 249b 249b green open logs-ti_rapid7_threat_command_latest.dest_ioc-2 8jwc-qJYTcqdvbhVeNEdgQ 1 0 0 0 249b 249b 249b green open logs-ti_rapid7_threat_command_latest.dest_vulnerability-1 XCwiLMiASSWqjOIMHlX5NQ 1 0 0 0 249b 249b 249b green open logs-ti_rapid7_threat_command_latest.dest_vulnerability-2 pgSP1R-qSD6CGoaOx0E38g 1 0 0 0 249b 249b 249b green open logs-ti_recordedfuture_latest.threat-2 AXiOILJbTEqCqbXyRhIKQQ 1 0 0 0 249b 249b 249b green open logs-ti_recordedfuture_latest.threat-3 TyV5thXsR5Od_TojgY3TVg 1 0 0 0 249b 249b 249b green open logs-ti_threatconnect_latest.dest_indicator-3 WoOnEBE3QiGKzWdXuvbCSw 1 0 0 0 249b 249b 249b green open logs-ti_threatq_latest.dest_threat-2 7UUiZ3p8S7Odd7bsw8yJtQ 1 0 0 0 249b 249b 249b green open logs-ti_threatq_latest.dest_threat-3 VYF3eqzwQUSiNbOYv-MV9Q 1 0 0 0 249b 249b 249b green open logs-tychon_latest.dest_arp-1 Gkc8BOY9ROyj-TvInhozfA 1 0 0 0 249b 249b 249b green open logs-tychon_latest.dest_browser-1 IQgTVp2nRPefh80GgnESeg 1 0 0 0 249b 249b 249b green open logs-tychon_latest.dest_ciphers-1 1N5cQNyKSzCUp-_r-ypTGw 1 0 0 0 249b 249b 249b green open logs-tychon_latest.dest_coams-1 QMh2f0Z1RtGm3imLBhEi6g 1 0 0 0 249b 249b 249b green open logs-tychon_latest.dest_cpu-1 OfnECDWFQ7OgypgA3gLXvw 1 0 0 0 249b 249b 249b green open logs-tychon_latest.dest_cve-1 m_79xeHgTEqBcewykXlYgw 1 0 0 0 249b 249b 249b green open logs-tychon_latest.dest_epp-1 yX_Qgyp5Rd2k7ArTw3qxYQ 1 0 0 0 249b 249b 249b green open logs-tychon_latest.dest_exposedservice-1 SzLmivFQR8imCpXXryJ-6w 1 0 0 0 249b 249b 249b green open logs-tychon_latest.dest_externaldevicecontrol-1 aAJWT_oITpGu2UI1bPDJiQ 1 0 0 0 249b 249b 249b green open logs-tychon_latest.dest_features-1 CiINQBewQJ-ISwvyjcjO0g 1 0 0 0 249b 249b 249b green open logs-tychon_latest.dest_harddrive-1 AvoVr-EKRsSebYhJBq8Isw 1 0 0 0 249b 249b 249b green open logs-tychon_latest.dest_hardware-1 Rb_to_emS3SkDKVIMri7hQ 1 0 0 0 249b 249b 249b green open logs-tychon_latest.dest_host-1 RWoRQ5ZATRyklFDTcvNw0w 1 0 0 0 249b 249b 249b green open logs-tychon_latest.dest_networkadapter-1 aFjK6KCnSEmwlfFCiqovuA 1 0 0 0 249b 249b 249b green open logs-tychon_latest.dest_softwareinventory-1 IAMRBMkeQhWWUtklrfu0qw 1 0 0 0 249b 249b 249b green open logs-tychon_latest.dest_stig-1 kIQ7Xs64SUqoUWy4DziLJw 1 0 0 0 249b 249b 249b green open logs-tychon_latest.dest_systemcerts-1 4DSupvemSomxHYVWCW9w5g 1 0 0 0 249b 249b 249b green open logs-tychon_latest.dest_volume-1 9nFkdtUhQyCi513b07v2Lg 1 0 0 0 249b 249b 249b green open metrics-endpoint.metadata_current_default -JdT-PvBTcargEZ23cTDkw 1 0 0 0 249b 249b 249b green open metrics-o365_metrics.groups_activity_group_detail_latest-1 J1v2bBDtQWumGyeqswDVew 1 0 0 0 249b 249b 249b green open metrics-o365_metrics.onedrive_usage_account_detail_latest-1 nTEfMNh0T3ay0BrZJD7fUw 1 0 0 0 249b 249b 249b green open metrics-o365_metrics.teams_user_activity_user_detail_latest-1 t--Q3GmARsCBe0KCX_H_rw 1 0 0 0 249b 249b 249b green open metrics-o365_metrics.viva_engage_groups_activity_group_detail_latest-1 vNb2tuDRTiaQ0bSQZpyOjQ 1 0 0 0 249b 249b 249b green open security_solution-aws.misconfiguration_latest-v1 BjXzBaaTQvWHhY8fGDh5AA 1 0 0 0 249b 249b 249b green open security_solution-wiz.misconfiguration_latest-v1 4hGzky5XTYaM0N9f_Pgr0A 1 0 0 0 249b 249b 249b green open security_solution-wiz.vulnerability_latest-v1 U1im_6GgSa2om1kic36sOg 1 0 0 0 249b 249b 249b green open so-detection SBzv5QhvS520HhHeEZmbUg 1 0 64335 16211 44.8mb 44.8mb 44.8mb green open so-detectionhistory hLBiKnFcSnS6QIC-lOhuTg 1 0 178432 0 112.2mb 112.2mb 112.2mb Sent from Outlook<http://aka.ms/weboutlook> From: Chris Morgret ***@***.***> Sent: Thursday, May 1, 2025 3:50 PM To: Security-Onion-Solutions/securityonion ***@***.***> Cc: murph146 ***@***.***>; Author ***@***.***> Subject: Re: [Security-Onion-Solutions/securityonion] Elastic Search Query Fault - High Disk Watermark (Discussion #14536) What do you have for indices? sudo so-index-list — Reply to this email directly, view it on GitHub<#14536 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AARTPNDM4MGNN7VLKKFMHHT24J3F3AVCNFSM6AAAAAB3BGYVYKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGMBQG4YDIMA>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

[cm-ops]

Those don't look too terrible, how much disk space is PCAP taking up? If PCAP is not as important as the Elastic data, lower the amount you are keeping.

Your Zeek indices are using ~25% of your /nsm partition currently. That might be where the bulk of your index data will be. If you are consistently hitting watermark, you might want to set that to 60 days delete. Alternatively, if Zeek is what you want, set the delete on the other indices lower to save space, they are much smaller, for example, 10s of MBs for Suricata alert indices.

Also, with all those smaller indices/shards, keep in mind in a cluster with one data node, you have a maximum of 1000 shards (active). A way around that is to tune ILM to rollover the index with a longer timeframe, like 60 or 90 days versus 30 days or 50 GB.

[murph146]

Hey Chris,

Below are the full metrics from the Grid. I am not oppose to rebuilding the whole thing to get it back to a usable state. Currently the Hunt, Cases, and Detection will not load at all and I admit I'm still a novice when it comes to all the settings in 2.4 and while I was able to find the global overrides in the settings of Elasticsearch, I'm not sure if those are the values you are recommending I change in your previous response.

It seems like the hard drive space has been the main issue so If i rebuilt it is there a baseline size for the Root and NSM partitions if I'm not ingesting any other type of logging and using all the default settings at least out of the box. I definitely have a lot of space where i can make the Root and NSM partitions much larger.

Currently the Root and NSM drives were provisioned for 200GB.

[cm-ops]

If log retention greater than 60 days is the goal, then having more space on /nsm would be better. If 60 days is sufficient, then you could lower your delete to 60 days and retain just enough of the data to keep you under the watermark setting.

If you see the /nsm partition sitting consistently at 85-87%, you could also adjust the watermark settings and increase the low/high percentages.

[murph146]

Chris,

1.) Can you clarify which settings I should be looking to modify specifically under administration > configuration? My assumption was you're referring to the global overrides, but I'm not positive.

Assuming the following settings for the low and high watermark. Which are currently set at

low -80%
high 85%

elasticsearch > config > cluster > routing > allocation > disk > watermark > low
elasticsearch > config > cluster > routing > allocation > disk > watermark > high

2.) While waiting for a response i increased the NSM partition from 200GB to 300GB and the NSM Usage has dropped to around 40% with Elastic Storage is using 38.9 GB. All all alerts are clear now the appliances is functionality again.

I also set the cold, delete, and warm gobal overrides to 30d using the settings below.

elasticsearch > index_settings > global_overrides > policy > phases > warm > min_age
elasticsearch > index_settings > global_overrides > policy > phases > cold > min_age
elasticsearch > index_settings > global_overrides > policy > phases > warm > min_age

Thank you!

[cm-ops]

Yes, I did mean the global overrides. But if you wanted to change the setting for a specific index, you can use that index's settings to change the policy, that should take precidence.

[murph146]

Got it. I’ve updated the global overrides and increased the NSM drive by another 100GB. Currently sitting around 46% disk usage and all the other errors are now cleared. I’ve also updated to 2.4.150 so will continue to monitor. Thank Chris. Sent from Outlook<http://aka.ms/weboutlook> From: Chris Morgret ***@***.***> Sent: Monday, May 12, 2025 3:49 PM To: Security-Onion-Solutions/securityonion ***@***.***> Cc: murph146 ***@***.***>; Author ***@***.***> Subject: Re: [Security-Onion-Solutions/securityonion] Elastic Search Query Fault - High Disk Watermark (Discussion #14536) Yes, I did mean the global overrides. But if you wanted to change the setting for a specific index, you can use that index's settings to change the policy, that should take precidence. — Reply to this email directly, view it on GitHub<#14536 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AARTPNGD22RTUIW2MMEQ7AD26D3JLAVCNFSM6AAAAAB3BGYVYKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGMJSGI2TAMY>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>