Elasticsearch retention_pct

[argwfm]

Version

2.4.150

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

8

RAM

32

Storage for /

245

Storage for /nsm

512

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

Following upgrade to SOv2.4.150 Elasticsearch ILM does not appear to be honoring the retention_pct setting. It was being honored previously (upgraded from SOv2.4.141). Since upgrading all search nodes are hitting the low/high/flood watermarks, all of which are above the retention_pct setting. We've had to adjust the min_age setting to workaround this to get events flowing again. retention_pct is set to 90%. Not seeing this listed under issues and wanting to raise to see if anyone else is experiencing similar? Thanks all!

[WARN ][org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor] high disk watermark [98%] exceeded on [...][search1][/usr/share/elasticsearch/data] free: 66.9gb[1.6%], shards will be relocated away from this node; currently relocating away shards totalling [-613280123] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[dougburks]

Yes, this is expected in 2.4.150 and is documented here:
https://blog.securityonion.net/2025/05/security-onion-24150-celebrating.html
https://blog.securityonion.net/2025/04/upcoming-change-to-elasticsearch-index.html
https://docs.securityonion.net/en/2.4/elasticsearch.html#index-management

[argwfm]

Thanks much, Doug!

[argwfm]

Follow-up... my grid config is showing index_clean remains enabled. Is this disabled someplace else code wise under the hood? Is there an (unsafe) option to continue honoring retention_pct if we're comfortable w/ that corner case?

[dougburks]

my grid config is showing index_clean remains enabled. Is this disabled someplace else code wise under the hood?

Yes, version 2.4.150 automatically disables this setting under the hood for multi-node deployments.

Is there an (unsafe) option to continue honoring retention_pct if we're comfortable w/ that corner case?

No, it's fundamentally unsafe to run that old script on a multi-node deployment and we want to help you avoid issues. Additionally, ILM provides lots of additional features.

I've created an issue to update the description on the config to help avoid confusion in the future:
#14682

[argwfm]

TY again, Doug. Might be worth calling this out as a breaking change in the RN. It's currently there but as a fix line. I clearly missed the announcements/articles and it might save another someone the same troubles.

[dougburks]

https://docs.securityonion.net/en/2.4/release-notes.html has been updated: