Elastic logs missing long before configured ILM thresholds

[S6T0Sa0B1v]

Version

2.4.201

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

32

RAM

33.2 GB

Storage for /

264.0 GB

Storage for /nsm

67068.1 GB

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

Hello Security Onion,

I am writing in to troubleshoot an issue our organization has recently experienced with our production Security Onion deployment. We are running our new <2 month-old installation on third-party hardware with the following ILM retention settings configured:

Warm: 30d
Cold: 90d
Delete: 365d

We are currently ingesting into Elastic from Entra ID (via an Elastic integration), firewalls (via a different integration), and Elastic Agents on some endpoints, with the intention of having at least one year of historical log data to pull from. so-elasticsearch-indices-delete is configured with a threshold setting of 50, but our storage usage on /nsm is less than 2% of the maximum at time of writing.

The problem we are having is that, despite the ILM settings configured above, we have found that our logs currently go back much less further than intended. Our firewall logs, for instance, seem to disappear in the Dashboards console before 21FEB26, while our Entra ID logs also disappear before 23FEB26. This was unknown to our security team until today, when this issue directly complicated an investigation into a security matter which occurred days before these dates. Therefore, we are trying to determine why the Elastic logs from before these dates do not appear to be recoverable.

Given the description above, where should we begin to look for possible indications of what happened to these logs? Which log files should we begin our troubleshooting process with? We would obviously also like to ensure that this does not happen again in the future, so we would be interested to hear from others who have experienced similar issues if possible.

For additional context our use case is somewhat demanding for the hardware we are currently running, and we have had some sporadic performance issues relating to memory and CPU exhaustion on the machine (33.2 GB of RAM with 8 GB of swap), which rebooting usually seems to fix.

Please let me know if you have questions/need any additional information, and I will reply as soon as I am able. Thank you!

Result of salt-call state.highstate:

local:
Data failed to compile:

The function "state.highstate" is running as PID 1444706 and was started at 2026, Mar 04 17:57:56.688529 with jid 20260304175756688529

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

https://docs.securityonion.net/en/2.4/elasticsearch.html#so-elasticsearch-indices-delete If you are running a standalone this script is probably removing data. Check this log to see - /opt/so/log/elasticsearch/so-elasticsearch-indices-delete.log