Alerts disappearing and elasticsearch not closing/deleting older indices

[chergett]

Version

2.4.50

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

40

RAM

384Gb

Storage for /

222Gb

Storage for /nsm

8.8Tb

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

The behavior I'm seeing is that alerts show up but disappear after a while. It seems to have something to do with elasticsearch running out of disk space based on the below log messages. This server is doing the same thing it was doing when I was running on version 2.3, but 2.3 never behaved this way. I think that something is not cleaning up after itself in elasticsearch but I don't really know what to check. Looking at the indices, etc. in Kibana, it seems that older indices are not being removed as the disk space fills up.

so-elasticsearch-indices-delete.log contains many "Used disk space exceeds LOG_SIZE_LIMIT (7372 GB) - There is only one backing index" entries.
logstash log contains many "[WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch." entries.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

These are the two ways Elasticsearch data gets deleted - https://docs.securityonion.net/en/2.4/elasticsearch.html#deleting-indices

Can you check your Suricata data stream? sudo so-elasticsearch-query _cat/indices | grep suricata It might not be hitting the 50GB max to rollover and is getting removed. If that is the case, change the rollover in SOC Administration > Configuration > elasticsearch > index_settings > so-suricata > policy > phases > hot > rollover. Defaults are 30 days or 50 GB. Try dropping the days to what time period you want it to rollover.

[chergett]

I think my problem is the reverse of what you suggest. I have a very high EPS (18k) so I'm easily reaching the 50Gb max limit. I think perhaps I'm simply running out of disk space faster than the policies are set to delete an index. With this information, I think I'll have to either add more storage space or reduce the amount of time an index is kept. Of course, I could also try to trim down the EPS to a lower number to toss out some additional traffic that I may not need to inspect. Does it sound like I'm thinking correctly here?

[cm-ops]

That could be, how many shards are you writing for your indices? You could be ingesting a lot of data, rolling over frequently, but still only retaining a few days of indices. If you run sudo so-elasticsearch-query _cat/indices | sort that should give you an idea of your days in ES.

How much tuning in Zeek/Suricata have you done?

[chergett]

I have tuned Suricata pretty heavily since I began using it. Attached is the output of the sudo so-elasticsearch-query _cat/indices | sort command. Perhaps something will jump out at you if I have things configured incorrectly. Some of the indices go back to 2023. Suricata seems to have about 7 days worth at the moment if I'm reading it correctly. I'm also seeing quite a few .ds-logs-system.security-default* indices but I don't know what is stored in them.
[root@seconion log]# so-elasticsear.txt

[cm-ops]

What does this show? sudo so-elasticsearch-ilm-policy-view | grep -A40 so-suricata-logs

[chergett]

See attached for output of sudo so-elasticsearch-ilm-policy-view | grep -A40 so-suricata-logs
output.txt

[cm-ops]

Do you have global_overrides set to anything? Looking at your indices, your indices are over the 50gb max_primary_shard_size set in ILM policy. Can you also share your /opt/so/log/elasticsearch/cron-elasticsearch-indices-delete.log? I want to see what is getting deleted by the sized based deletion.

To confirm, you are running Suricata as metadata?