Indices on manager issue

[mckenziemmack]

Version

2.4.100

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

64

RAM

251G

Storage for /

558G

Storage for /nsm

38T

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

Hello,

I have been having issues with logs being stored on my manager, causing my manager storage to fill up quickly even though there is ample room on my two storage nodes. Once the storage threshold is reached on the manager, the shards do not show up in Kibana, or via sudo so-elasticsearch-indices-list as if they’ve been deleted. This is a significant issue for us.

These are some of the logs I'm seeing under /opt/so/log/elasticsearch:

[2024-11-01T04:55:45,584][INFO ][org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor] high disk watermark [80%] exceeded on [WwR9Zt1HRDO3_DpoFsHktA][siemmanager][/usr/share/elasticsearch/data] free: 104.6gb[19.9%], shards will be relocated away from this node; currently relocating away shards totalling [13928445110] bytes; the node is expected to be below the high disk watermark when these relocations are complete

[2024-11-01T04:58:14,825][INFO ][org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor] high disk watermark [80%] no longer exceeded on [WwR9Zt1HRDO3_DpoFsHktA][siemmanager][/usr/share/elasticsearch/data] free: 105.1gb[20%], but low disk watermark [75%] is still exceeded

There's probably a configuration change that can help fix this allocation issue. Is it possible to change just my managers watermark setting to 45% low, 50% high, and 90% flood and keep my storage nodes at 75% low, 80% high, 90% flood?

Thank you for your help!

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[rh-ops]

You can use the following command on the manager CLI to migrate data off your manager onto your other nodes.

so-elasticsearch-query _cluster/settings -d '{"transient": {"cluster.routing.allocation.exclude._ip": "manager_ip" } }' -XPUT

When this finishes moving over the shards, then you can log into your SOC and navigate to Administration > Configuration > (enable advanced options at the top of the screen) > elasticsearch > so_roles > so_manager > config > node > roles

And from there you can remove the data role from the manager so it wont ingest more data. After you remove the data role, click the green checkmark to save it and then click "Synchronize Grid" in the options drop down menu. This should resolve the issue of data filling up your manager.

[mckenziemmack]

Thank you for your response! The query worked but my manager is not happy when I delete the data role in SOC. Deleting "data" from that configuration put my manager info fault and so-elasticsearch is missing.

[rh-ops]

Might be an issue with data still being on the manager. Lets check to see where all the indices are. What is the output of this command?

So-elasticsearch-query _cat/allocation?v