Distributed 2.4.40 deployment NSM will not go above 53% even though watermarks are set to default 80 85 and 90

[Bal33p]

Version

2.4.40

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

cloud

Hardware Specs

Exceeds minimum requirements

CPU

32

RAM

128

Storage for /

500

Storage for /nsm

13tb

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

more than 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Hello,
I was experiencing the issues referenced below where the manager node was being used as an indexer I lowered the watermarks and followed the guide below to exclude the manager from being able to have logs sent to it. The logs moved off the manager and I put the watermarks back to default.
I have noticed that the Search Node for the last several weeks has not moved above 53%, and I am seeing logs falling off daily.
What should I look for to try and resolve this? As mentioned, all watermarks have been put back to default.

REF:
https://discuss.elastic.co/t/how-to-solve-we-couldnt-log-you-in-please-try-again-error-in-kibana/332658/4

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

What is this setting in the GUI set to? elasticsearch -> retention -> retention_pct If it set to 50%, try and increase it for the search node.