Dynamic shard sizing based on search nodes operating system memory

[AntonEbel]

Description

This PR implements dynamic shard sizing for the index rotation. TIME_SIZE_OPTIMIZING_ROTATION_MIN_SHARD_SIZE and TIME_SIZE_OPTIMIZING_ROTATION_MAX_SHARD_SIZE no longer have default values. If either of these configuration fields is not set, dynamic shard sizing is enabled. This feature uses the operating system memory of Opensearch/Elasticsearch nodes with the data role to calculate the maximum index shard size. For this purpose, the node with the smallest memory is used.

Motivation and Context

closes #23947 (Pitch)

How Has This Been Tested?

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Refactoring (non-breaking change)
• Breaking change (fix or feature that would cause existing functionality to change)

/prd Graylog2/graylog-plugin-enterprise#12643

[patrickmann]

Codex noticed this:
ShardsMetricsSupplier.java:43-50 calls getTimeSizeOptimizingRotationMin/MaxShardSize().getQuantity() unconditionally. These settings default to null for the new dynamic sizing mode, so the telemetry job will throw an NPE before emitting metrics (and ShardsMetricsSupplierTest still assumes non-null sizes). Needs null guards/fallbacks.

[boosty]

@AntonEbel @patrickmann Is the memory calculation container aware? E.g. when running OpenSearch as a pod in Kubernetes and setting memory limits?

[AntonEbel]

@boosty We currently use the following value nodes.os.mem.total_in_bytes from this API call /_nodes/stats/os. As far as I know, it is not container aware. Opensearch has the following open issue to support cgroup v2. Once this issue is resolved, we can make it container aware.

[boosty]

@AntonEbel Thanks for the quick feedback! That is a very important piece of information. I think it's not uncommon to deploy DataNode (or raw OpenSearch) as containers these days. The new official Graylog Helm chart will probably accelerate this.

@tellistone pinging you for awareness.