Adding configurable resiliency features to MergedSegmentWarmer #19629

kh3ra · 2025-10-14T13:12:06Z

Description

[Describe what this change achieves]

Standardized merged_segment_warmer related config names.
Added a threshold property that controls which merged segments get pre-copied (warmed), ensuring only segments larger than the specified size are processed.
Added a multiplier to allow fine grained control over indexing throttling caused by ongoing merges.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2025-10-14T13:59:27Z

❌ Gradle check result for 3aeef75: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

ashking94 · 2025-10-14T15:59:31Z

server/src/main/java/org/opensearch/index/engine/MergedSegmentWarmer.java

+        if (segmentSize < indexShard.getRecoverySettings().getMergedSegmentWarmerSegmentSizeThreshold().getBytes()) {
+            return false;
+        }
+
+        return true;


this can be simplified to :

return segmentSize >= indexShard.getRecoverySettings().getMergedSegmentWarmerSegmentSizeThreshold().getBytes();

server/src/main/java/org/opensearch/indices/recovery/RecoverySettings.java

server/src/main/java/org/opensearch/index/engine/InternalEngine.java

ashking94 · 2025-10-14T16:16:51Z

server/src/main/java/org/opensearch/index/engine/MergedSegmentWarmer.java

+        if (segmentSize < indexShard.getRecoverySettings().getMergedSegmentWarmerSegmentSizeThreshold().getBytes()) {
+            return false;
+        }
+


lets add some trace/debug logs on the final return with diagnostic data

github-actions · 2025-10-15T00:29:48Z

❌ Gradle check result for f0feabe: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-10-15T09:10:45Z

❌ Gradle check result for bbd9dd4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Added a threshold property that controls which merged segments get pre-copied (warmed), ensuring only segments larger than the specified size are processed. Added cluster defaults for max_merge_count and max_merge_threads Signed-off-by: Aditya Khera <[email protected]>

github-actions · 2025-10-15T10:19:23Z

❌ Gradle check result for 2459597: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Aditya Khera <[email protected]>

github-actions · 2025-10-15T15:26:30Z

❌ Gradle check result for 36d0a37: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Aditya Khera <[email protected]>

github-actions · 2025-10-15T16:42:28Z

❌ Gradle check result for 4aec3d9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Aditya Khera <[email protected]>

github-actions · 2025-10-15T17:09:51Z

❌ Gradle check result for b06e2ac: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Aditya Khera <[email protected]>

github-actions · 2025-10-16T07:15:18Z

❌ Gradle check result for cd48be6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Aditya Khera <[email protected]>

github-actions · 2025-10-17T10:37:08Z

❌ Gradle check result for 2ce4ef8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Aditya Khera <[email protected]>

github-actions · 2025-10-17T11:23:39Z

❌ Gradle check result for 87cad2d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-10-17T13:21:20Z

❌ Gradle check result for f8cf13c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Aditya Khera <[email protected]>

github-actions · 2025-10-17T14:09:04Z

❌ Gradle check result for 2f943c3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Bukhtawar · 2025-10-18T09:24:58Z

server/src/main/java/org/opensearch/indices/IndicesService.java

+    public static final Setting<Integer> CLUSTER_DEFAULT_MAX_THREAD_COUNT_SETTING = new Setting<>(
+        "cluster.default.index.merge.scheduler.max_thread_count",
+        (s) -> Integer.toString(Math.max(1, Math.min(4, OpenSearchExecutors.allocatedProcessors(s) / 2))),
+        (s) -> Setting.parseInt(s, 1, "cluster.default.index.merge.scheduler.max_thread_count"),
+        Property.Dynamic,
+        Property.NodeScope
+    );


nit: Maybe good to remove default in the settings name given it can be overridden, dynamically?

Bukhtawar · 2025-10-18T09:26:51Z

server/src/main/java/org/opensearch/indices/recovery/RecoverySettings.java

+    /**
+     * Dynamic setting to set a threshold for minimum size of a merged segment to be warmed.
+     */
+    public static final Setting<ByteSizeValue> INDICES_REPLICATION_MERGES_WARMER_SEGMENT_SIZE_THRESHOLD_SETTING = Setting.byteSizeSetting(
+        "indices.replication.merges.warmer.segment_size_threshold",
+        new ByteSizeValue(500, ByteSizeUnit.MB),
+        Property.Dynamic,
+        Property.NodeScope
+    );


nit: The name doesn't reflect a minimum threshold size?

kh3ra requested review from a team, Bukhtawar, CEHENKLE, Rishikesh1159, anasalkouz, andrross, ashking94, cwperks, dbwiddis, gbbafna, jed326, kotwanikunal, mch2, msfroh, owaiskazi19, reta, sachinpkale, saratvemulapalli, shwetathareja and sohami as code owners October 14, 2025 13:12

ashking94 requested changes Oct 14, 2025

View reviewed changes

kh3ra force-pushed the dev/merged-segment-warmer-configs branch from f0feabe to 34610d9 Compare October 15, 2025 08:49

kh3ra force-pushed the dev/merged-segment-warmer-configs branch from bbd9dd4 to 2459597 Compare October 15, 2025 10:08

Adding MaxMergeCount and MergeThreadCount settings

36d0a37

Signed-off-by: Aditya Khera <[email protected]>

Bug fixes

4aec3d9

Signed-off-by: Aditya Khera <[email protected]>

spotless apply

b06e2ac

Signed-off-by: Aditya Khera <[email protected]>

Bug fixes

cd48be6

Signed-off-by: Aditya Khera <[email protected]>

opensearch-ci-bot mentioned this pull request Oct 17, 2025

[AUTOCUT] Gradle Check Flaky Test Report for InstallPluginCommandTests #19256

Open

Bug fixes, addressing review comments, adding tests

2ce4ef8

Signed-off-by: Aditya Khera <[email protected]>

spotless apply

87cad2d

Signed-off-by: Aditya Khera <[email protected]>

spotless apply

2f943c3

Signed-off-by: Aditya Khera <[email protected]>

kh3ra force-pushed the dev/merged-segment-warmer-configs branch from f8cf13c to 2f943c3 Compare October 17, 2025 13:55

Bukhtawar reviewed Oct 18, 2025

View reviewed changes

Adding configurable resiliency features to MergedSegmentWarmer #19629

Are you sure you want to change the base?

Adding configurable resiliency features to MergedSegmentWarmer #19629

Uh oh!

Conversation

kh3ra commented Oct 14, 2025

Description

Related Issues

Check List

Uh oh!

github-actions bot commented Oct 14, 2025

Uh oh!

ashking94 Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ashking94 Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 15, 2025

Uh oh!

github-actions bot commented Oct 15, 2025

Uh oh!

github-actions bot commented Oct 15, 2025

Uh oh!

github-actions bot commented Oct 15, 2025

Uh oh!

github-actions bot commented Oct 15, 2025

Uh oh!

github-actions bot commented Oct 15, 2025

Uh oh!

github-actions bot commented Oct 16, 2025

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

Bukhtawar Oct 18, 2025

Choose a reason for hiding this comment

Uh oh!

Bukhtawar Oct 18, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants