Allow opting out of force-merging on a cloned index in ILM's searchable snapshot action #137375

nielsbauman · 2025-10-30T12:52:07Z

In #133954, we modified ILM's searchable snapshot action to perform the force-merge on a clone of the index with 0 replicas. This optimization avoids performing the force-merge redundantly on replicas, as the subsequent snapshot operation only looks at primary shards.

We've seen some cases where cloning the index resulted in issues; there was a bug in the clone API that caused shards to be initializing permanently under specific circumstances (fixed by #137096), and cloned shards are unable to be assigned if their source lives on a node that is close/past the low watermark disk threshold (will be fixed soon by the Distributed Coordination team).

Therefore, we implement an opt-out flag that users can configure in the searchable_snapshot action of their ILM policy if they don't want to clone the index with 0 replicas before performing the force-merge. We implement an opt-out instead of an opt-in, as we believe these issues to be rather specific (and soon resolved), and the clone is worth doing by default.

…le snapshot action In elastic#133954, we modified ILM's searchable snapshot action to perform the force-merge on a clone of the index with 0 replicas. This optimization avoids performing the force-merge redundantly on replicas, as the subsequent snapshot operation only looks at primary shards. We've seen some cases where cloning the index resulted in issues; there was a bug in the clone API that caused shards to be initializing permanently under specific circumstances (fixed by elastic#137096), and cloned shards are unable to be assigned if their source lives on a node that is close/past the low watermark disk threshold (will be fixed soon by the Distributed Coordination team). Therefore, we implement an opt-out flag that users can configure in the `searchable_snapshot` action of their ILM policy if they don't want to clone the index with 0 replicas before performing the force-merge. We implement an opt-out instead of an opt-in, as we believe these issues to be rather specific (and soon resolved), and the clone is worth doing by default.

elasticsearchmachine · 2025-10-30T12:52:34Z

Pinging @elastic/es-data-management (Team:Data Management)

elasticsearchmachine · 2025-10-30T12:52:58Z

Hi @nielsbauman, I've created a changelog YAML for you.

github-actions · 2025-10-30T12:54:55Z

🔍 Preview links for changed docs

docs/reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md

github-actions · 2025-10-30T12:54:56Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

Copilot

Pull Request Overview

This PR introduces an opt-out mechanism for the force-merge-on-clone optimization in ILM's searchable snapshot action. The optimization, which clones an index with 0 replicas before force-merging to avoid redundant work on replicas, can now be disabled via a new force_merge_on_clone field when issues arise with cloning (such as shard assignment problems near disk watermarks).

Key Changes:

Added a new force_merge_on_clone boolean field to SearchableSnapshotAction that defaults to true (maintaining current behavior)
Modified the step execution logic to conditionally skip clone-related steps when force_merge_on_clone is false
Added validation to ensure force_merge_on_clone is only set when force_merge_index is true

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
SearchableSnapshotAction.java	Core implementation adding the `force_merge_on_clone` field with validation and step execution logic
SearchableSnapshotActionTests.java	Updated unit tests to cover the new field in serialization, mutation, and step key generation
SearchableSnapshotActionIT.java	Added integration test validating behavior when opting out of force-merge-on-clone
RestPutLifecycleAction.java	Added capability flag for the new feature
90_searchable_snapshot.yml	Added YAML REST test verifying the new field can be set and retrieved
ilm-searchable-snapshot.md	Added documentation for the new `force_merge_on_clone` parameter
TimeseriesLifecycleTypeTests.java, LifecyclePolicyTests.java	Updated test utilities to pass null for the new parameter
transport version files	Added new transport version for backward compatibility
137375.yaml	Added changelog entry

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ilm/SearchableSnapshotAction.java

masseyke · 2025-10-30T13:36:12Z

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ilm/SearchableSnapshotAction.java

        }
        this.replicateFor = replicateFor;
+
+        if (forceMergeIndex == false && forceMergeOnClone != null) {


I haven't ready through the whole PR yet so I might be missing something, but what if forceMergeOnClone is false? We don't want to throw an exception here do we? Is there any reason for forceMergeOnClone to be a Boolean rather than a boolean? Is there any real value in it being able to have 3 states instead of 2?

Yeah I was a bit on the fence here. My reason for going with a Boolean was that it better represents what the user actually does (null, false, true). However, we don't currently make a lot of use of that null state, so I don't have super strong feelings on this. If you strongly prefer having a boolean, I'm ok with changing it.

It makes sense with the toXcontent not returning it when null. But is this a bug here that you can't say force_merge_on_clone: false and force_merge_index: false?

Ah sorry, I forgot to comment on that. I intentionally went with that behavior, as I thought it was cleaner and less confusing if force_merge_on_clone is simply not supported when force_merge_index: false. It wouldn't have any effect if you were able to specify it, but I think it's confusing to allow configuring fields that are ignored. This way, it's clear that the force_merge_on_clone field is not relevant/supported. What are your thoughts?

Seems a bit odd to me, but it sort of makes sense.

masseyke · 2025-10-30T13:43:06Z

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ilm/SearchableSnapshotAction.java

        if (replicateFor != null) {
            builder.field(REPLICATE_FOR.getPreferredName(), replicateFor);
        }
+        if (forceMergeOnClone != null) {


Maybe this answers my question from above -- we don't want it in the xcontent if it has not been set explicitly.

docs/reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md

...gin/ilm/qa/rest/src/yamlRestTest/resources/rest-api-spec/test/ilm/90_searchable_snapshot.yml

szybia

lgtm 🚀

masseyke

LGTM

elasticsearchmachine · 2025-10-30T22:09:01Z

💔 Backport failed

Status	Branch	Result
❌	9.2	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 137375

nielsbauman · 2025-10-31T07:08:21Z

💚 All backports created successfully

Status	Branch	Result
✅	9.2

Questions ?

Please refer to the Backport tool documentation

…le snapshot action (elastic#137375) In elastic#133954, we modified ILM's searchable snapshot action to perform the force-merge on a clone of the index with 0 replicas. This optimization avoids performing the force-merge redundantly on replicas, as the subsequent snapshot operation only looks at primary shards. We've seen some cases where cloning the index resulted in issues; there was a bug in the clone API that caused shards to be initializing permanently under specific circumstances (fixed by elastic#137096), and cloned shards are unable to be assigned if their source lives on a node that is close/past the low watermark disk threshold (will be fixed soon by the Distributed Coordination team). Therefore, we implement an opt-out flag that users can configure in the `searchable_snapshot` action of their ILM policy if they don't want to clone the index with 0 replicas before performing the force-merge. We implement an opt-out instead of an opt-in, as we believe these issues to be rather specific (and soon resolved), and the clone is worth doing by default. (cherry picked from commit 0ab3240) # Conflicts: # server/src/main/resources/transport/upper_bounds/9.2.csv # server/src/main/resources/transport/upper_bounds/9.3.csv # x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/action/RestPutLifecycleAction.java

nielsbauman · 2025-10-31T15:07:26Z

💚 All backports created successfully

Status	Branch	Result
✅	9.2

Questions ?

Please refer to the Backport tool documentation

…le snapshot action (elastic#137375) In elastic#133954, we modified ILM's searchable snapshot action to perform the force-merge on a clone of the index with 0 replicas. This optimization avoids performing the force-merge redundantly on replicas, as the subsequent snapshot operation only looks at primary shards. We've seen some cases where cloning the index resulted in issues; there was a bug in the clone API that caused shards to be initializing permanently under specific circumstances (fixed by elastic#137096), and cloned shards are unable to be assigned if their source lives on a node that is close/past the low watermark disk threshold (will be fixed soon by the Distributed Coordination team). Therefore, we implement an opt-out flag that users can configure in the `searchable_snapshot` action of their ILM policy if they don't want to clone the index with 0 replicas before performing the force-merge. We implement an opt-out instead of an opt-in, as we believe these issues to be rather specific (and soon resolved), and the clone is worth doing by default. (cherry picked from commit 0ab3240) # Conflicts: # server/src/main/resources/transport/upper_bounds/9.3.csv # x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/action/RestPutLifecycleAction.java

…archable snapshot action (#137375) (#137463) * Allow opting out of force-merging on a cloned index in ILM's searchable snapshot action (#137375) In #133954, we modified ILM's searchable snapshot action to perform the force-merge on a clone of the index with 0 replicas. This optimization avoids performing the force-merge redundantly on replicas, as the subsequent snapshot operation only looks at primary shards. We've seen some cases where cloning the index resulted in issues; there was a bug in the clone API that caused shards to be initializing permanently under specific circumstances (fixed by #137096), and cloned shards are unable to be assigned if their source lives on a node that is close/past the low watermark disk threshold (will be fixed soon by the Distributed Coordination team). Therefore, we implement an opt-out flag that users can configure in the `searchable_snapshot` action of their ILM policy if they don't want to clone the index with 0 replicas before performing the force-merge. We implement an opt-out instead of an opt-in, as we believe these issues to be rather specific (and soon resolved), and the clone is worth doing by default. (cherry picked from commit 0ab3240) # Conflicts: # server/src/main/resources/transport/upper_bounds/9.3.csv # x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/action/RestPutLifecycleAction.java * Fix transport version

nielsbauman added >enhancement :Data Management/ILM+SLM Index and Snapshot lifecycle management auto-backport Automatically create backport pull requests when merged branch:9.2 labels Oct 30, 2025

elasticsearchmachine added Team:Data Management Meta label for data/management team v9.3.0 v9.2.1 labels Oct 30, 2025

elasticsearchmachine removed the branch:9.2 label Oct 30, 2025

Update docs/changelog/137375.yaml

91f74ae

[CI] Auto commit changes from spotless

9157717

nielsbauman requested a review from Copilot October 30, 2025 12:58

Add applies_to tag to docs

1c9386d

Copilot AI reviewed Oct 30, 2025

View reviewed changes

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ilm/SearchableSnapshotAction.java Outdated Show resolved Hide resolved

nielsbauman and others added 2 commits October 30, 2025 14:01

Apply copilot suggestion

ab5a0f5

[CI] Update transport version definitions

0c1188d

masseyke reviewed Oct 30, 2025

View reviewed changes

szybia reviewed Oct 30, 2025

View reviewed changes

docs/reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md Outdated Show resolved Hide resolved

...gin/ilm/qa/rest/src/yamlRestTest/resources/rest-api-spec/test/ilm/90_searchable_snapshot.yml Outdated Show resolved Hide resolved

nielsbauman and others added 3 commits October 30, 2025 16:54

PR feedback

66f7f6b

Merge branch 'main' into allow-opt-out-ilm

6f5081c

[CI] Update transport version definitions

605795f

szybia approved these changes Oct 30, 2025

View reviewed changes

masseyke approved these changes Oct 30, 2025

View reviewed changes

nielsbauman added 2 commits October 30, 2025 21:31

Merge branch 'main' into allow-opt-out-ilm

811dabe

./gradlew generateTransportVersion

3521a16

Fix transport versions

77a79ae

nielsbauman enabled auto-merge (squash) October 30, 2025 20:36

nielsbauman merged commit 0ab3240 into elastic:main Oct 30, 2025
34 checks passed

nielsbauman deleted the allow-opt-out-ilm branch October 30, 2025 22:07

elasticsearchmachine added the backport pending label Oct 30, 2025

nielsbauman mentioned this pull request Oct 31, 2025

[9.2] Allow opting out of force-merging on a cloned index in ILM's searchable snapshot action (#137375) #137420

Closed

nielsbauman removed the backport pending label Oct 31, 2025

nielsbauman mentioned this pull request Oct 31, 2025

[9.2] Allow opting out of force-merging on a cloned index in ILM's searchable snapshot action (#137375) #137463

Merged

Allow opting out of force-merging on a cloned index in ILM's searchable snapshot action #137375

Allow opting out of force-merging on a cloned index in ILM's searchable snapshot action #137375

Conversation

nielsbauman commented Oct 30, 2025

Uh oh!

elasticsearchmachine commented Oct 30, 2025

Uh oh!

elasticsearchmachine commented Oct 30, 2025

Uh oh!

github-actions bot commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

github-actions bot commented Oct 30, 2025

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

masseyke Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

nielsbauman Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

masseyke Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

nielsbauman Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

masseyke Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

masseyke Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

szybia left a comment

Choose a reason for hiding this comment

Uh oh!

masseyke left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Oct 30, 2025

💔 Backport failed

Uh oh!

nielsbauman commented Oct 31, 2025

💚 All backports created successfully

Questions ?

Uh oh!

nielsbauman commented Oct 31, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions bot commented Oct 30, 2025 •

edited

Loading