Fix mapping conflicts in clone/split/shrink APIs #137096

nielsbauman · 2025-10-24T11:33:58Z

If an index is in either logsdb or time_series mode and specifies a non-default @timestamp type mapping (e.g. date_nanos), using the clone, split, or shrink API will result in shards that are unable to initialize/recover due to the following mapping conflict:

Caused by: java.lang.IllegalArgumentException: mapper [@timestamp] cannot be changed from type [date] to [date_nanos]
	at org.elasticsearch.index.mapper.FieldMapper.checkIncomingMergeType(FieldMapper.java:419)
	at org.elasticsearch.index.mapper.FieldMapper.merge(FieldMapper.java:399)
	at org.elasticsearch.index.mapper.FieldMapper.merge(FieldMapper.java:63)
	at org.elasticsearch.index.mapper.ObjectMapper$MergeResult.buildMergedMappers(ObjectMapper.java:738)
	at org.elasticsearch.index.mapper.ObjectMapper$MergeResult.build(ObjectMapper.java:664)
	at org.elasticsearch.index.mapper.RootObjectMapper.merge(RootObjectMapper.java:249)
	at org.elasticsearch.index.mapper.Mapping.merge(Mapping.java:173)
	at org.elasticsearch.index.mapper.MapperService.mergeMappings(MapperService.java:684)
	at org.elasticsearch.index.mapper.MapperService.mergeMappings(MapperService.java:651)
	at org.elasticsearch.index.mapper.MapperService.doMerge(MapperService.java:599)
	at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:584)
	at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:436)
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$5(StoreRecovery.java:131)

As of #133954, the searchable_snapshot ILM action makes use of the clone API by default - if the index has more than 0 replicas - and will thus also run into this issue.

elasticsearchmachine · 2025-10-24T11:34:23Z

Pinging @elastic/es-data-management (Team:Data Management)

elasticsearchmachine · 2025-10-24T11:34:47Z

Hi @nielsbauman, I've created a changelog YAML for you.

nielsbauman · 2025-10-24T13:08:01Z

docs/changelog/137096.yaml

+summary: Fix mapping conflicts in clone/split/shrink APIs
+area: Indices APIs
+type: bug
+issues: []


Should we make changelog more elaborate/descriptive? cc @mattc58

Yes, and probably needs to be a release highlight since this might warrant a special release. But that can wait until we determine the impact and precise triggering conditions of this.

Alright, then I'm merging this as-is. We can come back and update the changelog later if we want to.

masseyke

LGTM

masseyke · 2025-10-24T13:56:34Z

Niels shared this outside of the ticket but it was very useful. The following will crash a server without this fix if assertions are enabled:

PUT my-source
{
    "settings": {
        "index.mode": "logsdb",
        "index.blocks.write": true,
        "index.number_of_replicas": 1
    },
    "mappings": {
        "properties": {
            "@timestamp": {
                "type": "date_nanos"
            }
        }
    }
}


POST my-source/_clone/my-target?timeout=5s

henningandersen

LGTM.

...src/internalClusterTest/java/org/elasticsearch/action/admin/indices/create/CloneIndexIT.java

If an index is in either `logsdb` or `time_series` mode and specifies a non-default `@timestamp` type mapping (e.g. `date_nanos`), using the clone, split, or shrink API will result in shards that are unable to initialize/recover due to a mapping conflict. As of elastic#133954, the `searchable_snapshot` ILM action makes use of the clone API by default - if the index has more than `0` replicas - and will thus also run into this issue.

elasticsearchmachine · 2025-10-24T15:48:46Z

💚 Backport successful

Status	Branch	Result
✅	9.2
✅	8.19

If an index is in either `logsdb` or `time_series` mode and specifies a non-default `@timestamp` type mapping (e.g. `date_nanos`), using the clone, split, or shrink API will result in shards that are unable to initialize/recover due to a mapping conflict. As of #133954, the `searchable_snapshot` ILM action makes use of the clone API by default - if the index has more than `0` replicas - and will thus also run into this issue.

If an index is in either `logsdb` or `time_series` mode and specifies a non-default `@timestamp` type mapping (e.g. `date_nanos`), using the clone, split, or shrink API will result in shards that are unable to initialize/recover due to a mapping conflict. As of elastic#133954, the `searchable_snapshot` ILM action makes use of the clone API by default - if the index has more than `0` replicas - and will thus also run into this issue.

If an index is in either `logsdb` or `time_series` mode and specifies a non-default `@timestamp` type mapping (e.g. `date_nanos`), using the clone, split, or shrink API will result in shards that are unable to initialize/recover due to a mapping conflict. As of #133954, the `searchable_snapshot` ILM action makes use of the clone API by default - if the index has more than `0` replicas - and will thus also run into this issue.

…le snapshot action In elastic#133954, we modified ILM's searchable snapshot action to perform the force-merge on a clone of the index with 0 replicas. This optimization avoids performing the force-merge redundantly on replicas, as the subsequent snapshot operation only looks at primary shards. We've seen some cases where cloning the index resulted in issues; there was a bug in the clone API that caused shards to be initializing permanently under specific circumstances (fixed by elastic#137096), and cloned shards are unable to be assigned if their source lives on a node that is close/past the low watermark disk threshold (will be fixed soon by the Distributed Coordination team). Therefore, we implement an opt-out flag that users can configure in the `searchable_snapshot` action of their ILM policy if they don't want to clone the index with 0 replicas before performing the force-merge. We implement an opt-out instead of an opt-in, as we believe these issues to be rather specific (and soon resolved), and the clone is worth doing by default.

…le snapshot action (#137375) In #133954, we modified ILM's searchable snapshot action to perform the force-merge on a clone of the index with 0 replicas. This optimization avoids performing the force-merge redundantly on replicas, as the subsequent snapshot operation only looks at primary shards. We've seen some cases where cloning the index resulted in issues; there was a bug in the clone API that caused shards to be initializing permanently under specific circumstances (fixed by #137096), and cloned shards are unable to be assigned if their source lives on a node that is close/past the low watermark disk threshold (will be fixed soon by the Distributed Coordination team). Therefore, we implement an opt-out flag that users can configure in the `searchable_snapshot` action of their ILM policy if they don't want to clone the index with 0 replicas before performing the force-merge. We implement an opt-out instead of an opt-in, as we believe these issues to be rather specific (and soon resolved), and the clone is worth doing by default.

…le snapshot action (elastic#137375) In elastic#133954, we modified ILM's searchable snapshot action to perform the force-merge on a clone of the index with 0 replicas. This optimization avoids performing the force-merge redundantly on replicas, as the subsequent snapshot operation only looks at primary shards. We've seen some cases where cloning the index resulted in issues; there was a bug in the clone API that caused shards to be initializing permanently under specific circumstances (fixed by elastic#137096), and cloned shards are unable to be assigned if their source lives on a node that is close/past the low watermark disk threshold (will be fixed soon by the Distributed Coordination team). Therefore, we implement an opt-out flag that users can configure in the `searchable_snapshot` action of their ILM policy if they don't want to clone the index with 0 replicas before performing the force-merge. We implement an opt-out instead of an opt-in, as we believe these issues to be rather specific (and soon resolved), and the clone is worth doing by default. (cherry picked from commit 0ab3240) # Conflicts: # server/src/main/resources/transport/upper_bounds/9.2.csv # server/src/main/resources/transport/upper_bounds/9.3.csv # x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/action/RestPutLifecycleAction.java

…le snapshot action (elastic#137375) In elastic#133954, we modified ILM's searchable snapshot action to perform the force-merge on a clone of the index with 0 replicas. This optimization avoids performing the force-merge redundantly on replicas, as the subsequent snapshot operation only looks at primary shards. We've seen some cases where cloning the index resulted in issues; there was a bug in the clone API that caused shards to be initializing permanently under specific circumstances (fixed by elastic#137096), and cloned shards are unable to be assigned if their source lives on a node that is close/past the low watermark disk threshold (will be fixed soon by the Distributed Coordination team). Therefore, we implement an opt-out flag that users can configure in the `searchable_snapshot` action of their ILM policy if they don't want to clone the index with 0 replicas before performing the force-merge. We implement an opt-out instead of an opt-in, as we believe these issues to be rather specific (and soon resolved), and the clone is worth doing by default. (cherry picked from commit 0ab3240) # Conflicts: # server/src/main/resources/transport/upper_bounds/9.3.csv # x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/action/RestPutLifecycleAction.java

…archable snapshot action (#137375) (#137463) * Allow opting out of force-merging on a cloned index in ILM's searchable snapshot action (#137375) In #133954, we modified ILM's searchable snapshot action to perform the force-merge on a clone of the index with 0 replicas. This optimization avoids performing the force-merge redundantly on replicas, as the subsequent snapshot operation only looks at primary shards. We've seen some cases where cloning the index resulted in issues; there was a bug in the clone API that caused shards to be initializing permanently under specific circumstances (fixed by #137096), and cloned shards are unable to be assigned if their source lives on a node that is close/past the low watermark disk threshold (will be fixed soon by the Distributed Coordination team). Therefore, we implement an opt-out flag that users can configure in the `searchable_snapshot` action of their ILM policy if they don't want to clone the index with 0 replicas before performing the force-merge. We implement an opt-out instead of an opt-in, as we believe these issues to be rather specific (and soon resolved), and the clone is worth doing by default. (cherry picked from commit 0ab3240) # Conflicts: # server/src/main/resources/transport/upper_bounds/9.3.csv # x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/action/RestPutLifecycleAction.java * Fix transport version

If an index is in either `logsdb` or `time_series` mode and specifies a non-default `@timestamp` type mapping (e.g. `date_nanos`), using the clone, split, or shrink API will result in shards that are unable to initialize/recover due to a mapping conflict. As of elastic#133954, the `searchable_snapshot` ILM action makes use of the clone API by default - if the index has more than `0` replicas - and will thus also run into this issue.

Fix mapping conflicts in clone/split/shrink APIs

5ec65b4

nielsbauman added >bug :Data Management/Indices APIs APIs to create and manage indices and templates auto-backport Automatically create backport pull requests when merged branch:9.2 branch:8.19 labels Oct 24, 2025

elasticsearchmachine added Team:Data Management Meta label for data/management team v9.3.0 v9.2.1 v8.19.7 labels Oct 24, 2025

elasticsearchmachine removed branch:9.2 branch:8.19 labels Oct 24, 2025

Update docs/changelog/137096.yaml

5d459f6

nielsbauman added 2 commits October 24, 2025 15:06

Fix comment

42c211d

Merge branch 'main' into fix-clone

2f8f0ad

nielsbauman commented Oct 24, 2025

View reviewed changes

masseyke approved these changes Oct 24, 2025

View reviewed changes

henningandersen approved these changes Oct 24, 2025

View reviewed changes

...src/internalClusterTest/java/org/elasticsearch/action/admin/indices/create/CloneIndexIT.java Show resolved Hide resolved

...src/internalClusterTest/java/org/elasticsearch/action/admin/indices/create/CloneIndexIT.java Show resolved Hide resolved

nielsbauman added 2 commits October 24, 2025 16:23

Also ensureGreen at the end of tests

da0c541

Merge branch 'main' into fix-clone

5278988

nielsbauman enabled auto-merge (squash) October 24, 2025 14:24

nielsbauman merged commit 4a53551 into elastic:main Oct 24, 2025
34 checks passed

nielsbauman deleted the fix-clone branch October 24, 2025 15:47

This was referenced Oct 24, 2025

[9.2] Fix mapping conflicts in clone/split/shrink APIs (#137096) #137117

Merged

[8.19] Fix mapping conflicts in clone/split/shrink APIs (#137096) #137118

Merged

nielsbauman mentioned this pull request Oct 30, 2025

Allow opting out of force-merging on a cloned index in ILM's searchable snapshot action #137375

Merged

This was referenced Nov 6, 2025

Mirror upstream elastic/elasticsearch#137375 for AI review (snapshot of HEAD tree) phananh1010/elasticsearch#240

Closed

Mirror upstream elastic/elasticsearch#137375 for AI review (snapshot of HEAD tree) phananh1010/elasticsearch#270

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix mapping conflicts in clone/split/shrink APIs #137096

Fix mapping conflicts in clone/split/shrink APIs #137096

Uh oh!

nielsbauman commented Oct 24, 2025

Uh oh!

elasticsearchmachine commented Oct 24, 2025

Uh oh!

elasticsearchmachine commented Oct 24, 2025

Uh oh!

nielsbauman Oct 24, 2025

Uh oh!

mattc58 Oct 24, 2025

Uh oh!

nielsbauman Oct 24, 2025

Uh oh!

masseyke left a comment

Uh oh!

masseyke commented Oct 24, 2025

Uh oh!

henningandersen left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticsearchmachine commented Oct 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Fix mapping conflicts in clone/split/shrink APIs #137096

Fix mapping conflicts in clone/split/shrink APIs #137096

Uh oh!

Conversation

nielsbauman commented Oct 24, 2025

Uh oh!

elasticsearchmachine commented Oct 24, 2025

Uh oh!

elasticsearchmachine commented Oct 24, 2025

Uh oh!

nielsbauman Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

mattc58 Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

nielsbauman Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

masseyke left a comment

Choose a reason for hiding this comment

Uh oh!

masseyke commented Oct 24, 2025

Uh oh!

henningandersen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticsearchmachine commented Oct 24, 2025

💚 Backport successful

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants