Skip to content

Commit b344a15

Browse files
jimczijimczi
authored
Add index.mapping.nested_parents.limit and raise nested fields limit to 100 (elastic#138961)
This change introduces a new index setting, index.mapping.nested_parents.limit, which limits the number of nested fields that act as parents of other nested fields. Each nested parent requires its own parent bitset in memory, so this new limit (defaulting to 50) prevents excessive bitset creation in deeply nested mappings. Additionally, the existing index.mapping.nested_fields.limit is increased from 50 to 100 to better accommodate common use cases while still protecting against overly complex mappings.
1 parent 7cb125b commit b344a15

File tree

12 files changed

+174
-12
lines changed

12 files changed

+174
-12
lines changed

‎docs/changelog/138961.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
pr: 138961
2+
summary: Add `index.mapping.nested_parents.limit` and raise nested fields limit to
3+
100
4+
area: Mapping
5+
type: enhancement
6+
issues: []

‎docs/reference/elasticsearch/index-settings/mapping-limit.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,10 @@ $$$ignore-dynamic-beyond-limit$$$
3838
: The maximum depth for a field, which is measured as the number of inner objects. For instance, if all fields are defined at the root object level, then the depth is `1`. If there is one object mapping, then the depth is `2`, etc. Default is `20`.
3939

4040
`index.mapping.nested_fields.limit`
41-
: The maximum number of distinct `nested` mappings in an index. The `nested` type should only be used in special cases, when arrays of objects need to be queried independently of each other. To safeguard against poorly designed mappings, this setting limits the number of unique `nested` types per index. Default is `50`.
41+
: The maximum number of distinct `nested` mappings in an index. The `nested` type should only be used in special cases, when arrays of objects need to be queried independently of each other. To safeguard against poorly designed mappings, this setting limits the number of unique `nested` types per index. Default is `100`.
42+
43+
`index.mapping.nested_parents.limit`
44+
: The maximum number of nested fields that act as parents of other nested fields. Each nested parent requires its own in-memory parent bitset. Root-level nested fields share a parent bitset, but nested fields under other nested fields require additional bitsets. This setting limits the number of unique nested parents to prevent excessive memory usage. Default is `50`.
4245

4346
`index.mapping.nested_objects.limit`
4447
: The maximum number of nested JSON objects that a single document can contain across all `nested` types. This limit helps to prevent out of memory errors when a document contains too many nested objects. Default is `10000`.

‎docs/reference/elasticsearch/mapping-reference/nested.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -197,10 +197,13 @@ The following parameters are accepted by `nested` fields:
197197
As described earlier, each nested object is indexed as a separate Lucene document. Continuing with the previous example, if we indexed a single document containing 100 `user` objects, then 101 Lucene documents would be created: one for the parent document, and one for each nested object. Because of the expense associated with `nested` mappings, Elasticsearch puts settings in place to guard against performance problems:
198198

199199
`index.mapping.nested_fields.limit`
200-
: The maximum number of distinct `nested` mappings in an index. The `nested` type should only be used in special cases, when arrays of objects need to be queried independently of each other. To safeguard against poorly designed mappings, this setting limits the number of unique `nested` types per index. Default is `50`.
200+
: The maximum number of distinct `nested` mappings in an index. The `nested` type should only be used in special cases, when arrays of objects need to be queried independently of each other. To safeguard against poorly designed mappings, this setting limits the number of unique `nested` types per index. Default is `100`.
201201

202202
In the previous example, the `user` mapping would count as only 1 towards this limit.
203203

204+
`index.mapping.nested_parents.limit`
205+
: The maximum number of nested fields that act as parents of other nested fields. Each nested parent requires its own in-memory parent bitset. Root-level nested fields share a parent bitset, but nested fields under other nested fields require additional bitsets. This setting limits the number of unique nested parents to prevent excessive memory usage. Default is `50`.
206+
204207
`index.mapping.nested_objects.limit`
205208
: The maximum number of nested JSON objects that a single document can contain across all `nested` types. This limit helps to prevent out of memory errors when a document contains too many nested objects. Default is `10000`.
206209

‎server/src/main/java/org/elasticsearch/common/settings/IndexScopedSettings.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,7 @@ public final class IndexScopedSettings extends AbstractScopedSettings {
167167
FieldMapper.COERCE_SETTING,
168168
Store.INDEX_STORE_STATS_REFRESH_INTERVAL_SETTING,
169169
MapperService.INDEX_MAPPING_NESTED_FIELDS_LIMIT_SETTING,
170+
MapperService.INDEX_MAPPING_NESTED_PARENTS_LIMIT_SETTING,
170171
MapperService.INDEX_MAPPING_NESTED_DOCS_LIMIT_SETTING,
171172
MapperService.INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_LIMIT_SETTING,
172173
MapperService.INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_FIELD_NAME_LENGTH_SETTING,

‎server/src/main/java/org/elasticsearch/common/settings/Setting.java‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1474,6 +1474,11 @@ public static Setting<Long> longSetting(String key, long defaultValue, long minV
14741474
return new Setting<>(key, Long.toString(defaultValue), s -> parseLong(s, minValue, key, isFiltered), properties);
14751475
}
14761476

1477+
public static Setting<Long> longSetting(String key, Function<Settings, String> defaultValueFn, long minValue, Property... properties) {
1478+
boolean isFiltered = isFiltered(properties);
1479+
return new Setting<>(key, defaultValueFn, s -> parseLong(s, minValue, key, isFiltered), properties);
1480+
}
1481+
14771482
public static Setting<Instant> dateSetting(String key, Instant defaultValue, Validator<Instant> validator, Property... properties) {
14781483
final String defaultString = defaultValue.toString();
14791484
return new Setting<>(

‎server/src/main/java/org/elasticsearch/index/IndexSettings.java‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,7 @@
5656
import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_LIMIT_SETTING;
5757
import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_NESTED_DOCS_LIMIT_SETTING;
5858
import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_NESTED_FIELDS_LIMIT_SETTING;
59+
import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_NESTED_PARENTS_LIMIT_SETTING;
5960
import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_TOTAL_FIELDS_LIMIT_SETTING;
6061

6162
/**
@@ -1004,6 +1005,7 @@ private void setRetentionLeaseMillis(final TimeValue retentionLease) {
10041005
private volatile String defaultPipeline;
10051006
private volatile String requiredPipeline;
10061007
private volatile long mappingNestedFieldsLimit;
1008+
private volatile long mappingNestedParentsLimit;
10071009
private volatile long mappingNestedDocsLimit;
10081010
private volatile long mappingTotalFieldsLimit;
10091011
private volatile boolean ignoreDynamicFieldsBeyondLimit;
@@ -1183,6 +1185,7 @@ public IndexSettings(final IndexMetadata indexMetadata, final Settings nodeSetti
11831185
searchIdleAfter = scopedSettings.get(INDEX_SEARCH_IDLE_AFTER);
11841186
defaultPipeline = scopedSettings.get(DEFAULT_PIPELINE);
11851187
mappingNestedFieldsLimit = scopedSettings.get(INDEX_MAPPING_NESTED_FIELDS_LIMIT_SETTING);
1188+
mappingNestedParentsLimit = scopedSettings.get(INDEX_MAPPING_NESTED_PARENTS_LIMIT_SETTING);
11861189
mappingNestedDocsLimit = scopedSettings.get(INDEX_MAPPING_NESTED_DOCS_LIMIT_SETTING);
11871190
mappingTotalFieldsLimit = scopedSettings.get(INDEX_MAPPING_TOTAL_FIELDS_LIMIT_SETTING);
11881191
ignoreDynamicFieldsBeyondLimit = scopedSettings.get(INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_LIMIT_SETTING);
@@ -1323,6 +1326,7 @@ public IndexSettings(final IndexMetadata indexMetadata, final Settings nodeSetti
13231326
scopedSettings.addSettingsUpdateConsumer(INDEX_SOFT_DELETES_RETENTION_OPERATIONS_SETTING, this::setSoftDeleteRetentionOperations);
13241327
scopedSettings.addSettingsUpdateConsumer(INDEX_SOFT_DELETES_RETENTION_LEASE_PERIOD_SETTING, this::setRetentionLeaseMillis);
13251328
scopedSettings.addSettingsUpdateConsumer(INDEX_MAPPING_NESTED_FIELDS_LIMIT_SETTING, this::setMappingNestedFieldsLimit);
1329+
scopedSettings.addSettingsUpdateConsumer(INDEX_MAPPING_NESTED_PARENTS_LIMIT_SETTING, this::setMappingNestedParentsLimit);
13261330
scopedSettings.addSettingsUpdateConsumer(INDEX_MAPPING_NESTED_DOCS_LIMIT_SETTING, this::setMappingNestedDocsLimit);
13271331
scopedSettings.addSettingsUpdateConsumer(
13281332
INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_LIMIT_SETTING,
@@ -1859,6 +1863,14 @@ private void setMappingNestedFieldsLimit(long value) {
18591863
this.mappingNestedFieldsLimit = value;
18601864
}
18611865

1866+
public long getMappingNestedParentsLimit() {
1867+
return mappingNestedParentsLimit;
1868+
}
1869+
1870+
private void setMappingNestedParentsLimit(long value) {
1871+
this.mappingNestedParentsLimit = value;
1872+
}
1873+
18621874
public long getMappingNestedDocsLimit() {
18631875
return mappingNestedDocsLimit;
18641876
}

‎server/src/main/java/org/elasticsearch/index/IndexVersions.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -198,6 +198,7 @@ private static Version parseUnchecked(String version) {
198198
public static final IndexVersion UPGRADE_TO_LUCENE_10_3_2 = def(9_047_0_00, Version.LUCENE_10_3_2);
199199
public static final IndexVersion SECURITY_MIGRATIONS_METADATA_FLATTENED_UPDATE = def(9_048_0_00, Version.LUCENE_10_3_2);
200200
public static final IndexVersion STANDARD_INDEXES_USE_SKIPPERS = def(9_049_0_00, Version.LUCENE_10_3_2);
201+
public static final IndexVersion NESTED_PATH_LIMIT = def(9_050_0_00, Version.LUCENE_10_3_2);
201202

202203
/*
203204
* STOP! READ THIS FIRST! No, really,

‎server/src/main/java/org/elasticsearch/index/mapper/MapperService.java‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -101,9 +101,31 @@ public boolean isAutoUpdate() {
101101

102102
public static final String SINGLE_MAPPING_NAME = "_doc";
103103
public static final String TYPE_FIELD_NAME = "_type";
104+
105+
private static final int LEGACY_NESTED_FIELDS_LIMIT = 50;
106+
private static final int DEFAULT_NESTED_FIELDS_LIMIT = 100;
104107
public static final Setting<Long> INDEX_MAPPING_NESTED_FIELDS_LIMIT_SETTING = Setting.longSetting(
105108
"index.mapping.nested_fields.limit",
106-
50L,
109+
settings -> {
110+
final IndexVersion indexVersionCreated = IndexMetadata.SETTING_INDEX_VERSION_CREATED.get(settings);
111+
return Integer.toString(
112+
indexVersionCreated.onOrAfter(IndexVersions.NESTED_PATH_LIMIT) ? DEFAULT_NESTED_FIELDS_LIMIT : LEGACY_NESTED_FIELDS_LIMIT
113+
);
114+
},
115+
0,
116+
Property.Dynamic,
117+
Property.IndexScope
118+
);
119+
private static final int LEGACY_NESTED_PARENTS_LIMIT = Integer.MAX_VALUE;
120+
private static final int DEFAULT_NESTED_PARENTS_LIMIT = 50;
121+
public static final Setting<Long> INDEX_MAPPING_NESTED_PARENTS_LIMIT_SETTING = Setting.longSetting(
122+
"index.mapping.nested_parents.limit",
123+
settings -> {
124+
final IndexVersion indexVersionCreated = IndexMetadata.SETTING_INDEX_VERSION_CREATED.get(settings);
125+
return Integer.toString(
126+
indexVersionCreated.onOrAfter(IndexVersions.NESTED_PATH_LIMIT) ? DEFAULT_NESTED_PARENTS_LIMIT : LEGACY_NESTED_PARENTS_LIMIT
127+
);
128+
},
107129
0,
108130
Property.Dynamic,
109131
Property.IndexScope

‎server/src/main/java/org/elasticsearch/index/mapper/MappingLookup.java‎

Lines changed: 20 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99

1010
package org.elasticsearch.index.mapper;
1111

12+
import org.apache.lucene.search.Query;
1213
import org.elasticsearch.cluster.metadata.DataStream;
1314
import org.elasticsearch.cluster.metadata.InferenceFieldMetadata;
1415
import org.elasticsearch.core.Nullable;
@@ -23,6 +24,7 @@
2324
import java.util.Collection;
2425
import java.util.Collections;
2526
import java.util.HashMap;
27+
import java.util.HashSet;
2628
import java.util.List;
2729
import java.util.Map;
2830
import java.util.Set;
@@ -290,7 +292,8 @@ void checkLimits(IndexSettings settings) {
290292
checkFieldLimit(settings.getMappingTotalFieldsLimit());
291293
checkObjectDepthLimit(settings.getMappingDepthLimit());
292294
checkFieldNameLengthLimit(settings.getMappingFieldNameLengthLimit());
293-
checkNestedLimit(settings.getMappingNestedFieldsLimit());
295+
checkNestedFieldsLimit(settings.getMappingNestedFieldsLimit());
296+
checkNestedParentsLimit(settings.getMappingNestedParentsLimit());
294297
checkDimensionFieldLimit(settings.getMappingDimensionFieldsLimit());
295298
}
296299

@@ -362,18 +365,26 @@ private static void validateMapperNameIn(Collection<? extends Mapper> mappers, l
362365
}
363366
}
364367

365-
private void checkNestedLimit(long limit) {
366-
long actualNestedFields = 0;
367-
for (ObjectMapper objectMapper : objectMappers.values()) {
368-
if (objectMapper.isNested()) {
369-
actualNestedFields++;
370-
}
371-
}
372-
if (actualNestedFields > limit) {
368+
private void checkNestedFieldsLimit(long limit) {
369+
if (nestedLookup.getNestedMappers().size() > limit) {
373370
throw new IllegalArgumentException("Limit of nested fields [" + limit + "] has been exceeded");
374371
}
375372
}
376373

374+
private void checkNestedParentsLimit(long limit) {
375+
if (nestedLookup.getNestedMappers().size() < limit) {
376+
return;
377+
}
378+
379+
Set<Query> nestedPaths = new HashSet<>();
380+
for (var nested : nestedLookup.getNestedMappers().values()) {
381+
nestedPaths.add(nested.parentTypeFilter());
382+
}
383+
if (nestedPaths.size() > limit) {
384+
throw new IllegalArgumentException("Limit of nested parents [" + limit + "] has been exceeded");
385+
}
386+
}
387+
377388
public Map<String, ObjectMapper> objectMappers() {
378389
return objectMappers;
379390
}

‎server/src/test/java/org/elasticsearch/index/mapper/NestedObjectMapperTests.java‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -833,6 +833,72 @@ public void testLimitOfNestedFieldsPerIndex() throws Exception {
833833
merge(mapperService, MergeReason.MAPPING_RECOVERY, mapping.apply("_doc"));
834834
}
835835

836+
public void testLimitOfNestedParentsPerIndex() throws Exception {
837+
String mapping = """
838+
{
839+
"_doc": {
840+
"properties": {
841+
"nested1": {
842+
"type": "nested",
843+
"properties": {
844+
"nested2": {
845+
"type": "nested"
846+
}
847+
}
848+
},
849+
"nested3": {
850+
"type": "nested"
851+
}
852+
}
853+
}
854+
}
855+
""";
856+
// default limit allows at least two nested fields
857+
createMapperService(mapping);
858+
859+
// explicitly setting limit to 0 prevents nested fields
860+
Exception e = expectThrows(IllegalArgumentException.class, () -> {
861+
Settings settings = Settings.builder().put(MapperService.INDEX_MAPPING_NESTED_PARENTS_LIMIT_SETTING.getKey(), 0).build();
862+
createMapperService(settings, mapping);
863+
});
864+
assertThat(e.getMessage(), containsString("Limit of nested parents [0] has been exceeded"));
865+
866+
// setting limit to 1 with 2 nested parent fails
867+
e = expectThrows(IllegalArgumentException.class, () -> {
868+
Settings settings = Settings.builder().put(MapperService.INDEX_MAPPING_NESTED_PARENTS_LIMIT_SETTING.getKey(), 1).build();
869+
createMapperService(settings, mapping);
870+
});
871+
assertThat(e.getMessage(), containsString("Limit of nested parents [1] has been exceeded"));
872+
873+
{
874+
Settings settings = Settings.builder().put(MapperService.INDEX_MAPPING_NESTED_PARENTS_LIMIT_SETTING.getKey(), 2).build();
875+
var mapperService = createMapperService(settings, mapping);
876+
merge(mapperService, mapping(b -> b.startObject("nested3").field("type", "nested")).endObject());
877+
var iae = expectThrows(
878+
IllegalArgumentException.class,
879+
() -> merge(
880+
mapperService,
881+
mapping(
882+
b -> b.startObject("nested3")
883+
.field("type", "nested")
884+
.startObject("properties")
885+
.startObject("nested4")
886+
.field("type", "nested")
887+
.endObject()
888+
.endObject()
889+
.endObject()
890+
)
891+
)
892+
);
893+
assertThat(iae.getMessage(), containsString("Limit of nested parents [2] has been exceeded"));
894+
}
895+
896+
// do not check nested fields limit if mapping is not updated
897+
Settings settings = Settings.builder().put(MapperService.INDEX_MAPPING_NESTED_PARENTS_LIMIT_SETTING.getKey(), 0).build();
898+
MapperService mapperService = createMapperService(settings, mapping(b -> {}));
899+
merge(mapperService, MergeReason.MAPPING_RECOVERY, mapping);
900+
}
901+
836902
public void testLimitNestedDocsDefaultSettings() throws Exception {
837903
Settings settings = Settings.builder().build();
838904
DocumentMapper docMapper = createDocumentMapper(mapping(b -> b.startObject("nested1").field("type", "nested").endObject()));
@@ -1934,4 +2000,34 @@ public void testIsInNestedContext() {
19342000
MapperBuilderContext childContext = context.createChildContext("child", false, Dynamic.FALSE);
19352001
assertTrue(childContext.isInNestedContext());
19362002
}
2003+
2004+
public void testNestedLimitDefaults() throws IOException {
2005+
// current defaults
2006+
{
2007+
var version = IndexVersionUtils.randomVersionBetween(random(), IndexVersions.NESTED_PATH_LIMIT, IndexVersion.current());
2008+
var mapperService = createMapperService(version, Settings.builder().build(), mapping(b -> {}));
2009+
assertThat(
2010+
MapperService.INDEX_MAPPING_NESTED_FIELDS_LIMIT_SETTING.get(mapperService.getIndexSettings().getSettings()),
2011+
equalTo(100L)
2012+
);
2013+
assertThat(
2014+
MapperService.INDEX_MAPPING_NESTED_PARENTS_LIMIT_SETTING.get(mapperService.getIndexSettings().getSettings()),
2015+
equalTo(50L)
2016+
);
2017+
}
2018+
2019+
// defaults previous to IndexVersions.NESTED_PATH_LIMIT
2020+
{
2021+
var version = IndexVersionUtils.randomPreviousCompatibleVersion(random(), IndexVersions.NESTED_PATH_LIMIT);
2022+
var mapperService = createMapperService(version, Settings.builder().build(), mapping(b -> {}));
2023+
assertThat(
2024+
MapperService.INDEX_MAPPING_NESTED_FIELDS_LIMIT_SETTING.get(mapperService.getIndexSettings().getSettings()),
2025+
equalTo(50L)
2026+
);
2027+
assertThat(
2028+
MapperService.INDEX_MAPPING_NESTED_PARENTS_LIMIT_SETTING.get(mapperService.getIndexSettings().getSettings()),
2029+
equalTo((long) Integer.MAX_VALUE)
2030+
);
2031+
}
2032+
}
19372033
}

0 commit comments

Comments
 (0)