Skip to content

Commit 1388110

Browse files
jordan-powersKubik42
jordan-powers
authored and
Kubik42
committed
Optionally ignore field when indexed field name exceeds length limit (elastic#136143)
This PR adds an index setting `index.mapping.field_name_length .ignore_dynamic_beyond_limit`. When this setting is set, fields dynamically added to the mapping that would violate the `index.mapping .field_name_length.limit` are silently ignored instead of causing an exception and preventing indexing of the document.
1 parent 2c8c075 commit 1388110

File tree

11 files changed

+289
-2
lines changed

11 files changed

+289
-2
lines changed

‎docs/changelog/136143.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
pr: 136143
2+
summary: Optionally ignore field when indexed field name exceeds length limit
3+
area: Mapping
4+
type: enhancement
5+
issues:
6+
- 135700

‎docs/reference/elasticsearch/index-settings/mapping-limit.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,9 @@ $$$ignore-dynamic-beyond-limit$$$
4646
`index.mapping.field_name_length.limit`
4747
: Setting for the maximum length of a field name. This setting isn’t really something that addresses mappings explosion but might still be useful if you want to limit the field length. It usually shouldn’t be necessary to set this setting. The default is okay unless a user starts to add a huge number of fields with really long names. Default is `Long.MAX_VALUE` (no limit).
4848

49+
`index.mapping.field_name_length.ignore_dynamic_beyond_limit` {applies_to}`stack: ga 9.3`
50+
: This setting determines what happens when a the name of a dynamically mapped field would exceed the configured maximum length. When set to `false` (the default), the index request of the document that tries to add a dynamic field to the mapping will fail with the message `Field name [x] is longer than the limit of [y] characters`. When set to `true`, the index request will not fail. Instead, fields that would exceed the limit are not added to the mapping, similar to [`dynamic: false`](/reference/elasticsearch/mapping-reference/dynamic.md). The fields that were not added to the mapping will be added to the [`_ignored` field](/reference/elasticsearch/mapping-reference/mapping-ignored-field.md). The default value is `false`.
51+
4952
`index.mapping.dimension_fields.limit`
5053
: (Dynamic, integer) Maximum number of [time series dimensions](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md#time-series-dimension) for the index. Defaults to `32768`.
5154

‎rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/index/100_field_name_length_limit.yml‎

Lines changed: 213 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,213 @@
1+
setup:
2+
- requires:
3+
cluster_features: ["mapper.ignore_dynamic_field_names_beyond_limit"]
4+
reason: "setting must be present to be tested"
5+
6+
---
7+
8+
"Test field name length limit":
9+
- do:
10+
indices.create:
11+
index: test_index
12+
body:
13+
settings:
14+
index:
15+
mapping:
16+
field_name_length:
17+
ignore_dynamic_beyond_limit: true
18+
limit: 45
19+
mappings:
20+
dynamic: true
21+
22+
- do:
23+
index:
24+
index: test_index
25+
id: 1
26+
refresh: true
27+
body:
28+
foo:
29+
test_really_long_field_name_that_will_be_ignored: foo
30+
field_name_not_ignored: 10
31+
32+
- do:
33+
indices.get_mapping:
34+
index: test_index
35+
36+
- match: { test_index.mappings.properties.foo.properties.test_really_long_field_name_that_will_be_ignored: null }
37+
- match: { test_index.mappings.properties.foo.properties.field_name_not_ignored.type: long }
38+
39+
- do:
40+
get:
41+
index: test_index
42+
id: 1
43+
_source: true
44+
stored_fields: [_ignored]
45+
- match:
46+
_source:
47+
foo:
48+
test_really_long_field_name_that_will_be_ignored: foo
49+
field_name_not_ignored: 10
50+
- match: { _ignored: [ "foo.test_really_long_field_name_that_will_be_ignored" ] }
51+
52+
---
53+
54+
"Test field name length limit array":
55+
- do:
56+
indices.create:
57+
index: test_index
58+
body:
59+
settings:
60+
index:
61+
mapping:
62+
field_name_length:
63+
ignore_dynamic_beyond_limit: true
64+
limit: 45
65+
mappings:
66+
dynamic: true
67+
68+
- do:
69+
index:
70+
index: test_index
71+
id: 1
72+
refresh: true
73+
body:
74+
foo:
75+
test_really_long_field_name_that_will_be_ignored: [ foo, bar, baz ]
76+
field_name_not_ignored: [ 10, 20, 30 ]
77+
78+
- do:
79+
indices.get_mapping:
80+
index: test_index
81+
82+
- match: { test_index.mappings.properties.foo.properties.test_really_long_field_name_that_will_be_ignored: null }
83+
- match: { test_index.mappings.properties.foo.properties.field_name_not_ignored.type: long }
84+
85+
- do:
86+
get:
87+
index: test_index
88+
id: 1
89+
_source: true
90+
stored_fields: [ _ignored ]
91+
92+
- match:
93+
_source:
94+
foo:
95+
test_really_long_field_name_that_will_be_ignored: [ foo, bar, baz ]
96+
field_name_not_ignored: [ 10, 20, 30 ]
97+
- match: {_ignored:["foo.test_really_long_field_name_that_will_be_ignored"]}
98+
99+
---
100+
101+
"Test field name length limit synthetic source":
102+
- do:
103+
indices.create:
104+
index: test_index
105+
body:
106+
settings:
107+
index:
108+
mapping:
109+
source.mode: synthetic
110+
field_name_length:
111+
ignore_dynamic_beyond_limit: true
112+
limit: 45
113+
mappings:
114+
dynamic: true
115+
116+
- do:
117+
index:
118+
index: test_index
119+
id: 1
120+
refresh: true
121+
body:
122+
foo:
123+
test_really_long_field_name_that_will_be_ignored: foo
124+
field_name_not_ignored: 10
125+
126+
- do:
127+
indices.get_mapping:
128+
index: test_index
129+
130+
- match: { test_index.mappings.properties.foo.properties.test_really_long_field_name_that_will_be_ignored: null }
131+
- match: { test_index.mappings.properties.foo.properties.field_name_not_ignored.type: long }
132+
133+
- do:
134+
get:
135+
index: test_index
136+
id: 1
137+
_source: true
138+
stored_fields: [ _ignored ]
139+
- match:
140+
_source:
141+
foo:
142+
test_really_long_field_name_that_will_be_ignored: foo
143+
field_name_not_ignored: 10
144+
- match: {_ignored:["foo.test_really_long_field_name_that_will_be_ignored"]}
145+
146+
---
147+
148+
"Test field name length limit array synthetic source":
149+
- do:
150+
indices.create:
151+
index: test_index
152+
body:
153+
settings:
154+
index:
155+
mapping:
156+
source.mode: synthetic
157+
field_name_length:
158+
ignore_dynamic_beyond_limit: true
159+
limit: 45
160+
mappings:
161+
dynamic: true
162+
163+
- do:
164+
index:
165+
index: test_index
166+
id: 1
167+
refresh: true
168+
body:
169+
foo:
170+
test_really_long_field_name_that_will_be_ignored: [ foo, bar, baz ]
171+
field_name_not_ignored: [ 10, 20, 30 ]
172+
173+
- do:
174+
indices.get_mapping:
175+
index: test_index
176+
177+
- match: { test_index.mappings.properties.foo.properties.test_really_long_field_name_that_will_be_ignored: null }
178+
- match: { test_index.mappings.properties.foo.properties.field_name_not_ignored.type: long }
179+
180+
- do:
181+
get:
182+
index: test_index
183+
id: 1
184+
_source: true
185+
stored_fields: [ _ignored ]
186+
- match:
187+
_source:
188+
foo:
189+
test_really_long_field_name_that_will_be_ignored: [ foo, bar, baz ]
190+
field_name_not_ignored: [ 10, 20, 30 ]
191+
- match: {_ignored:["foo.test_really_long_field_name_that_will_be_ignored"]}
192+
193+
---
194+
195+
"Test field name length limit with static mapping":
196+
- do:
197+
catch: /illegal_argument_exception/
198+
indices.create:
199+
index: test_index
200+
body:
201+
settings:
202+
index:
203+
mapping:
204+
field_name_length:
205+
ignore_dynamic_beyond_limit: true
206+
limit: 45
207+
mappings:
208+
dynamic: false
209+
properties:
210+
test_really_long_field_name_that_will_be_ignored:
211+
type: text
212+
213+
- match: {error.reason: "Field name [test_really_long_field_name_that_will_be_ignored] is longer than the limit of [45] characters"}

‎server/src/main/java/org/elasticsearch/common/settings/IndexScopedSettings.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -168,6 +168,7 @@ public final class IndexScopedSettings extends AbstractScopedSettings {
168168
MapperService.INDEX_MAPPING_NESTED_FIELDS_LIMIT_SETTING,
169169
MapperService.INDEX_MAPPING_NESTED_DOCS_LIMIT_SETTING,
170170
MapperService.INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_LIMIT_SETTING,
171+
MapperService.INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_FIELD_NAME_LENGTH_SETTING,
171172
MapperService.INDEX_MAPPING_TOTAL_FIELDS_LIMIT_SETTING,
172173
MapperService.INDEX_MAPPING_DEPTH_LIMIT_SETTING,
173174
MapperService.INDEX_MAPPING_DIMENSION_FIELDS_LIMIT_SETTING,

‎server/src/main/java/org/elasticsearch/index/IndexSettings.java‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@
5252
import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_DEPTH_LIMIT_SETTING;
5353
import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_DIMENSION_FIELDS_LIMIT_SETTING;
5454
import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_FIELD_NAME_LENGTH_LIMIT_SETTING;
55+
import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_FIELD_NAME_LENGTH_SETTING;
5556
import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_LIMIT_SETTING;
5657
import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_NESTED_DOCS_LIMIT_SETTING;
5758
import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_NESTED_FIELDS_LIMIT_SETTING;
@@ -926,6 +927,7 @@ private void setRetentionLeaseMillis(final TimeValue retentionLease) {
926927
private volatile long mappingNestedDocsLimit;
927928
private volatile long mappingTotalFieldsLimit;
928929
private volatile boolean ignoreDynamicFieldsBeyondLimit;
930+
private volatile boolean ignoreDynamicFieldNamesBeyondLimit;
929931
private volatile long mappingDepthLimit;
930932
private volatile long mappingFieldNameLengthLimit;
931933
private volatile long mappingDimensionFieldsLimit;
@@ -1101,6 +1103,7 @@ public IndexSettings(final IndexMetadata indexMetadata, final Settings nodeSetti
11011103
mappingNestedDocsLimit = scopedSettings.get(INDEX_MAPPING_NESTED_DOCS_LIMIT_SETTING);
11021104
mappingTotalFieldsLimit = scopedSettings.get(INDEX_MAPPING_TOTAL_FIELDS_LIMIT_SETTING);
11031105
ignoreDynamicFieldsBeyondLimit = scopedSettings.get(INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_LIMIT_SETTING);
1106+
ignoreDynamicFieldNamesBeyondLimit = scopedSettings.get(INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_FIELD_NAME_LENGTH_SETTING);
11041107
mappingDepthLimit = scopedSettings.get(INDEX_MAPPING_DEPTH_LIMIT_SETTING);
11051108
mappingFieldNameLengthLimit = scopedSettings.get(INDEX_MAPPING_FIELD_NAME_LENGTH_LIMIT_SETTING);
11061109
mappingDimensionFieldsLimit = scopedSettings.get(INDEX_MAPPING_DIMENSION_FIELDS_LIMIT_SETTING);
@@ -1218,6 +1221,10 @@ public IndexSettings(final IndexMetadata indexMetadata, final Settings nodeSetti
12181221
INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_LIMIT_SETTING,
12191222
this::setIgnoreDynamicFieldsBeyondLimit
12201223
);
1224+
scopedSettings.addSettingsUpdateConsumer(
1225+
INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_FIELD_NAME_LENGTH_SETTING,
1226+
this::setIgnoreDynamicFieldNamesBeyondLimit
1227+
);
12211228
scopedSettings.addSettingsUpdateConsumer(INDEX_MAPPING_TOTAL_FIELDS_LIMIT_SETTING, this::setMappingTotalFieldsLimit);
12221229
scopedSettings.addSettingsUpdateConsumer(INDEX_MAPPING_DEPTH_LIMIT_SETTING, this::setMappingDepthLimit);
12231230
scopedSettings.addSettingsUpdateConsumer(INDEX_MAPPING_FIELD_NAME_LENGTH_LIMIT_SETTING, this::setMappingFieldNameLengthLimit);
@@ -1765,10 +1772,18 @@ private void setIgnoreDynamicFieldsBeyondLimit(boolean ignoreDynamicFieldsBeyond
17651772
this.ignoreDynamicFieldsBeyondLimit = ignoreDynamicFieldsBeyondLimit;
17661773
}
17671774

1775+
private void setIgnoreDynamicFieldNamesBeyondLimit(boolean ignoreDynamicFieldNamesBeyondLimit) {
1776+
this.ignoreDynamicFieldNamesBeyondLimit = ignoreDynamicFieldNamesBeyondLimit;
1777+
}
1778+
17681779
public boolean isIgnoreDynamicFieldsBeyondLimit() {
17691780
return ignoreDynamicFieldsBeyondLimit;
17701781
}
17711782

1783+
public boolean isIgnoreDynamicFieldNamesBeyondLimit() {
1784+
return ignoreDynamicFieldNamesBeyondLimit;
1785+
}
1786+
17721787
public long getMappingDepthLimit() {
17731788
return mappingDepthLimit;
17741789
}

‎server/src/main/java/org/elasticsearch/index/mapper/DocumentParser.java‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -679,6 +679,26 @@ private static void parseArrayDynamic(DocumentParserContext context, String curr
679679
context.addIgnoredField(context.path().pathAsText(currentFieldName));
680680
return;
681681
}
682+
683+
if (context.indexSettings().isIgnoreDynamicFieldNamesBeyondLimit()
684+
&& currentFieldName.length() > context.indexSettings().getMappingFieldNameLengthLimit()) {
685+
if (context.canAddIgnoredField()) {
686+
try {
687+
context.addIgnoredField(
688+
IgnoredSourceFieldMapper.NameValue.fromContext(
689+
context,
690+
context.path().pathAsText(currentFieldName),
691+
context.encodeFlattenedToken()
692+
)
693+
);
694+
} catch (IOException e) {
695+
throw new IllegalArgumentException("failed to parse field [" + currentFieldName + "]", e);
696+
}
697+
}
698+
context.addIgnoredField(context.path().pathAsText(currentFieldName));
699+
return;
700+
}
701+
682702
parseNonDynamicArray(context, objectMapperFromTemplate, currentFieldName, currentFieldName);
683703
} else {
684704
if (parsesArrayValue(objectMapperFromTemplate)) {

‎server/src/main/java/org/elasticsearch/index/mapper/DocumentParserContext.java‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -559,6 +559,24 @@ public final boolean addDynamicMapper(Mapper mapper) {
559559
mappingLookup.checkFieldLimit(indexSettings().getMappingTotalFieldsLimit(), additionalFieldsToAdd);
560560
}
561561
dynamicMappersSize.add(mapperSize);
562+
563+
if (indexSettings().isIgnoreDynamicFieldNamesBeyondLimit()) {
564+
if (mapper.leafName().length() > indexSettings().getMappingFieldNameLengthLimit()) {
565+
if (canAddIgnoredField()) {
566+
try {
567+
addIgnoredField(
568+
IgnoredSourceFieldMapper.NameValue.fromContext(this, mapper.fullPath(), encodeFlattenedToken())
569+
);
570+
} catch (IOException e) {
571+
throw new IllegalArgumentException("failed to parse field [" + mapper.fullPath() + "]", e);
572+
}
573+
}
574+
addIgnoredField(mapper.fullPath());
575+
return false;
576+
} else {
577+
mappingLookup.checkFieldNameLengthLimit(indexSettings().getMappingFieldNameLengthLimit());
578+
}
579+
}
562580
}
563581
if (mapper instanceof ObjectMapper objectMapper) {
564582
dynamicObjectMappers.put(objectMapper.fullPath(), objectMapper);

‎server/src/main/java/org/elasticsearch/index/mapper/MapperFeatures.java‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,9 @@ public class MapperFeatures implements FeatureSpecification {
5454
static final NodeFeature PATTERN_TEXT_RENAME = new NodeFeature("mapper.pattern_text_rename");
5555
static final NodeFeature DISKBBQ_ON_DISK_RESCORING = new NodeFeature("mapper.vectors.diskbbq_on_disk_rescoring");
5656
static final NodeFeature PROVIDE_INDEX_SORT_SETTING_DEFAULTS = new NodeFeature("mapper.provide_index_sort_setting_defaults");
57+
static final NodeFeature INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_FIELD_NAME_LIMIT = new NodeFeature(
58+
"mapper.ignore_dynamic_field_names_beyond_limit"
59+
);
5760

5861
@Override
5962
public Set<NodeFeature> getTestFeatures() {
@@ -93,7 +96,8 @@ public Set<NodeFeature> getTestFeatures() {
9396
MATCH_ONLY_TEXT_BLOCK_LOADER_FIX,
9497
PATTERN_TEXT_RENAME,
9598
DISKBBQ_ON_DISK_RESCORING,
96-
PROVIDE_INDEX_SORT_SETTING_DEFAULTS
99+
PROVIDE_INDEX_SORT_SETTING_DEFAULTS,
100+
INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_FIELD_NAME_LIMIT
97101
);
98102
}
99103
}

‎server/src/main/java/org/elasticsearch/index/mapper/MapperService.java‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -157,6 +157,12 @@ public boolean isAutoUpdate() {
157157
Property.Dynamic,
158158
Property.IndexScope
159159
);
160+
public static final Setting<Boolean> INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_FIELD_NAME_LENGTH_SETTING = Setting.boolSetting(
161+
"index.mapping.field_name_length.ignore_dynamic_beyond_limit",
162+
false,
163+
Property.Dynamic,
164+
Property.IndexScope
165+
);
160166
public static final Setting<Long> INDEX_MAPPING_DIMENSION_FIELDS_LIMIT_SETTING = Setting.longSetting(
161167
"index.mapping.dimension_fields.limit",
162168
32_768,

‎server/src/main/java/org/elasticsearch/index/mapper/MappingLookup.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -347,7 +347,7 @@ static void checkObjectDepthLimit(long limit, String objectPath) {
347347
}
348348
}
349349

350-
private void checkFieldNameLengthLimit(long limit) {
350+
void checkFieldNameLengthLimit(long limit) {
351351
validateMapperNameIn(objectMappers.values(), limit);
352352
validateMapperNameIn(fieldMappers.values(), limit);
353353
}

0 commit comments

Comments
 (0)