Skip to content

Store arrays offsets for numeric fields natively with synthetic source #124594

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 41 commits into from
Mar 20, 2025
Merged

Store arrays offsets for numeric fields natively with synthetic source #124594

Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
6213d87
WIP numeric fields native synthetic array support
jordan-powers Mar 7, 2025
7fe15da
Disable native synthetic source support within nested contexts
jordan-powers Mar 10, 2025
3d79319
Fix hasValue logic for case of empty array
jordan-powers Mar 10, 2025
e3961ec
Fix missing field name for all-null arrays
jordan-powers Mar 10, 2025
484dabb
Add LongOffsetDocValuesLoaderTests.java
jordan-powers Mar 10, 2025
f0d6914
Always store array offsets
jordan-powers Mar 11, 2025
188a478
Use abstract getRandomValue for testSynthesizeArrayRandom
jordan-powers Mar 11, 2025
1ea276b
Use correct fieldType in verifySyntheticArrayInObject
jordan-powers Mar 11, 2025
0656ba6
Avoid string-encoding expected values
jordan-powers Mar 11, 2025
b790084
Don't assume response will be array in NativeArrayIntegrationTestCase
jordan-powers Mar 11, 2025
2533b0b
Add LongSyntheticSourceNativeArrayIntegrationTests
jordan-powers Mar 11, 2025
578fc42
Fix x-pack build issues
jordan-powers Mar 11, 2025
7bfd82b
Merge remote-tracking branch 'upstream/main' into array-offset-encodi…
jordan-powers Mar 11, 2025
3d7347c
Update docs/changelog/124594.yaml
jordan-powers Mar 11, 2025
6ab26af
Revert "Always store array offsets"
jordan-powers Mar 12, 2025
c514dc4
Remove NumberType#sortableLongValue
jordan-powers Mar 12, 2025
e19c6e5
Revert "Don't assume response will be array in NativeArrayIntegration…
jordan-powers Mar 12, 2025
52e2aba
Merge remote-tracking branch 'origin/array-offset-encoding-numbers' i…
jordan-powers Mar 13, 2025
86d6c6b
Add testSynthesizeArrayRandomIgnoresMalformed
jordan-powers Mar 13, 2025
ed90189
Convert FieldLoader to FieldLoaderLayer
jordan-powers Mar 13, 2025
e08929f
Mask offsets field in roundtrip tests
jordan-powers Mar 13, 2025
5ce5bcc
Fix testSyntheticSourceKeepArrays expecting precision to be preserved
jordan-powers Mar 13, 2025
0743b31
Merge remote-tracking branch 'upstream/main' into array-offset-encodi…
jordan-powers Mar 13, 2025
1bd67e0
Fix UnsignedLongFieldMapperTests
jordan-powers Mar 13, 2025
f6793c1
Merge remote-tracking branch 'upstream/main' into array-offset-encodi…
jordan-powers Mar 13, 2025
f27a52e
Account for duplicate doc values in loader layer
jordan-powers Mar 17, 2025
d4185eb
Fix value cast in Byte number type
jordan-powers Mar 17, 2025
82031c1
Remove unused leafName from field loader layer
jordan-powers Mar 17, 2025
cd56130
Merge remote-tracking branch 'upstream/main' into array-offset-encodi…
jordan-powers Mar 17, 2025
4d2b559
Allow multiple documents in testSynthesizeArrayRandomIgnoresMalformed
jordan-powers Mar 17, 2025
316b94c
Add test verifying offset encoding is disabled in nested contexts
jordan-powers Mar 17, 2025
65c1bc4
Add test with duplicate values to LongOffsetDocValuesLoaderTests
jordan-powers Mar 18, 2025
a4cebf6
Fix deduplication logic in doc values loader
jordan-powers Mar 18, 2025
1cfa022
Add duplicate values to testOffsetArrayRandom
jordan-powers Mar 18, 2025
321f652
Add doc values loader tests for other numeric types
jordan-powers Mar 18, 2025
1b3472b
Merge remote-tracking branch 'upstream/main' into array-offset-encodi…
jordan-powers Mar 18, 2025
96534a0
Add native array integration tests for other numeric types
jordan-powers Mar 19, 2025
61831f1
Merge remote-tracking branch 'upstream/main' into array-offset-encodi…
jordan-powers Mar 19, 2025
4efc7fe
Remove 'Object' from type parameter
jordan-powers Mar 19, 2025
9ca9c7c
Fix native array integration tests
jordan-powers Mar 19, 2025
47f3bbc
Merge remote-tracking branch 'upstream/main' into array-offset-encodi…
jordan-powers Mar 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions server/src/main/java/org/elasticsearch/index/IndexVersions.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,7 @@ private static Version parseUnchecked(String version) {
public static final IndexVersion TIME_SERIES_ID_DOC_VALUES_SPARSE_INDEX = def(9_012_0_00, Version.LUCENE_10_1_0);
public static final IndexVersion SYNTHETIC_SOURCE_STORE_ARRAYS_NATIVELY_KEYWORD = def(9_013_0_00, Version.LUCENE_10_1_0);
public static final IndexVersion SYNTHETIC_SOURCE_STORE_ARRAYS_NATIVELY_IP = def(9_014_0_00, Version.LUCENE_10_1_0);
public static final IndexVersion SYNTHETIC_SOURCE_STORE_ARRAYS_NATIVELY_NUMBER = def(9_015_0_00, Version.LUCENE_10_1_0);
/*
* STOP! READ THIS FIRST! No, really,
* ____ _____ ___ ____ _ ____ _____ _ ____ _____ _ _ ___ ____ _____ ___ ____ ____ _____ _
Expand Down
4 changes: 3 additions & 1 deletion server/src/main/java/org/elasticsearch/index/mapper/DocumentParser.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -460,7 +460,9 @@ static void parseObjectOrField(DocumentParserContext context, Mapper mapper) thr
if (context.canAddIgnoredField()
&& (fieldMapper.syntheticSourceMode() == FieldMapper.SyntheticSourceMode.FALLBACK
|| sourceKeepMode == Mapper.SourceKeepMode.ALL
|| (sourceKeepMode == Mapper.SourceKeepMode.ARRAYS && context.inArrayScope())
|| (sourceKeepMode == Mapper.SourceKeepMode.ARRAYS
&& context.inArrayScope()
&& fieldMapper.supportStoringArrayOffsets() == false)
Copy link
Member

@martijnvg martijnvg Mar 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain why this change is needed?

Copy link
Contributor Author

@jordan-powers jordan-powers Mar 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was part of the change removing the check for context.isImmediateParentAnArray(). Adding this to the check disabled the fallback source for that object array so that we could use the offset encoding.

Since we're rethinking that solution, I'll probably end up reverting this.

|| (context.isWithinCopyTo() == false && context.isCopyToDestinationField(mapper.fullPath())))) {
context = context.addIgnoredFieldFromContext(
IgnoredSourceFieldMapper.NameValue.fromContext(context, fieldMapper.fullPath(), null)
Expand Down
6 changes: 4 additions & 2 deletions server/src/main/java/org/elasticsearch/index/mapper/DynamicFieldsBuilder.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -353,7 +353,8 @@ public boolean newDynamicLongField(DocumentParserContext context, String name) t
ScriptCompiler.NONE,
context.indexSettings().getSettings(),
context.indexSettings().getIndexVersionCreated(),
context.indexSettings().getMode()
context.indexSettings().getMode(),
context.indexSettings().sourceKeepMode()
),
context
);
Expand All @@ -371,7 +372,8 @@ public boolean newDynamicDoubleField(DocumentParserContext context, String name)
ScriptCompiler.NONE,
context.indexSettings().getSettings(),
context.indexSettings().getIndexVersionCreated(),
context.indexSettings().getMode()
context.indexSettings().getMode(),
context.indexSettings().sourceKeepMode()
),
context
);
Expand Down
1 change: 1 addition & 0 deletions server/src/main/java/org/elasticsearch/index/mapper/FieldArrayContext.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ static String getOffsetsFieldName(
&& sourceKeepMode == Mapper.SourceKeepMode.ARRAYS
&& hasDocValues
&& isStored == false
&& context.isInNestedContext() == false
Copy link
Member

@martijnvg martijnvg Mar 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes sense. Can you add a test to NativeArrayIntegrationTestCase, which checks that if leaf array field has nested parent field, then we always fall back to ignored source?

&& fieldMapperBuilder.copyTo.copyToFields().isEmpty()
&& fieldMapperBuilder.multiFieldsBuilder.hasMultiFields() == false
&& indexVersionSupportStoringArraysNatively(indexCreatedVersion, minSupportedVersionMain)) {
Expand Down
Loading