Add sub-field support to flattened field type by parkertimmins · Pull Request #144451 · elastic/elasticsearch

parkertimmins · 2026-03-17T22:06:08Z

The flattened field type indexes all leaf values as untyped keywords, preventing type-aware operations (range queries, date math, numeric aggregations) on individual keys. Users needing typed behavior must switch the entire object to object type/

A new optional properties parameter lets users declare specific paths with real leaf field types while leaving the rest on the default untyped flattened path.

  {
    "labels": {
      "type": "flattened",
      "properties": {
        "host.name": { "type": "keyword" },
        "status_code": { "type": "long" }
      }
    }
  }

At index time, values matching a mapped property are delegated to that sub-field's mapper and excluded from the root/keyed flattened fields. Unmapped keys continue to behave as normal flattened keywords. Allowed sub-field types: keyword, constant_keyword, wildcard, text, long, integer, short, byte, double, float, half_float, scaled_float, unsigned_long, date, date_nanos, boolean, ip. Supported operations: typed search, sort (including index sort), aggregations, ESQL block loading, and synthetic _source. Restrictions: copy_to and fields (multi-fields) are disallowed on mapped properties. Only leaf types from the allow-list are permitted.

Made-with: Cursor

Allow specific keys within a flattened field to be mapped as typed sub-fields (keyword, ip, etc.) via a new "properties" mapping attribute. Mapped keys are indexed exclusively through their sub-field mapper and excluded from the flattened field's root/keyed representation. Made-with: Cursor

github-actions · 2026-03-17T22:08:35Z

🔍 Preview links for changed docs

docs/reference/elasticsearch/mapping-reference/flattened.md

github-actions · 2026-03-17T22:08:36Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

github-actions · 2026-03-17T22:11:06Z

Vale Linting Results

Summary: 1 suggestion found

💡 Suggestions (1)

File	Line	Rule	Message
docs/reference/elasticsearch/mapping-reference/flattened.md	255	Elastic.WordChoice	Consider using 'can, might' instead of 'may', unless the term is in the UI.

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

Tighten field count assertions to exact values, add test for mapped properties matched via nested object notation, simplify serialization roundtrip test, and fix minor doc formatting. Made-with: Cursor

Made-with: Cursor # Conflicts: # server/src/main/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldMapper.java # server/src/main/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldParser.java # server/src/test/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldParserTests.java

Add search tests covering term queries, sorting, aggregations, doc value field loading, and numeric range queries on mapped sub-fields within flattened fields. Made-with: Cursor

Extend FlattenedDocValuesSyntheticFieldLoader to compose mapped property loaders alongside the flattened field's own keyed doc values, so mapped properties are included in synthetic source. Made-with: Cursor

Register mapper.flattened.mapped_properties cluster feature and add REST tests covering queries, sorting, aggregations, synthetic source, and mapping serialization for mapped properties. Made-with: Cursor

Reset docValues to NO_VALUES in the binary doc values branch of FlattenedDocValuesSyntheticFieldLoader when no binary DVs exist for a segment, preventing stale state from a prior segment. Forward null tokens to mapped property mappers in FlattenedFieldParser so sub-field null_value handling works independently of the parent flattened field's null_value. Made-with: Cursor

Reject copy_to and multi_fields on flattened mapped properties since multi-fields are silently un-queryable and copy_to is inconsistent with the parent flattened field's restrictions. Add tests for multi-value arrays, null value forwarding, exists queries, property preservation on merge, additional types, empty strings, ignore_above/depth_limit interaction, cross-index queries, synthetic source ordering, and all disallowed types. Made-with: Cursor

rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search/340_flattened.yml

The RootFlattenedDocValuesBlockLoader was not passing mapped property loaders through to FlattenedDocValuesSyntheticFieldLoader, causing ES|QL block loading to omit all mapped property values from the resulting JSON. Also fixed writeToBlock to use hasValue() so it does not emit null when only mapped properties have data. Made-with: Cursor

elasticsearchmachine · 2026-03-18T22:56:45Z

Hi @parkertimmins, I've created a changelog YAML for you.

Replace LinkedHashMap with TreeMap for propertyBuilders and mappedProperties, and use unmodifiableSortedMap to preserve sort order through Map.copyOf. This removes the need to wrap in new TreeMap<>() at each usage site. Made-with: Cursor

Made-with: Cursor

elasticsearchmachine · 2026-03-18T23:47:23Z

Pinging @elastic/es-storage-engine (Team:StorageEngine)

The test relied on unsorted search returning results in index sort order, which fails when CCS searches a replica that is still being peer-recovered from the primary. Made-with: Cursor

Flattened type cannot be used as a multi-field since the feature branch changed its TypeParser from FieldMapper.TypeParser to Mapper.TypeParser. Use object properties instead. Made-with: Cursor

Made-with: Cursor

mouhc1ne · 2026-03-20T20:29:22Z

x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/260_flattened_subfield.yml

          # [foo] is flattened in index5
-          # [bar] is keyword in index5
-          # [bar].[baz] is flattened in index5
+          # [bar.baz] is flattened in index5


Not super necessary but I think I'd be nice to tell immediately, just by looking at comments like these, if foo.bar was itself the field name or bar is a field under foo ([foo].[bar]). Seems like it's still the latter case in index5.

@mouhc1ne Thanks for the review. Unfortunately, we ended up needing the keep the NOOP behavior that currently exists, because changing it would be a breaking change. Martijn pointed out that, because the flattened mapping is serialized, not being backwards compatible here would cause failures during a shard upgrade.

mouhc1ne · 2026-03-23T13:52:16Z

Just fyi that you might need to pull in #144741 in case you run into CI errors in 260_flattened_subfield.yml related to load not allowed.

The feature branch changed FlattenedFieldMapper.PARSER from a FieldMapper.TypeParser to a Mapper.TypeParser to handle the new properties key. This broke the instanceof check in TypeParsers.parseMultiField, rejecting flattened as a multi-field. Fix by subclassing FieldMapper.TypeParser instead: extract properties from the node before delegating to super.parse(). This preserves backwards compatibility while supporting the new properties parsing. Made-with: Cursor

server/src/main/java/org/elasticsearch/index/mapper/FieldMapper.java

docs/reference/elasticsearch/mapping-reference/flattened.md

jordan-powers · 2026-03-24T18:30:31Z

server/src/test/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldMapperTests.java

+        });
+    }
+
+    public void testBlockLoaderWithMappedPropertiesOnly() throws IOException {


From what I can tell, these two blockloader tests are testing the root blockloader. We should probably add a test for the KeyedFlattenedDocValuesBlockLoader too.

Good point, I've added a test here: https://github.com/elastic/elasticsearch/pull/144451/changes#diff-7113916e741652f4a9e73b8cf5573b3af612664e2c54e22cbc697b49468a8886R1760

This brings up an important question. Should the mapped values be included in the KeyedFlattenedDocValuesBlockLoader? Currently they are not. They have to be obtained through a separate, and correctly typed, blockloader. I can also see the argument that they should included, though this would involve casting everything to strings. For this reason, I'm inclined to say they should be left out.

@jordan-powers How will this choice effect query time behavior ES|QL. @martijnvg Any thoughts?

Good question. Not really related, but synthetic source does include the values of the mapped sub fields. The root block loader's purpose is to include all the content of a flattened field. So I think the root block loader should also include the content of mapped sub fields.

Maybe we can quickly implement this by falling back to the source based block loader if there are mapped sub fields (in RootFlattenedFieldType#blockLoader(...))? Which should do the right thing, given that synthetic source already has this behaviour.

Then in a follow up we can improve RootFlattenedDocValuesBlockLoader?

I misunderstood the question :) Never mind my previous comment.

Should the mapped values be included in the KeyedFlattenedDocValuesBlockLoader?

No, these fields have their own block loader via their mapped sub field.

How will this choice affect query time behavior in ES|QL?

So long as the sub-field mappers can be resolved by FieldTypeLookup#get, then the fields will be loaded with the correct blockloaders.

This reverts commit d2313f1.

Switch PARSER from a manual Mapper.TypeParser to FieldMapper.TypeParser via createTypeParserWithLegacySupport, allowing flattened fields as multi-fields for bwc. The "properties" field is now a Parameter<Map<String, Builder>> with parsing handled by the standard parameter framework. Made-with: Cursor

Made-with: Cursor

martijnvg

LGTM2

Use explicit sort: _doc to ensure deterministic result order after force merge, avoiding reliance on undefined tiebreaker behavior with equal _score values. Made-with: Cursor

elasticsearchmachine added the v9.4.0 label Mar 17, 2026

parkertimmins added 9 commits March 17, 2026 17:35

Improve flattened properties test assertions

a53c321

Tighten field count assertions to exact values, add test for mapped properties matched via nested object notation, simplify serialization roundtrip test, and fix minor doc formatting. Made-with: Cursor

Add integration tests for flattened mapped properties

ae4afd9

Add search tests covering term queries, sorting, aggregations, doc value field loading, and numeric range queries on mapped sub-fields within flattened fields. Made-with: Cursor

Support synthetic source for flattened mapped properties

5632999

Extend FlattenedDocValuesSyntheticFieldLoader to compose mapped property loaders alongside the flattened field's own keyed doc values, so mapped properties are included in synthetic source. Made-with: Cursor

Add YAML REST tests for flattened mapped properties

4064d04

Register mapper.flattened.mapped_properties cluster feature and add REST tests covering queries, sorting, aggregations, synthetic source, and mapping serialization for mapped properties. Made-with: Cursor

Change XContent builders to json string

1d496cb

Dont' change existing tests to keep diff smaller

bcfbb9c

parkertimmins commented Mar 18, 2026

View reviewed changes

rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search/340_flattened.yml Show resolved Hide resolved

parkertimmins added >enhancement :StorageEngine/Mapping The storage related side of mappings labels Mar 18, 2026

Update docs/changelog/144451.yaml

a1ac0a8

parkertimmins and others added 4 commits March 18, 2026 17:57

Merge branch 'main' into parker/flattened-sub-fields

c248681

Extract toPropertyLoaders helper for mapped properties

7a3dec6

Made-with: Cursor

[CI] Auto commit changes from spotless

b29b0e8

parkertimmins marked this pull request as ready for review March 18, 2026 23:47

parkertimmins requested a review from a team as a code owner March 18, 2026 23:47

parkertimmins requested review from jordan-powers and martijnvg March 18, 2026 23:47

elasticsearchmachine added the Team:StorageEngine label Mar 18, 2026

parkertimmins and others added 6 commits March 20, 2026 12:10

Fix flaky index sort test in CCS suite

18c516f

The test relied on unsorted search returning results in index sort order, which fails when CCS searches a replica that is still being peer-recovered from the primary. Made-with: Cursor

remove forbidden api

497bab3

[CI] Auto commit changes from spotless

9712db5

Fix flattened subfield YAML test mapping

eabfa99

Flattened type cannot be used as a multi-field since the feature branch changed its TypeParser from FieldMapper.TypeParser to Mapper.TypeParser. Use object properties instead. Made-with: Cursor

Add test for flattened as multi-field rejection

ad58602

Made-with: Cursor

Move multivalued test to FlattenedMapperTests

706406e

mouhc1ne reviewed Mar 23, 2026

View reviewed changes

parkertimmins commented Mar 23, 2026

View reviewed changes

server/src/main/java/org/elasticsearch/index/mapper/FieldMapper.java Show resolved Hide resolved

jordan-powers reviewed Mar 23, 2026

View reviewed changes

docs/reference/elasticsearch/mapping-reference/flattened.md Show resolved Hide resolved

parkertimmins added 2 commits March 24, 2026 08:58

Merge branch 'main' into parker/flattened-sub-fields

bb2e4d7

update docs to mention that copy_to and fields are disallowed

f5df0f2

jordan-powers reviewed Mar 24, 2026

View reviewed changes

parkertimmins added 3 commits March 24, 2026 13:59

Revert "Allow flattened type as multi-field of another field"

24074eb

This reverts commit d2313f1.

simplify property merge logic

8ece334

susan-shu-c mentioned this pull request Mar 24, 2026

[RFC 0052] Stage 2: Update additional GenAI fields elastic/ecs#2532

Open

parkertimmins added 5 commits March 24, 2026 20:57

Add KeyedFlattenedDocValuesBlockLoader tests

545512a

Made-with: Cursor

Fix incorrect assertion in yaml

24608c9

Remove settings disallowed in serverless from yaml test

f11b7bd

Merge branch 'main' into parker/flattened-sub-fields

b7c3f00

Merge branch 'main' into parker/flattened-sub-fields

7cac114

martijnvg approved these changes Mar 26, 2026

View reviewed changes

parkertimmins added 2 commits March 26, 2026 09:22

Fix flaky index sort yaml test for flattened fields

1412da9

Use explicit sort: _doc to ensure deterministic result order after force merge, avoiding reliance on undefined tiebreaker behavior with equal _score values. Made-with: Cursor

refresh after force merge in sort yaml test

e3f7e98

elasticsearchmachine added the serverless-linked Added by automation, don't add manually label Mar 26, 2026

Merge branch 'main' into parker/flattened-sub-fields

d6bc406

parkertimmins merged commit fee6b18 into elastic:main Mar 26, 2026
36 checks passed

parkertimmins deleted the parker/flattened-sub-fields branch March 26, 2026 22:30

Conversation

parkertimmins commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

github-actions bot commented Mar 17, 2026

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

Uh oh!

github-actions bot commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Vale Linting Results

Uh oh!

Uh oh!

elasticsearchmachine commented Mar 18, 2026

Uh oh!

elasticsearchmachine commented Mar 18, 2026

Uh oh!

mouhc1ne Mar 20, 2026

Choose a reason for hiding this comment

Uh oh!

parkertimmins Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

mouhc1ne commented Mar 23, 2026

Uh oh!

Uh oh!

Uh oh!

jordan-powers Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

parkertimmins Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

martijnvg Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

martijnvg Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

jordan-powers Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

martijnvg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

parkertimmins commented Mar 17, 2026 •

edited

Loading

github-actions bot commented Mar 17, 2026 •

edited

Loading

github-actions bot commented Mar 17, 2026 •

edited

Loading