Add object param for keeping synthetic source #113690

kkrik-es · 2024-09-27T14:04:02Z

Add per-object param for controlling source recording, in line with #112706 for leaf fields.

Document how to keep synthetic source per field, object or index.

Related to #112012

elasticsearchmachine · 2024-09-27T14:04:26Z

Hi @kkrik-es, I've created a changelog YAML for you.

…am' into synthetic-source/keep-object-param

# Conflicts: # server/src/main/java/org/elasticsearch/index/mapper/ObjectMapper.java # server/src/test/java/org/elasticsearch/index/mapper/DocumentParserTests.java # server/src/test/java/org/elasticsearch/index/mapper/ObjectMapperTests.java

elasticsearchmachine · 2024-09-30T14:55:06Z

Pinging @elastic/es-storage-engine (Team:StorageEngine)

martijnvg

LGTM - I left a comment about what we say we guarantee with synthetic source keep arrays.

martijnvg · 2024-10-02T13:25:25Z

docs/reference/mapping/fields/synthetic-source.asciidoc

+====== Keeping the origical source
+
+It is possible to record the original source of an object or field, at extra storage cost, using param
+`synthetic_source_keep`. The default value is `none`; setting it to `arrays` leads to storing the original source for


Maybe we shouldn't guarantee that we store original source when synthetic_source_keep is set to arrays is enabled? Maybe we we should just guarantee the array ordering is guaranteed? This gives us better ways to reduce the overhead later on without breaking bwc.

If we need to main bwc with things like field: [1, 2, [3], [[4, [5]]], ["5"], 6] , then that makes things more difficult. Ideally we would synthesize this as: field: [1, 2, 3, 5, 5, 6] (assuming this is a number field).

But this is how it's implemented right now. Are you suggesting that we don't document the current behavior? I think it'll be a breaking change if we change it to return [1, 2, 3, 5, 5, 6] in the example above, whether we document it or not..

I guess we can be explicit that we don't guarantee the exact form.. is this what you have in mind?

I guess we can be explicit that we don't guarantee the exact form.. is this what you have in mind?

Apologies, yes, this is what I mean.

I've updated the wording to reflect that this may change, ptal.

# Conflicts: # server/src/main/java/org/elasticsearch/index/mapper/DocumentParser.java

This reverts commit 18dc913.

This reverts commit f4ddb0e.

martijnvg · 2024-10-03T14:57:01Z

...pi-spec/src/yamlRestTest/resources/rest-api-spec/test/indices.create/20_synthetic_source.yml

 subobjects auto:
  - requires:
-      cluster_features: ["mapper.subobjects_auto"]
+      cluster_features: ["mapper.subobjects_auto", "mapper.bwc_workaround_9_0"]


Would adding gte_v8.16.0 as cluster feature avoid adding the new mapper.bwc_workaround_9_0 cluster feature?

If added in skip, it will disable the test in main. If added here, it won't fix the test failures.

martijnvg · 2024-10-03T14:59:18Z

docs/reference/mapping/fields/synthetic-source.asciidoc

+This is controlled through param `synthetic_source_keep` with the following option:
+
+ - `none`: synthetic source diverges from the original source as described above (default).
+ - `arrays`: arrays of the corresponding field or object preserve the original element ordering and duplicate elements.


elasticsearchmachine · 2024-10-03T18:19:27Z

💔 Backport failed

The backport operation could not be completed due to the following error:

An unexpected error occurred when attempting to backport this PR.

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 113690

kkrik-es · 2024-10-03T18:36:08Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Questions ?

Please refer to the Backport tool documentation

* Add object param for keeping synthetic source * Update docs/changelog/113690.yaml * fix merging * add tests * merge * fix randomized tests * add documentation * dedup id in docs * update documentation * update documentation * fix bwc * fix bwc * fix unintended * Revert "fix bwc" This reverts commit 18dc913. * Revert "fix bwc" This reverts commit f4ddb0e. * add missing test * fix transform * fix transform * fix transform * fix transform * fix transform (cherry picked from commit dd20248) # Conflicts: # rest-api-spec/build.gradle

* Add object param for keeping synthetic source (#113690) * Add object param for keeping synthetic source * Update docs/changelog/113690.yaml * fix merging * add tests * merge * fix randomized tests * add documentation * dedup id in docs * update documentation * update documentation * fix bwc * fix bwc * fix unintended * Revert "fix bwc" This reverts commit 18dc913. * Revert "fix bwc" This reverts commit f4ddb0e. * add missing test * fix transform * fix transform * fix transform * fix transform * fix transform (cherry picked from commit dd20248) # Conflicts: # rest-api-spec/build.gradle * Update build.gradle * Update MapperFeatures.java * Update 20_synthetic_source.yml * Update 21_synthetic_source_stored.yml * Update 21_synthetic_source_stored.yml * Update 21_synthetic_source_stored.yml * Update 21_synthetic_source_stored.yml

* Add object param for keeping synthetic source * Update docs/changelog/113690.yaml * fix merging * add tests * merge * fix randomized tests * add documentation * dedup id in docs * update documentation * update documentation * fix bwc * fix bwc * fix unintended * Revert "fix bwc" This reverts commit 18dc913. * Revert "fix bwc" This reverts commit f4ddb0e. * add missing test * fix transform * fix transform * fix transform * fix transform * fix transform

Add object param for keeping synthetic source

77277c2

kkrik-es added >enhancement auto-backport-and-merge Team:StorageEngine :StorageEngine/Mapping The storage related side of mappings v8.16.0 labels Sep 27, 2024

kkrik-es self-assigned this Sep 27, 2024

elasticsearchmachine added the v9.0.0 label Sep 27, 2024

Update docs/changelog/113690.yaml

468d250

kkrik-es added 8 commits September 27, 2024 22:13

Merge branch 'main' into synthetic-source/keep-object-param

efa3c77

fix merging

3b30147

Merge remote-tracking branch 'origin/synthetic-source/keep-object-par…

d0d31b5

…am' into synthetic-source/keep-object-param

Merge branch 'main' into synthetic-source/keep-object-param

3ee3a86

add tests

16c9865

merge

51a5f71

fix randomized tests

6dcdf44

kkrik-es marked this pull request as ready for review September 30, 2024 14:54

kkrik-es requested a review from a team as a code owner September 30, 2024 14:54

kkrik-es requested review from lkts and martijnvg and removed request for a team and lkts September 30, 2024 14:54

kkrik-es added 4 commits October 1, 2024 11:35

Merge branch 'refs/heads/main' into synthetic-source/keep-object-param

9d8232c

add documentation

b8b85c7

dedup id in docs

ee047bc

Merge branch 'main' into synthetic-source/keep-object-param

aea0cc5

martijnvg approved these changes Oct 2, 2024

View reviewed changes

kkrik-es added 9 commits October 3, 2024 13:38

Merge branch 'main' into synthetic-source/keep-object-param

58fd5b9

# Conflicts: # server/src/main/java/org/elasticsearch/index/mapper/DocumentParser.java

update documentation

c214eb4

update documentation

97b6c01

fix bwc

f4ddb0e

fix bwc

18dc913

fix unintended

b5f1d71

Revert "fix bwc"

432757f

This reverts commit 18dc913.

Revert "fix bwc"

771bc38

This reverts commit f4ddb0e.

add missing test

895e08f

kkrik-es added auto-backport Automatically create backport pull requests when merged and removed auto-backport-and-merge labels Oct 3, 2024

martijnvg reviewed Oct 3, 2024

View reviewed changes

fix transform

d3b5c30

kkrik-es requested a review from a team as a code owner October 3, 2024 15:14

kkrik-es and others added 5 commits October 3, 2024 18:45

fix transform

de651f8

fix transform

6887504

fix transform

92d5c1b

fix transform

c4d0111

Merge branch 'elastic:main' into synthetic-source/keep-object-param

f9d0330

lkts approved these changes Oct 3, 2024

View reviewed changes

kkrik-es merged commit dd20248 into elastic:main Oct 3, 2024
16 checks passed

elasticsearchmachine added the backport pending label Oct 3, 2024

kkrik-es mentioned this pull request Oct 3, 2024

[8.x] Add object param for keeping synthetic source (#113690) #114058

Merged

Add object param for keeping synthetic source #113690

Add object param for keeping synthetic source #113690

Uh oh!

Conversation

kkrik-es commented Sep 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Sep 27, 2024

Uh oh!

elasticsearchmachine commented Sep 30, 2024

Uh oh!

martijnvg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kkrik-es Oct 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Oct 3, 2024

💔 Backport failed

Uh oh!

kkrik-es commented Oct 3, 2024

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kkrik-es commented Sep 27, 2024 •

edited

Loading

kkrik-es Oct 3, 2024 •

edited

Loading