Add object param for keeping synthetic source

docs/changelog/113690.yaml

@@ -0,0 +1,5 @@
+pr: 113690
+summary: Add object param for keeping synthetic source
+area: Mapping
+type: enhancement
+issues: []

docs/reference/mapping/fields/synthetic-source.asciidoc

@@ -32,18 +32,25 @@ space. Additional latency can be avoided by not loading `_source` field in queri
 
 [[synthetic-source-fields]]
 ===== Supported fields
-Synthetic `_source` is supported by all field types. Depending on implementation details, field types have different properties when used with synthetic `_source`.
+Synthetic `_source` is supported by all field types. Depending on implementation details, field types have different
+properties when used with synthetic `_source`.
 
-<<synthetic-source-fields-native-list, Most field types>> construct synthetic `_source` using existing data, most commonly <<doc-values,`doc_values`>> and <<stored-fields, stored fields>>. For these field types, no additional space is needed to store the contents of `_source` field. Due to the storage layout of <<doc-values,`doc_values`>>, the generated `_source` field undergoes <<synthetic-source-modifications, modifications>> compared to original document.
+<<synthetic-source-fields-native-list, Most field types>> construct synthetic `_source` using existing data, most
+commonly <<doc-values,`doc_values`>> and <<stored-fields, stored fields>>. For these field types, no additional space
+is needed to store the contents of `_source` field. Due to the storage layout of <<doc-values,`doc_values`>>, the
+generated `_source` field undergoes <<synthetic-source-modifications, modifications>> compared to original document.
 
-For all other field types, the original value of the field is stored as is, in the same way as the `_source` field in non-synthetic mode. In this case there are no modifications and field data in `_source` is the same as in the original document. Similarly, malformed values of fields that use <<ignore-malformed,`ignore_malformed`>> or <<ignore-above,`ignore_above`>> need to be stored as is. This approach is less storage efficient since data needed for `_source` reconstruction is stored in addition to other data required to index the field (like `doc_values`).
+For all other field types, the original value of the field is stored as is, in the same way as the `_source` field in
+non-synthetic mode. In this case there are no modifications and field data in `_source` is the same as in the original
+document. Similarly, malformed values of fields that use <<ignore-malformed,`ignore_malformed`>> or
+<<ignore-above,`ignore_above`>> need to be stored as is. This approach is less storage efficient since data needed for
+`_source` reconstruction is stored in addition to other data required to index the field (like `doc_values`).
 
 [[synthetic-source-restrictions]]
 ===== Synthetic `_source` restrictions
 
-Synthetic `_source` cannot be used together with field mappings that use <<copy-to,`copy_to`>>. 
-
-Some field types have additional restrictions. These restrictions are documented in the **synthetic `_source`** section of the field type's <<mapping-types,documentation>>.
+Some field types have additional restrictions. These restrictions are documented in the **synthetic `_source`** section
+of the field type's <<mapping-types,documentation>>.
 
 [[synthetic-source-modifications]]
 ===== Synthetic `_source` modifications
@@ -144,6 +151,42 @@ Will become:
 ----
 // TEST[s/^/{"_source":/ s/\n$/}/]
 
+This impacts how source contents can be referenced in <<modules-scripting-using,scripts>>. For instance, referencing
+a script in its original source form will return null:
+
+[source,js]
+----
+"script": { "source": """  emit(params._source['foo.bar.baz'])  """ }
+----
+// NOTCONSOLE
+
+Instead, source references need to be in line with the mapping structure:
+
+[source,js]
+----
+"script": { "source": """  emit(params._source['foo']['bar']['baz'])  """ }
+----
+// NOTCONSOLE
+
+or simply
+
+[source,js]
+----
+"script": { "source": """  emit(params._source.foo.bar.baz)  """ }
+----
+// NOTCONSOLE
+
+The following <<modules-scripting-fields, field APIs>> are preferable as, in addition to being agnostic to the
+mapping structure, they make use of docvalues if available and fall back to synthetic source only when needed. This
+reduces source synthesizing, a slow and costly operation.
+
+[source,js]
+----
+"script": { "source": """  emit(field('foo.bar.baz').get(null))   """ }
+"script": { "source": """  emit($('foo.bar.baz', null))   """ }
+----
+// NOTCONSOLE
+
 [[synthetic-source-modifications-alphabetical]]
 ====== Alphabetical sorting
 Synthetic `_source` fields are sorted alphabetically. The
@@ -155,18 +198,99 @@ that ordering.
 
 [[synthetic-source-modifications-ranges]]
 ====== Representation of ranges
-Range field values (e.g. `long_range`) are always represented as inclusive on both sides with bounds adjusted accordingly. See <<range-synthetic-source-inclusive, examples>>.
+Range field values (e.g. `long_range`) are always represented as inclusive on both sides with bounds adjusted
+accordingly. See <<range-synthetic-source-inclusive, examples>>.
 
 [[synthetic-source-precision-loss-for-point-types]]
 ====== Reduced precision of `geo_point` values
-Values of `geo_point` fields are represented in synthetic `_source` with reduced precision. See <<geo-point-synthetic-source, examples>>.
+Values of `geo_point` fields are represented in synthetic `_source` with reduced precision. See
+<<geo-point-synthetic-source, examples>>.
+
+[[synthetic-source-keep]]
+====== Minimizing source modifications
+
+It is possible to avoid synthetic source modifications for a particular object or field, at extra storage cost.
+This is controlled through param `synthetic_source_keep` with the following option:
+
+ - `none`: synthetic source diverges from the original source as described above (default).
+ - `arrays`: arrays of the corresponding field or object preserve the original element ordering and duplicate elements.
+The synthetic source fragment for such arrays is not guaranteed to match the original source exactly, e.g. array
+`[1, 2, [5], [[4, [3]]], 5]` may appear as-is or in an equivalent format like `[1, 2, 5, 4, 3, 5]`. The exact format
+may change in the future, in an effort to reduce the storage overhead of this option.
+- `all`: the source for both singleton instances and arrays of the corresponding field or object gets recorded. When
+applied to objects, the source of all sub-objects and sub-fields gets captured. Furthermore, the original source of
+arrays gets captured and appears in synthetic source with no modifications.
+
+For instance:
+
+[source,console,id=create-index-with-synthetic-source-keep]
+----
+PUT idx_keep
+{
+  "mappings": {
+    "_source": {
+      "mode": "synthetic"
+    },
+    "properties": {
+      "path": {
+        "type": "object",
+        "synthetic_source_keep": "all"
+      },
+      "ids": {
+        "type": "integer",
+        "synthetic_source_keep": "arrays"
+      }
+    }
+  }
+}
+----
+// TEST
+
+[source,console,id=synthetic-source-keep-example]
+----
+PUT idx_keep/_doc/1
+{
+  "path": {
+    "to": [
+      { "foo": [3, 2, 1] },
+      { "foo": [30, 20, 10] }
+    ],
+    "bar": "baz"
+  },
+  "ids": [ 200, 100, 300, 100 ]
+}
+----
+// TEST[s/$/\nGET idx_keep\/_doc\/1?filter_path=_source\n/]
+
+returns the original source, with no array deduplication and sorting:
+
+[source,console-result]
+----
+{
+  "path": {
+    "to": [
+      { "foo": [3, 2, 1] },
+      { "foo": [30, 20, 10] }
+    ],
+    "bar": "baz"
+  },
+  "ids": [ 200, 100, 300, 100 ]
+}
+----
+// TEST[s/^/{"_source":/ s/\n$/}/]
 
+The option for capturing the source of arrays can be applied at index level, by setting
+`index.mapping.synthetic_source_keep` to `arrays`. This applies to all objects and fields in the index, except for
+the ones with explicit overrides of `synthetic_source_keep` set to `none`. In this case, the storage overhead grows
+with the number and sizes of arrays present in source of each document, naturally.
 
 [[synthetic-source-fields-native-list]]
 ===== Field types that support synthetic source with no storage overhead
-The following field types support synthetic source using data from <<doc-values,`doc_values`>> or <<stored-fields, stored fields>>, and require no additional storage space to construct the `_source` field. 
+The following field types support synthetic source using data from <<doc-values,`doc_values`>> or
+<stored-fields, stored fields>>, and require no additional storage space to construct the `_source` field.
 
-NOTE: If you enable the <<ignore-malformed,`ignore_malformed`>> or <<ignore-above,`ignore_above`>> settings, then additional storage is required to store ignored field values for these types.
+NOTE: If you enable the <<ignore-malformed,`ignore_malformed`>> or <<ignore-above,`ignore_above`>> settings, then
+additional storage is required to store ignored field values for these types.
 
 ** <<aggregate-metric-double-synthetic-source, `aggregate_metric_double`>>
 ** {plugins}/mapper-annotated-text-usage.html#annotated-text-synthetic-source[`annotated-text`]

rest-api-spec/build.gradle

@@ -57,5 +57,5 @@ tasks.named("precommit").configure {
 tasks.named("yamlRestCompatTestTransform").configure({task ->
     task.skipTest("indices.sort/10_basic/Index Sort", "warning does not exist for compatibility")
     task.skipTest("search/330_fetch_fields/Test search rewrite", "warning does not exist for compatibility")
-    task.skipTestsByFilePattern("indices.create/synthetic_source*.yml", "@UpdateForV9 -> tests do not pass after bumping API version to 9 [ES-9597]")
+    task.skipTest("indices.create/21_synthetic_source_stored/object param - nested object with stored array", "skip test to submit #113690")
 })

rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/indices.create/20_synthetic_source.yml

@@ -895,7 +895,7 @@ doubly nested object:
 ---
 subobjects auto:
   - requires:
-      cluster_features: ["mapper.subobjects_auto"]
+      cluster_features: ["mapper.subobjects_auto", "mapper.bwc_workaround_9_0"]
       reason: requires tracking ignored source and supporting subobjects auto setting
 
   - do:
@@ -920,7 +920,7 @@ subobjects auto:
                       id:
                         type: keyword
               stored:
-                store_array_source: true
+                synthetic_source_keep: arrays
                 properties:
                   span:
                     properties: