Add object param for keeping synthetic source

docs/changelog/113690.yaml

@@ -0,0 +1,5 @@
+pr: 113690
+summary: Add object param for keeping synthetic source
+area: Mapping
+type: enhancement
+issues: []

docs/reference/mapping/fields/synthetic-source.asciidoc

@@ -32,18 +32,25 @@ space. Additional latency can be avoided by not loading `_source` field in queri
 
 [[synthetic-source-fields]]
 ===== Supported fields
-Synthetic `_source` is supported by all field types. Depending on implementation details, field types have different properties when used with synthetic `_source`.
+Synthetic `_source` is supported by all field types. Depending on implementation details, field types have different
+properties when used with synthetic `_source`.
 
-<<synthetic-source-fields-native-list, Most field types>> construct synthetic `_source` using existing data, most commonly <<doc-values,`doc_values`>> and <<stored-fields, stored fields>>. For these field types, no additional space is needed to store the contents of `_source` field. Due to the storage layout of <<doc-values,`doc_values`>>, the generated `_source` field undergoes <<synthetic-source-modifications, modifications>> compared to original document.
+<<synthetic-source-fields-native-list, Most field types>> construct synthetic `_source` using existing data, most
+commonly <<doc-values,`doc_values`>> and <<stored-fields, stored fields>>. For these field types, no additional space
+is needed to store the contents of `_source` field. Due to the storage layout of <<doc-values,`doc_values`>>, the
+generated `_source` field undergoes <<synthetic-source-modifications, modifications>> compared to original document.
 
-For all other field types, the original value of the field is stored as is, in the same way as the `_source` field in non-synthetic mode. In this case there are no modifications and field data in `_source` is the same as in the original document. Similarly, malformed values of fields that use <<ignore-malformed,`ignore_malformed`>> or <<ignore-above,`ignore_above`>> need to be stored as is. This approach is less storage efficient since data needed for `_source` reconstruction is stored in addition to other data required to index the field (like `doc_values`).
+For all other field types, the original value of the field is stored as is, in the same way as the `_source` field in
+non-synthetic mode. In this case there are no modifications and field data in `_source` is the same as in the original
+document. Similarly, malformed values of fields that use <<ignore-malformed,`ignore_malformed`>> or
+<<ignore-above,`ignore_above`>> need to be stored as is. This approach is less storage efficient since data needed for
+`_source` reconstruction is stored in addition to other data required to index the field (like `doc_values`).
 
 [[synthetic-source-restrictions]]
 ===== Synthetic `_source` restrictions
 
-Synthetic `_source` cannot be used together with field mappings that use <<copy-to,`copy_to`>>. 
-
-Some field types have additional restrictions. These restrictions are documented in the **synthetic `_source`** section of the field type's <<mapping-types,documentation>>.
+Some field types have additional restrictions. These restrictions are documented in the **synthetic `_source`** section
+of the field type's <<mapping-types,documentation>>.
 
 [[synthetic-source-modifications]]
 ===== Synthetic `_source` modifications
@@ -144,6 +151,42 @@ Will become:
 ----
 // TEST[s/^/{"_source":/ s/\n$/}/]
 
+This impacts how source contents can be referenced in <<modules-scripting-using,scripts>>. For instance, referencing
+a script in its original source form will return null:
+
+[source,js]
+----
+"script": { "source": """  emit(params._source['foo.bar.baz'])  """ }
+----
+// NOTCONSOLE
+
+Instead, source references need to be in line with the mapping structure:
+
+[source,js]
+----
+"script": { "source": """  emit(params._source['foo']['bar']['baz'])  """ }
+----
+// NOTCONSOLE
+
+or simply
+
+[source,js]
+----
+"script": { "source": """  emit(params._source.foo.bar.baz)  """ }
+----
+// NOTCONSOLE
+
+The following <<modules-scripting-fields, field APIs>> are preferable as, in addition to being agnostic to the
+mapping structure, they make use of docvalues if available and fall back to synthetic source only when needed. This
+reduces source synthesizing, a slow and costly operation.
+
+[source,js]
+----
+"script": { "source": """  emit(field('foo.bar.baz').get(null))   """ }
+"script": { "source": """  emit($('foo.bar.baz', null))   """ }
+----
+// NOTCONSOLE
+
 [[synthetic-source-modifications-alphabetical]]
 ====== Alphabetical sorting
 Synthetic `_source` fields are sorted alphabetically. The
@@ -155,18 +198,91 @@ that ordering.
 
 [[synthetic-source-modifications-ranges]]
 ====== Representation of ranges
-Range field values (e.g. `long_range`) are always represented as inclusive on both sides with bounds adjusted accordingly. See <<range-synthetic-source-inclusive, examples>>.
+Range field values (e.g. `long_range`) are always represented as inclusive on both sides with bounds adjusted
+accordingly. See <<range-synthetic-source-inclusive, examples>>.
 
 [[synthetic-source-precision-loss-for-point-types]]
 ====== Reduced precision of `geo_point` values
-Values of `geo_point` fields are represented in synthetic `_source` with reduced precision. See <<geo-point-synthetic-source, examples>>.
+Values of `geo_point` fields are represented in synthetic `_source` with reduced precision. See
+<<geo-point-synthetic-source, examples>>.
+
+[[synthetic-source-keep]]
+====== Keeping the origical source
+
+It is possible to record the original source of an object or field, at extra storage cost, using param
+`synthetic_source_keep`. The default value is `none`; setting it to `arrays` leads to storing the original source for
+arrays of the corresponding field or object, while setting it to `all` records both singleton instances and arrays of
+the object or value it is applied to. When applied to objects, the source of all sub-objects and sub-fields gets
+captured. For instance:
+
+[source,console,id=create-index-with-synthetic-source-keep]
+----
+PUT idx_keep
+{
+  "mappings": {
+    "_source": {
+      "mode": "synthetic"
+    },
+    "properties": {
+      "path": {
+        "type": "object",
+        "synthetic_source_keep": "all"
+      },
+      "ids": {
+        "type": "integer",
+        "synthetic_source_keep": "arrays"
+      }
+    }
+  }
+}
+----
+// TEST
+
+[source,console,id=synthetic-source-keep-example]
+----
+PUT idx_keep/_doc/1
+{
+  "path": {
+    "to": [
+      { "foo": [3, 2, 1] },
+      { "foo": [30, 20, 10] }
+    ],
+    "bar": "baz"
+  },
+  "ids": [ 200, 100, 300, 100 ]
+}
+----
+// TEST[s/$/\nGET idx_keep\/_doc\/1?filter_path=_source\n/]
+
+returns the original source, with no array deduplication and sorting:
+
+[source,console-result]
+----
+{
+  "path": {
+    "to": [
+      { "foo": [3, 2, 1] },
+      { "foo": [30, 20, 10] }
+    ],
+    "bar": "baz"
+  },
+  "ids": [ 200, 100, 300, 100 ]
+}
+----
+// TEST[s/^/{"_source":/ s/\n$/}/]
 
+The option for capturing the source of arrays can be applied at index level, by setting
+`index.mapping.synthetic_source_keep` to `arrays`. This applies to all objects and fields in the index, except for
+the ones with explicit overrides of `synthetic_source_keep` set to `none`. In this case, the storage overhead grows
+with the number and sizes of arrays present in source of each document, naturally.
 
 [[synthetic-source-fields-native-list]]
 ===== Field types that support synthetic source with no storage overhead
-The following field types support synthetic source using data from <<doc-values,`doc_values`>> or <<stored-fields, stored fields>>, and require no additional storage space to construct the `_source` field. 
+The following field types support synthetic source using data from <<doc-values,`doc_values`>> or
+<stored-fields, stored fields>>, and require no additional storage space to construct the `_source` field.
 
-NOTE: If you enable the <<ignore-malformed,`ignore_malformed`>> or <<ignore-above,`ignore_above`>> settings, then additional storage is required to store ignored field values for these types.
+NOTE: If you enable the <<ignore-malformed,`ignore_malformed`>> or <<ignore-above,`ignore_above`>> settings, then
+additional storage is required to store ignored field values for these types.
 
 ** <<aggregate-metric-double-synthetic-source, `aggregate_metric_double`>>
 ** {plugins}/mapper-annotated-text-usage.html#annotated-text-synthetic-source[`annotated-text`]

rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/indices.create/20_synthetic_source.yml

@@ -912,7 +912,7 @@ subobjects auto:
                       id:
                         type: keyword
               stored:
-                store_array_source: true
+                synthetic_source_keep: arrays
                 properties:
                   span:
                     properties:

rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/indices.create/21_synthetic_source_stored.yml

@@ -1,3 +1,67 @@
+---
+object param - store complex object:
+  - requires:
+      cluster_features: ["mapper.synthetic_source_keep"]
+      reason: requires tracking ignored source
+
+  - do:
+      indices.create:
+        index: test
+        body:
+          mappings:
+            _source:
+              mode: synthetic
+            properties:
+              id:
+                type: integer
+              stored:
+                synthetic_source_keep: all
+                properties:
+                  object_array:
+                    properties:
+                      trace:
+                        type: keyword
+                  nested:
+                    type: nested
+                  kw:
+                    type: keyword
+
+  - do:
+      bulk:
+        index: test
+        refresh: true
+        body:
+          - '{ "create": { } }'
+          - '{ "id": 1, "stored": { "object_array": [ {"trace": "B"}, {"trace": "A"} ], "nested": [ {"foo": 20}, {"foo": 10} ], "kw": 100 } }'
+          - '{ "create": { } }'
+          - '{ "id": 2, "stored": { "object_array": { "trace": ["D", "C"] }, "nested": { "bar": [ 40, 30] }, "kw": 200, "baz": "2000" } }'
+          - '{ "create": { } }'
+          - '{ "id": 3, "stored": [ { "object_array": { "trace": "E" } }, { "nested": { "bar": [ 60, 50] } }, { "kw": 300 } ] }'
+
+  - do:
+      search:
+        index: test
+        sort: id
+
+  - match: { hits.hits.0._source.id: 1 }
+  - match: { hits.hits.0._source.stored.object_array.0.trace: B }
+  - match: { hits.hits.0._source.stored.object_array.1.trace: A }
+  - match: { hits.hits.0._source.stored.nested.0.foo: 20 }
+  - match: { hits.hits.0._source.stored.nested.1.foo: 10 }
+  - match: { hits.hits.0._source.stored.kw: 100 }
+
+  - match: { hits.hits.1._source.id: 2 }
+  - match: { hits.hits.1._source.stored.object_array.trace: [D, C] }
+  - match: { hits.hits.1._source.stored.nested.bar: [40, 30] }
+  - match: { hits.hits.1._source.stored.kw: 200 }
+  - match: { hits.hits.1._source.stored.baz: "2000" }
+
+  - match: { hits.hits.2._source.id: 3 }
+  - match: { hits.hits.2._source.stored.0.object_array.trace: E }
+  - match: { hits.hits.2._source.stored.1.nested.bar: [ 60, 50 ] }
+  - match: { hits.hits.2._source.stored.2.kw: 300 }
+
+
 ---
 object param - object array:
   - requires:
@@ -25,7 +89,7 @@ object param - object array:
                       id:
                         type: keyword
               stored:
-                store_array_source: true
+                synthetic_source_keep: arrays
                 properties:
                   span:
                     properties:
@@ -77,10 +141,10 @@ object param - object array within array:
               mode: synthetic
             properties:
               stored:
-                store_array_source: true
+                synthetic_source_keep: arrays
                 properties:
                   path:
-                    store_array_source: true
+                    synthetic_source_keep: arrays
                     properties:
                       to:
                         properties:
@@ -120,7 +184,7 @@ object param - no object array:
               mode: synthetic
             properties:
               stored:
-                store_array_source: true
+                synthetic_source_keep: arrays
                 properties:
                   span:
                     properties:
@@ -164,7 +228,7 @@ object param - field ordering in object array:
               a:
                 type: keyword
               b:
-                store_array_source: true
+                synthetic_source_keep: arrays
                 properties:
                   aa:
                     type: keyword
@@ -173,7 +237,7 @@ object param - field ordering in object array:
               c:
                 type: keyword
               d:
-                store_array_source: true
+                synthetic_source_keep: arrays
                 properties:
                   aa:
                     type: keyword
@@ -215,7 +279,7 @@ object param - nested object array next to other fields:
               b:
                 properties:
                   c:
-                    store_array_source: true
+                    synthetic_source_keep: arrays
                     properties:
                       aa:
                         type: keyword
@@ -272,7 +336,7 @@ object param - nested object with stored array:
                 type: nested
               nested_array_stored:
                 type: nested
-                store_array_source: true
+                synthetic_source_keep: all
 
   - do:
       bulk:
@@ -322,7 +386,7 @@ index param - nested array within array:
                   to:
                     properties:
                       some:
-                        store_array_source: true
+                        synthetic_source_keep: arrays
                         properties:
                           id:
                             type: integer
@@ -369,7 +433,7 @@ stored field under object with store_array_source:
               name:
                 type: keyword
               obj:
-                store_array_source: true
+                synthetic_source_keep: arrays
                 properties:
                   foo:
                     type: keyword
@@ -772,6 +836,9 @@ index param - root arrays:
                     properties:
                       id:
                         type: keyword
+              obj_default:
+                type: object
+                synthetic_source_keep: none
 
   - do:
       bulk:
@@ -782,6 +849,8 @@ index param - root arrays:
           - '{  "id": 1, "leaf": [30, 20, 10], "leaf_default": [30, 20, 10], "obj": [ { "trace": { "id": "a" }, "span": { "id": "1" } }, { "trace": { "id": "b" }, "span": { "id": "1" } } ] }'
           - '{ "create": { } }'
           - '{  "id": 2, "leaf": [130, 120, 110], "leaf_default": [130, 120, 110], "obj": [ { "trace": { "id": "aa" }, "span": { "id": "2" } }, { "trace": { "id": "bb" }, "span": { "id": "2" } } ] }'
+          - '{ "create": { } }'
+          - '{  "id": 3, "obj_default": [ { "trace": { "id": "bb" }, "span": { "id": "2" } }, { "trace": { "id": "aa" }, "span": { "id": "2" } } ] }'
 
   - do:
       search:
@@ -799,13 +868,17 @@ index param - root arrays:
 
   - match: { hits.hits.1._source.id: 2 }
   - match: { hits.hits.1._source.leaf: [ 130, 120, 110 ] }
-  - match: { hits.hits.0._source.leaf_default: [10, 20, 30]  }
+  - match: { hits.hits.1._source.leaf_default: [110, 120, 130]  }
   - length: { hits.hits.1._source.obj: 2 }
   - match: { hits.hits.1._source.obj.0.trace.id: aa }
   - match: { hits.hits.1._source.obj.0.span.id: "2" }
   - match: { hits.hits.1._source.obj.1.trace.id: bb }
   - match: { hits.hits.1._source.obj.1.span.id: "2" }
 
+  - match: { hits.hits.2._source.id: 3 }
+  - match: { hits.hits.2._source.obj_default.trace.id: [aa, bb] }
+  - match: { hits.hits.2._source.obj_default.span.id: "2" }
+
 
 ---
 index param - dynamic root arrays: