Merge branch 'main' into markjhoy/default_token_pruning_sparse_vector

Author: markjhoy

docs/changelog/126805.yaml

@@ -0,0 +1,5 @@
+pr: 126805
+summary: Adding timeout to request for creating inference endpoint
+area: Machine Learning
+type: bug
+issues: []

docs/reference/elasticsearch/mapping-reference/keyword.md

@@ -232,6 +232,42 @@ Will become:
 }
 ```
 
+If `null_value` is configured, `null` values are replaced with the `null_value` in synthetic source:
+
+$$$synthetic-source-keyword-example-null-value$$$
+
+```console
+PUT idx
+{
+  "settings": {
+    "index": {
+      "mapping": {
+        "source": {
+          "mode": "synthetic"
+        }
+      }
+    }
+  },
+  "mappings": {
+    "properties": {
+      "kwd": { "type": "keyword", "null_value": "NA" }
+    }
+  }
+}
+PUT idx/_doc/1
+{
+  "kwd": ["foo", null, "bar"]
+}
+```
+
+Will become:
+
+```console-result
+{
+  "kwd": ["bar", "foo", "NA"]
+}
+```
+
 
 ## Constant keyword field type [constant-keyword-field-type]
 

docs/reference/elasticsearch/mapping-reference/text.md

@@ -104,11 +104,17 @@ Synthetic `_source` is Generally Available only for TSDB indices (indices that h
 ::::
 
 
-`text` fields support [synthetic `_source`](/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source) if they have a [`keyword`](/reference/elasticsearch/mapping-reference/keyword.md#keyword-synthetic-source) sub-field that supports synthetic `_source` or if the `text` field sets `store` to `true`. Either way, it may not have [`copy_to`](/reference/elasticsearch/mapping-reference/copy-to.md).
+`text` fields can use a [`keyword`](/reference/elasticsearch/mapping-reference/keyword.md#keyword-synthetic-source) sub-field to support [synthetic `_source`](/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source) without storing values of the text field itself.
 
-If using a sub-`keyword` field, then the values are sorted in the same way as a `keyword` field’s values are sorted. By default, that means sorted with duplicates removed. So:
+In this case, the synthetic source of the `text` field will have the same [modifications](/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source) as a `keyword` field.
 
-$$$synthetic-source-text-example-default$$$
+These modifications can impact usage of `text` fields:
+* Reordering text fields can have an effect on [phrase](/reference/query-languages/query-dsl/query-dsl-match-query-phrase.md) and [span](/reference/query-languages/query-dsl/span-queries.md) queries. See the discussion about [`position_increment_gap`](/reference/elasticsearch/mapping-reference/position-increment-gap.md) for more details. You can avoid this by making sure the `slop` parameter on the phrase queries is lower than the `position_increment_gap`. This is the default.
+* Handling of `null` values is different. `text` fields ignore `null` values, but `keyword` fields support replacing `null` values with a value specified in the `null_value` parameter. This replacement is represented in synthetic source.
+
+For example:
+
+$$$synthetic-source-text-example-multi-field$$$
 
 ```console
 PUT idx
@@ -127,8 +133,9 @@ PUT idx
       "text": {
         "type": "text",
         "fields": {
-          "raw": {
-            "type": "keyword"
+          "kwd": {
+            "type": "keyword",
+            "null_value": "NA"
           }
         }
       }
@@ -138,9 +145,10 @@ PUT idx
 PUT idx/_doc/1
 {
   "text": [
+    null,
     "the quick brown fox",
     "the quick brown fox",
-    "jumped over the lazy dog"
+    "jumped over the lazy dog",
   ]
 }
 ```
@@ -150,18 +158,15 @@ Will become:
 ```console-result
 {
   "text": [
-    "jumped over the lazy dog",
+    "jumped over the lazy dog"
+    "NA",
     "the quick brown fox"
   ]
 }
 ```
 
-::::{note}
-Reordering text fields can have an effect on [phrase](/reference/query-languages/query-dsl/query-dsl-match-query-phrase.md) and [span](/reference/query-languages/query-dsl/span-queries.md) queries. See the discussion about [`position_increment_gap`](/reference/elasticsearch/mapping-reference/position-increment-gap.md) for more detail. You can avoid this by making sure the `slop` parameter on the phrase queries is lower than the `position_increment_gap`. This is the default.
-::::
 
-
-If the `text` field sets `store` to true then order and duplicates are preserved.
+If the `text` field sets `store` to `true` then the sub-field is not used and no modifications are applied. For example:
 
 $$$synthetic-source-text-example-stored$$$
 
@@ -179,7 +184,15 @@ PUT idx
   },
   "mappings": {
     "properties": {
-      "text": { "type": "text", "store": true }
+      "text": {
+        "type": "text",
+        "store": true,
+        "fields": {
+          "raw": {
+            "type": "keyword"
+          }
+        }
+      }
     }
   }
 }

docs/release-notes/breaking-changes.md

@@ -13,9 +13,6 @@ If you are migrating from a version prior to version 9.0, you must first upgrade
 % ## Next version [elasticsearch-nextversion-breaking-changes]
 
 ## 9.0.1 [elasticsearch-9.0.1-breaking-changes]
-```{applies_to}
-stack: coming 9.0.1
-```
 
 No breaking changes in this version.
 

docs/release-notes/deprecations.md

@@ -17,9 +17,6 @@ To give you insight into what deprecated features you’re using, {{es}}:
 % ## Next version [elasticsearch-nextversion-deprecations]
 
 ## 9.0.1 [elasticsearch-9.0.1-deprecations]
-```{applies_to}
-stack: coming 9.0.1
-```
 
 No deprecations in this version.
 

docs/release-notes/index.md

@@ -21,9 +21,6 @@ To check for security updates, go to [Security announcements for the Elastic sta
 % *
 
 ## 9.0.1 [elasticsearch-9.0.1-release-notes]
-```{applies_to}
-stack: coming 9.0.1
-```
 
 ### Features and enhancements [elasticsearch-9.0.1-features-enhancements]
 

muted-tests.yml

@@ -441,6 +441,9 @@ tests:
 - class: org.elasticsearch.xpack.esql.optimizer.OptimizerRulesTests
   method: testOptimizerExpressionRuleShouldNotVisitExcludedNodes
   issue: https://github.com/elastic/elasticsearch/issues/127754
+- class: org.elasticsearch.xpack.remotecluster.RemoteClusterSecurityFcActionAuthorizationIT
+  method: testIndicesPrivilegesAreEnforcedForCcrRestoreSessionActions
+  issue: https://github.com/elastic/elasticsearch/issues/127782
 
 # Examples:
 #

server/src/main/java/org/elasticsearch/TransportVersions.java

@@ -172,7 +172,8 @@ static TransportVersion def(int id) {
     public static final TransportVersion INTRODUCE_FAILURES_LIFECYCLE_BACKPORT_8_19 = def(8_841_0_25);
     public static final TransportVersion INTRODUCE_FAILURES_DEFAULT_RETENTION_BACKPORT_8_19 = def(8_841_0_26);
     public static final TransportVersion RESCORE_VECTOR_ALLOW_ZERO_BACKPORT_8_19 = def(8_841_0_27);
-    public static final TransportVersion SPARSE_VECTOR_FIELD_PRUNING_OPTIONS_8_19 = def(8_841_0_28);
+    public static final TransportVersion INFERENCE_ADD_TIMEOUT_PUT_ENDPOINT_8_19 = def(8_841_0_28);
+    public static final TransportVersion SPARSE_VECTOR_FIELD_PRUNING_OPTIONS_8_19 = def(8_841_0_29);
     public static final TransportVersion V_9_0_0 = def(9_000_0_09);
     public static final TransportVersion INITIAL_ELASTICSEARCH_9_0_1 = def(9_000_0_10);
     public static final TransportVersion INITIAL_ELASTICSEARCH_9_0_2 = def(9_000_0_11);
@@ -249,7 +250,8 @@ static TransportVersion def(int id) {
     public static final TransportVersion INTRODUCE_FAILURES_DEFAULT_RETENTION = def(9_071_0_00);
     public static final TransportVersion FILE_SETTINGS_HEALTH_INFO = def(9_072_0_00);
     public static final TransportVersion FIELD_CAPS_ADD_CLUSTER_ALIAS = def(9_073_0_00);
-    public static final TransportVersion SPARSE_VECTOR_FIELD_PRUNING_OPTIONS = def(9_074_0_00);
+    public static final TransportVersion INFERENCE_ADD_TIMEOUT_PUT_ENDPOINT = def(9_074_00_0);
+    public static final TransportVersion SPARSE_VECTOR_FIELD_PRUNING_OPTIONS = def(9_075_0_00);
 
     /*
      * STOP! READ THIS FIRST! No, really,

server/src/main/java/org/elasticsearch/action/search/SearchQueryThenFetchAsyncAction.java

@@ -23,7 +23,6 @@
 import org.elasticsearch.action.support.IndicesOptions;
 import org.elasticsearch.client.internal.Client;
 import org.elasticsearch.cluster.ClusterState;
-import org.elasticsearch.common.bytes.ReleasableBytesReference;
 import org.elasticsearch.common.io.stream.NamedWriteableRegistry;
 import org.elasticsearch.common.io.stream.RecyclerBytesStreamOutput;
 import org.elasticsearch.common.io.stream.StreamInput;
@@ -820,7 +819,7 @@ void onShardDone() {
                     out.close();
                 }
             }
-            ActionListener.respondAndRelease(channelListener, new BytesTransportResponse(new ReleasableBytesReference(out.bytes(), out)));
+            ActionListener.respondAndRelease(channelListener, new BytesTransportResponse(out.moveToBytesReference()));
         }
 
         private void maybeFreeContext(SearchPhaseResult result, BitSet relevantShardIndices) {

server/src/main/java/org/elasticsearch/cluster/coordination/JoinValidationService.java

@@ -391,9 +391,7 @@ private ReleasableBytesReference maybeSerializeClusterState(
         }
         assert clusterState.nodes().isLocalNodeElectedMaster();
 
-        final var bytesStream = transportService.newNetworkBytesStream();
-        var success = false;
-        try {
+        try (var bytesStream = transportService.newNetworkBytesStream()) {
             try (
                 var stream = new OutputStreamStreamOutput(
                     CompressorFactory.COMPRESSOR.threadLocalOutputStream(Streams.flushOnCloseStream(bytesStream))
@@ -404,22 +402,16 @@ private ReleasableBytesReference maybeSerializeClusterState(
             } catch (IOException e) {
                 throw new ElasticsearchException("failed to serialize cluster state for publishing to node {}", e, discoveryNode);
             }
-            final var newBytes = new ReleasableBytesReference(bytesStream.bytes(), bytesStream);
             logger.trace(
                 "serialized join validation cluster state version [{}] for transport version [{}] with size [{}]",
                 clusterState.version(),
                 version,
-                newBytes.length()
+                bytesStream.position()
             );
+            var newBytes = bytesStream.moveToBytesReference();
             final var previousBytes = statesByVersion.put(version, newBytes);
             assert previousBytes == null;
-            success = true;
             return newBytes;
-        } finally {
-            if (success == false) {
-                bytesStream.close();
-                assert false;
-            }
         }
     }
 }