Merge branch 'main' into push-down-mv_expand-past-project

Author: kanoshiou

docs/changelog/136167.yaml

@@ -0,0 +1,6 @@
+pr: 136167
+summary: "[Inference API] Remove worst-case additional 50ms latency for non-rate limited\
+  \ requests"
+area: Machine Learning
+type: bug
+issues: []

docs/reference/elasticsearch-plugins/repository-hdfs-usage.md

@@ -5,7 +5,7 @@ mapped_pages:
 
 # Getting started with HDFS [repository-hdfs-usage]
 
-The HDFS snapshot/restore plugin is built against the latest Apache Hadoop 2.x (currently 2.7.1). If the distro you are using is not protocol compatible with Apache Hadoop, consider replacing the Hadoop libraries inside the plugin folder with your own (you might have to adjust the security permissions required).
+The `repository-hdfs` snapshot/restore plugin uses the Apache Hadoop client libraries, version 3.4.1. Your HDFS implementation must be protocol-compatible with Apache Hadoop to use it with this plugin.
 
 Even if Hadoop is already installed on the Elasticsearch nodes, for security reasons, the required libraries need to be placed under the plugin folder. Note that in most cases, if the distro is compatible, one simply needs to configure the repository with the appropriate Hadoop configuration files (see below).
 

muted-tests.yml

@@ -483,6 +483,10 @@ tests:
 - class: org.elasticsearch.xpack.inference.integration.CCMServiceIT
   method: testIsEnabled_ReturnsTrue_WhenCCMConfigurationIsPresent
   issue: https://github.com/elastic/elasticsearch/issues/137798
+- class: org.elasticsearch.xpack.esql.vector.VectorSimilarityFunctionsIT
+  method: testSimilarityWithOneDimVector {functionName=v_cosine
+    similarityFunction=org.elasticsearch.xpack.esql.expression.function.vector.CosineSimilarity$1@3300f4fd elementType=byte}
+  issue: https://github.com/elastic/elasticsearch/issues/137812
 
 # Examples:
 #

server/src/main/java/org/elasticsearch/action/support/SubscribableListener.java

@@ -99,6 +99,23 @@
  *         .addListener(finalListener);
  * }
  * }</pre>
+ * <p>
+ * You must take care when using a chain of {@link SubscribableListener}s where one or more of the response objects have nontrivial
+ * lifecycles (e.g. they implement {@link org.elasticsearch.core.Releasable} or {@link org.elasticsearch.core.RefCounted}). For example:
+ * <ul>
+ *     <li>{@link SubscribableListener} silently discards all but one response, whether exceptional or otherwise. The caller must take steps
+ *     to release any discarded responses that need releasing.</li>
+ *     <li>When a {@link SubscribableListener} is completed it keeps hold of the response in case another listener subscribes to this
+ *     response in future (e.g. via {@link #addListener} or {@link #andThen}) but it does not formally take ownership of the response (e.g.
+ *     by calling {@link org.elasticsearch.core.RefCounted#incRef}). In particular, in the usual pattern ...
+ *     <pre>{@code
+ *         SubscribableListener.newForked(l1 -> step1(l1)).andThen((l2, r1) -> step2(r1, l2))...
+ *     }</pre>
+ *     ... if {@code r1} is ref-counted then usually {@code step1} will call {@link org.elasticsearch.core.RefCounted#decRef} immediately
+ *     after {@code l1.onResponse(r1)} returns since it no longer needs to keep the response alive itself. This is a problem because it may
+ *     happen before the {@link #andThen} adds the second step to the chain, so that by the time {@code step2} runs the response {@code r1}
+ *     may already be fully-released. The caller must take steps to keep responses alive until they are no longer needed.</li>
+ * </ul>
  */
 public class SubscribableListener<T> implements ActionListener<T> {
 

server/src/main/java/org/elasticsearch/inference/InputType.java

@@ -61,6 +61,10 @@ public static boolean isInternalTypeOrUnspecified(InputType inputType) {
         return inputType == InputType.INTERNAL_INGEST || inputType == InputType.INTERNAL_SEARCH || inputType == InputType.UNSPECIFIED;
     }
 
+    public static boolean isIngest(InputType inputType) {
+        return inputType == InputType.INGEST || inputType == InputType.INTERNAL_INGEST;
+    }
+
     public static boolean isSpecified(InputType inputType) {
         return inputType != null && inputType != InputType.UNSPECIFIED;
     }