Merge branch 'hnsw-directio-bfloat16-enabled' into int-hnsw-bfloat16-enabled

Author: thecoop

docs/changelog/135940.yaml

@@ -0,0 +1,5 @@
+pr: 135940
+summary: Enable directIO and bfloat16 for bbq and unquantized vector field types
+area: Vector Search
+type: feature
+issues: []

docs/changelog/136141.yaml

@@ -0,0 +1,6 @@
+pr: 136141
+summary: Add settings for health indicator `shard_capacity` thresholds
+area: Health
+type: enhancement
+issues:
+ - 116697

docs/reference/elasticsearch/configuration-reference/health-diagnostic-settings.md

@@ -47,4 +47,8 @@ The following are the *expert-level* settings available for configuring an inter
 `health.periodic_logger.poll_interval`
 :   ([Dynamic](docs-content://deploy-manage/stack-settings.md#dynamic-cluster-setting), [time unit value](/reference/elasticsearch/rest-apis/api-conventions.md#time-units)) How often {{es}} logs the health status of the cluster and of each health indicator as observed by the Health API. Defaults to `60s` (60 seconds).
 
+`health.shard_capacity.unhealthy_threshold.yellow` {applies_to}`stack: ga 9.3`
+:   ([Dynamic](docs-content://deploy-manage/stack-settings.md#dynamic-cluster-setting)) The minimum number of additional shards the cluster must still be able to allocate (on data or frozen nodes) for shard capacity health to remain `GREEN`. If fewer are available, health becomes `YELLOW`. Must be greater than `health.shard_capacity.unhealthy_threshold.red`. Defaults to `10`.
 
+`health.shard_capacity.unhealthy_threshold.red` {applies_to}`stack: ga 9.3`
+:   ([Dynamic](docs-content://deploy-manage/stack-settings.md#dynamic-cluster-setting)) The minimum number of additional shards the cluster must still be able to allocate (on data or frozen nodes) below which shard capacity health becomes `RED`. Must be less than `health.shard_capacity.unhealthy_threshold.yellow`. Defaults to `5`.

docs/reference/elasticsearch/mapping-reference/semantic-text.md

@@ -424,20 +424,20 @@ POST test-index/_search
 
 ## Updates and partial updates for `semantic_text` fields [semantic-text-updates]
 
-When updating documents that contain `semantic_text` fields, it’s important to understand how inference is triggered:
+When updating documents that contain `semantic_text` fields, it's important to understand how inference is triggered:
 
-* **Full document updates**
-  When you perform a full document update, **all `semantic_text` fields will re-run inference** even if their values did not change. This ensures that the embeddings are always consistent with the current document state but can increase ingestion costs.
+Full document updates
+:   Full document updates re-run inference on all `semantic_text` fields, even if their values did not change. This ensures that embeddings remain consistent with the current document state but can increase ingestion costs.
 
-* **Partial updates using the Bulk API**
-  Partial updates that **omit `semantic_text` fields** and are submitted through the [Bulk API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) will **reuse the existing embeddings** stored in the index. In this case, inference is **not triggered** for fields that were not updated, which can significantly reduce processing time and cost.
+Partial updates using the Bulk API
+:   Partial updates submitted through the [Bulk API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) reuse existing embeddings when you omit `semantic_text` fields. Inference does not run for omitted fields, which can significantly reduce processing time and cost.
 
-* **Partial updates using the Update API**
-  When using the [Update API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-update) with a `doc` object that **omits `semantic_text` fields**, inference **will still run** on all `semantic_text` fields. This means that even if the field values are not changed, embeddings will be re-generated.
+Partial updates using the Update API
+:   Partial updates submitted through the [Update API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-update) re-run inference on all `semantic_text` fields, even when you omit them from the `doc` object. Embeddings are re-generated regardless of whether field values changed.
 
-If you want to avoid unnecessary inference and keep existing embeddings:
+To preserve existing embeddings and avoid unnecessary inference costs:
 
- * Use **partial updates through the Bulk API**.
+ * Use partial updates with the Bulk API.
  * Omit any `semantic_text` fields that did not change from the `doc` object in your request.
 
 ### Scripted updates

docs/reference/elasticsearch/rest-apis/retrievers/retrievers-examples.md

@@ -113,7 +113,9 @@ First, let’s examine how to combine two different types of queries: a `kNN` qu
 While these queries may produce scores in different ranges, we can use Reciprocal Rank Fusion (`rrf`) to combine the results and generate a merged final result list.
 
 To implement this in the retriever framework, we start with the top-level element: our `rrf` retriever.
-This retriever operates on top of two other retrievers: a `knn` retriever and a `standard` retriever. Our query structure would look like this:
+This retriever operates on top of two other retrievers: a `knn` retriever and a `standard` retriever.
+We can specify weights to adjust the influence of each retriever on the final ranking.
+In this example, we're giving the `standard` retriever twice the influence of the `knn` retriever:
 
 ```console
 GET /retrievers_example/_search
@@ -197,6 +199,57 @@ This returns the following response based on the final rrf score for each result
 ::::
 
 
+### Using the expanded format with weights 
+```{applies_to}
+stack: ga 9.2
+```
+
+The same query can be written using the expanded format, which allows you to specify custom weights to adjust the influence of each retriever on the final ranking.
+In this example, we're giving the `standard` retriever twice the influence of the `knn` retriever:
+
+```console
+GET /retrievers_example/_search
+{
+    "retriever": {
+        "rrf": {
+            "retrievers": [
+                {
+                    "retriever": {
+                        "standard": {
+                            "query": {
+                                "query_string": {
+                                    "query": "(information retrieval) OR (artificial intelligence)",
+                                    "default_field": "text"
+                                }
+                            }
+                        }
+                    },
+                    "weight": 2.0
+                },
+                {
+                    "retriever": {
+                        "knn": {
+                            "field": "vector",
+                            "query_vector": [
+                                0.23,
+                                0.67,
+                                0.89
+                            ],
+                            "k": 3,
+                            "num_candidates": 5
+                        }
+                    },
+                    "weight": 1.0
+                }
+            ],
+            "rank_window_size": 10,
+            "rank_constant": 1
+        }
+    },
+    "_source": false
+}
+```
+
 
 ## Example: Hybrid search with linear retriever [retrievers-examples-linear-retriever]
 

docs/reference/elasticsearch/rest-apis/retrievers/rrf-retriever.md

@@ -6,7 +6,7 @@ applies_to:
 
 # RRF retriever [rrf-retriever]
 
-An [RRF](/reference/elasticsearch/rest-apis/reciprocal-rank-fusion.md) retriever returns top documents based on the RRF formula, equally weighting two or more child retrievers.
+An [RRF](/reference/elasticsearch/rest-apis/reciprocal-rank-fusion.md) retriever returns top documents based on the RRF formula, combining two or more child retrievers.
 Reciprocal rank fusion (RRF) is a method for combining multiple result sets with different relevance indicators into a single result set.
 
 
@@ -32,7 +32,13 @@ Combining `query` and `retrievers` is not supported.
 :   (Optional, array of retriever objects)
 
     A list of child retrievers to specify which sets of returned top documents will have the RRF formula applied to them.
-    Each child retriever carries an equal weight as part of the RRF formula. Two or more child retrievers are required.
+    Each retriever can optionally include a weight to adjust its influence on the final ranking. {applies_to}`stack: ga 9.2`
+    
+    When weights are specified, the final RRF score is calculated as:
+    ```
+    rrf_score = weight_1 × rrf_score_1 + weight_2 × rrf_score_2 + ... + weight_n × rrf_score_n
+    ```
+    where `rrf_score_i` is the RRF score for document from retriever `i`, and `weight_i` is the weight for that retriever.
 
 `rank_constant`
 :   (Optional, integer)
@@ -53,6 +59,82 @@ Combining `query` and `retrievers` is not supported.
 
     Applies the specified [boolean query filter](/reference/query-languages/query-dsl/query-dsl-bool-query.md) to all of the specified sub-retrievers, according to each retriever’s specifications.
 
+Each entry in the `retrievers` array can be specified using the direct format or the wrapped format. {applies_to}`stack: ga 9.2`
+
+**Direct format** (default weight of `1.0`):
+```json
+{
+  "rrf": {
+    "retrievers": [
+      {
+        "standard": {
+          "query": {
+            "multi_match": {
+              "query": "search text",
+              "fields": ["field1", "field2"]
+            }
+          }
+        }
+      },
+      {
+        "knn": {
+          "field": "vector",
+          "query_vector": [1, 2, 3],
+          "k": 10,
+          "num_candidates": 50
+        }
+      }
+    ]
+  }
+}
+```
+
+**Wrapped format with custom weights** {applies_to}`stack: ga 9.2`:
+```json
+{
+  "rrf": {
+    "retrievers": [
+      {
+        "retriever": {
+          "standard": {
+            "query": {
+              "multi_match": {
+                "query": "search text",
+                "fields": ["field1", "field2"]
+              }
+            }
+          }
+        },
+        "weight": 2.0
+      },
+      {
+        "retriever": {
+          "knn": {
+            "field": "vector",
+            "query_vector": [1, 2, 3],
+            "k": 10,
+            "num_candidates": 50
+          }
+        },
+        "weight": 1.0
+      }
+    ]
+  }
+}
+```
+
+In the wrapped format:
+
+`retriever`
+:   (Required, a retriever object)
+
+    Specifies a child retriever. Any valid retriever type can be used (e.g., `standard`, `knn`, `text_similarity_reranker`, etc.).
+
+`weight` {applies_to}`stack: ga 9.2`
+:   (Optional, float)
+
+    The weight that each score of this retriever's top docs will be multiplied in the RRF formula. Higher values increase this retriever's influence on the final ranking. Must be non-negative. Defaults to `1.0`.
+
 ## Example: Hybrid search [rrf-retriever-example-hybrid]
 
 A simple hybrid search example (lexical search + dense vector search) combining a `standard` retriever with a `knn` retriever using RRF:
@@ -182,6 +264,75 @@ GET /restaurants/_search
 5. The rank constant for the RRF retriever.
 6. The rank window size for the RRF retriever.
 
+## Example: Weighted hybrid search [rrf-retriever-example-weighted]
+
+{applies_to}`stack: ga 9.2`
+
+This example demonstrates how to use weights to adjust the influence of different retrievers in the RRF ranking.
+In this case, we're giving the `standard` retriever more importance (weight 2.0) compared to the `knn` retriever (weight 1.0):
+
+```console
+GET /restaurants/_search
+{
+  "retriever": {
+    "rrf": {
+      "retrievers": [
+        {
+          "retriever": { <1>
+            "standard": {
+              "query": {
+                "multi_match": {
+                  "query": "Austria",
+                  "fields": ["city", "region"]
+                }
+              }
+            }
+          },
+          "weight": 2.0 <2>
+        },
+        {
+          "retriever": { <3>
+            "knn": {
+              "field": "vector",
+              "query_vector": [10, 22, 77],
+              "k": 10,
+              "num_candidates": 10
+            }
+          },
+          "weight": 1.0 <4>
+        }
+      ],
+      "rank_constant": 60,
+      "rank_window_size": 50
+    }
+  }
+}
+```
+% TEST[continued]
+
+1. The first retriever in weighted format.
+2. This retriever has a weight of 2.0, giving it twice the influence of the kNN retriever.
+3. The second retriever in weighted format.
+4. This retriever has a weight of 1.0 (default weight).
+
+::::{note}
+You can mix weighted and non-weighted formats in the same query.
+The direct format (without explicit `retriever` wrapper) uses the default weight of `1.0`:
+
+```json
+{
+  "rrf": {
+    "retrievers": [
+      { "standard": { "query": {...} } },
+      { "retriever": { "knn": {...} }, "weight": 2.0 }
+    ]
+  }
+}
+```
+
+In this example, the `standard` retriever uses weight `1.0` (default), while the `knn` retriever uses weight `2.0`.
+::::
+
 ## Example: Hybrid search with sparse vectors [rrf-retriever-example-hybrid-sparse]
 
 A more complex hybrid search example (lexical search + ELSER sparse vector search + dense vector search) using RRF:

libs/exponential-histogram/src/main/java/org/elasticsearch/exponentialhistogram/FixedCapacityExponentialHistogram.java

@@ -30,6 +30,10 @@
  * <br>
  * Consumers must ensure that if the histogram is mutated, all previously acquired {@link BucketIterator}
  * instances are no longer used.
+ * <br>
+ * This implementation is thread-safe for all operations provided via {@link ReleasableExponentialHistogram} and its superclasses,
+ * as long as it is not mutated concurrently using any of the methods declared in addition in this class
+ * (e.g. {@link #tryAddBucket(long, long, boolean)}).
  */
 final class FixedCapacityExponentialHistogram extends AbstractExponentialHistogram implements ReleasableExponentialHistogram {
 
@@ -243,8 +247,10 @@ private class Buckets implements ExponentialHistogram.Buckets {
 
         private final boolean isPositive;
         private int numBuckets;
-        private int cachedValueSumForNumBuckets;
-        private long cachedValueSum;
+
+        private record CachedCountsSum(int numBuckets, long countsSum) {}
+
+        private CachedCountsSum cachedCountsSum;
 
         /**
          * @param isPositive true, if this object should represent the positive bucket range, false for the negative range
@@ -263,8 +269,7 @@ int startSlot() {
 
         final void reset() {
             numBuckets = 0;
-            cachedValueSumForNumBuckets = 0;
-            cachedValueSum = 0;
+            cachedCountsSum = null;
         }
 
         boolean tryAddBucket(long index, long count) {
@@ -297,12 +302,26 @@ public OptionalLong maxBucketIndex() {
 
         @Override
         public long valueCount() {
+            // copy a reference to the field to avoid problems with concurrent updates
+            CachedCountsSum cachedVal = cachedCountsSum;
+            if (cachedVal != null && cachedVal.numBuckets == numBuckets) {
+                return cachedVal.countsSum;
+            }
+
+            long countsSum = 0;
+            int position = 0;
+            if (cachedVal != null) {
+                countsSum = cachedVal.countsSum;
+                position = cachedVal.numBuckets;
+            }
+
             int startSlot = startSlot();
-            while (cachedValueSumForNumBuckets < numBuckets) {
-                cachedValueSum += bucketCounts[startSlot + cachedValueSumForNumBuckets];
-                cachedValueSumForNumBuckets++;
+            while (position < numBuckets) {
+                countsSum += bucketCounts[startSlot + position];
+                position++;
             }
-            return cachedValueSum;
+            this.cachedCountsSum = new CachedCountsSum(position, countsSum);
+            return countsSum;
         }
 
         @Override