Merge remote-tracking branch 'origin/main' into tracking-postings-memory-bytes-remove-ff

Author: jordan-powers

benchmarks/src/main/java/org/elasticsearch/benchmark/exponentialhistogram/ExponentialHistogramMergeBench.java

@@ -130,7 +130,7 @@ private ExponentialHistogram asCompressedHistogram(ExponentialHistogram histogra
             CompressedExponentialHistogram.writeHistogramBytes(histoBytes, histogram.scale(), negativeBuckets, positiveBuckets);
             CompressedExponentialHistogram result = new CompressedExponentialHistogram();
             BytesRef data = histoBytes.bytes().toBytesRef();
-            result.reset(histogram.zeroBucket().zeroThreshold(), totalCount, data);
+            result.reset(histogram.zeroBucket().zeroThreshold(), totalCount, histogram.sum(), data);
             return result;
         } catch (IOException e) {
             throw new RuntimeException(e);

distribution/docker/src/docker/dockerfiles/cloud_ess_fips/Dockerfile

@@ -24,7 +24,7 @@
 # Extract Elasticsearch artifact
 ################################################################################
 
-FROM docker.elastic.co/wolfi/chainguard-base-fips:latest@sha256:173cd032e2a8b194c04e08f605b40c743cc52f8eb4ecfc9c5e3280ae97c119b4 AS builder
+FROM docker.elastic.co/wolfi/chainguard-base-fips:latest@sha256:d9382de1615bfd6fe66348a9c80258c63fa311b1f583a607addde8ec5bf0f343 AS builder
 
 # Install required packages to extract the Elasticsearch distribution
 RUN <%= retry.loop(package_manager, "export DEBIAN_FRONTEND=noninteractive && ${package_manager} update && ${package_manager} update && ${package_manager} add --no-cache curl") %>
@@ -103,7 +103,7 @@ WORKDIR /usr/share/elasticsearch/config
 # Add entrypoint
 ################################################################################
 
-FROM docker.elastic.co/wolfi/chainguard-base-fips:latest@sha256:173cd032e2a8b194c04e08f605b40c743cc52f8eb4ecfc9c5e3280ae97c119b4
+FROM docker.elastic.co/wolfi/chainguard-base-fips:latest@sha256:d9382de1615bfd6fe66348a9c80258c63fa311b1f583a607addde8ec5bf0f343
 
 RUN <%= retry.loop(package_manager,
           "export DEBIAN_FRONTEND=noninteractive && \n" +

distribution/docker/src/docker/dockerfiles/wolfi/Dockerfile

@@ -24,7 +24,7 @@
 # Extract Elasticsearch artifact
 ################################################################################
 
-FROM docker.elastic.co/wolfi/chainguard-base:latest@sha256:45d6016686787642209440ee29f410a125d2392e6752701cb9960f0ac0c80c1d AS builder
+FROM docker.elastic.co/wolfi/chainguard-base:latest@sha256:4dbe94045c967ec89afcee3727cfb746820bd3f699bbbb621e529bde1ce122b1 AS builder
 
 # Install required packages to extract the Elasticsearch distribution
 RUN <%= retry.loop(package_manager, "export DEBIAN_FRONTEND=noninteractive && ${package_manager} update && ${package_manager} update && ${package_manager} add --no-cache curl") %>
@@ -79,7 +79,7 @@ RUN sed -i -e 's/ES_DISTRIBUTION_TYPE=tar/ES_DISTRIBUTION_TYPE=docker/' bin/elas
 # Add entrypoint
 ################################################################################
 
-FROM docker.elastic.co/wolfi/chainguard-base:latest@sha256:45d6016686787642209440ee29f410a125d2392e6752701cb9960f0ac0c80c1d
+FROM docker.elastic.co/wolfi/chainguard-base:latest@sha256:4dbe94045c967ec89afcee3727cfb746820bd3f699bbbb621e529bde1ce122b1
 
 RUN <%= retry.loop(package_manager,
           "export DEBIAN_FRONTEND=noninteractive && \n" +

docs/changelog/130824.yaml

@@ -0,0 +1,5 @@
+pr: 130824
+summary: Add read permissions for osquery manager result indices
+area: Security
+type: enhancement
+issues: []

docs/changelog/132536.yaml

@@ -0,0 +1,5 @@
+pr: 132536
+summary: Avoid stack overflow in `IndicesClusterStateService` `applyClusterState`
+area: Cluster Coordination
+type: bug
+issues: []

docs/changelog/133397.yaml

@@ -0,0 +1,5 @@
+pr: 133397
+summary: Push down loading of singleton dense double based field types to the …
+area: "Codec"
+type: enhancement
+issues: []

docs/changelog/133601.yaml

@@ -0,0 +1,8 @@
+pr: 133601
+summary: Fix bug in topn
+area: ES|QL
+type: bug
+issues:
+ - 133600
+ - 133574
+ - 133607

docs/reference/elasticsearch/mapping-reference/dense-vector.md

@@ -115,23 +115,23 @@ To retrieve vector values explicitly, you can use:
 
 * The `fields` option to request specific vector fields directly:
 
-```console
-POST my-index-2/_search
-{
-  "fields": ["my_vector"]
-}
-```
+  ```console
+  POST my-index-2/_search
+  {
+    "fields": ["my_vector"]
+  }
+  ```
 
 - The `_source.exclude_vectors` flag to re-enable vector inclusion in `_source` responses:
 
-```console
-POST my-index-2/_search
-{
-  "_source": {
-    "exclude_vectors": false
+  ```console
+  POST my-index-2/_search
+  {
+    "_source": {
+      "exclude_vectors": false
+    }
   }
-}
-```
+  ```
 
 ### Storage behavior and `_source`
 
@@ -309,7 +309,7 @@ $$$dense-vector-similarity$$$
 `l2_norm`
 :   Computes similarity based on the L2 distance (also known as Euclidean distance) between the vectors. The document `_score` is computed as `1 / (1 + l2_norm(query, vector)^2)`.
 
-For `bit` vectors, instead of using `l2_norm`, the `hamming` distance between the vectors is used. The `_score` transformation is `(numBits - hamming(a, b)) / numBits`
+    For `bit` vectors, instead of using `l2_norm`, the `hamming` distance between the vectors is used. The `_score` transformation is `(numBits - hamming(a, b)) / numBits`
 
 `dot_product`
 :   Computes the dot product of two unit vectors. This option provides an optimized way to perform cosine similarity. The constraints and computed score are defined by `element_type`.
@@ -341,15 +341,13 @@ $$$dense-vector-index-options$$$
 `type`
 :   (Required, string) The type of kNN algorithm to use. Can be either any of:
     * `hnsw` - This utilizes the [HNSW algorithm](https://arxiv.org/abs/1603.09320) for scalable approximate kNN search. This supports all `element_type` values.
-    * `int8_hnsw` - The default index type for some float vectors:
-
+    * `int8_hnsw` - The default index type for some float vectors:      
       * {applies_to}`stack: ga 9.1` Default for float vectors with less than 384 dimensions.
       * {applies_to}`stack: ga 9.0` Default for float all vectors.
-
       This utilizes the [HNSW algorithm](https://arxiv.org/abs/1603.09320) in addition to automatically scalar quantization for scalable approximate kNN search with `element_type` of `float`. This can reduce the memory footprint by 4x at the cost of some accuracy. See [Automatically quantize vectors for kNN search](#dense-vector-quantization).
     * `int4_hnsw` - This utilizes the [HNSW algorithm](https://arxiv.org/abs/1603.09320) in addition to automatically scalar quantization for scalable approximate kNN search with `element_type` of `float`. This can reduce the memory footprint by 8x at the cost of some accuracy. See [Automatically quantize vectors for kNN search](#dense-vector-quantization).
     * `bbq_hnsw` - This utilizes the [HNSW algorithm](https://arxiv.org/abs/1603.09320) in addition to automatically binary quantization for scalable approximate kNN search with `element_type` of `float`. This can reduce the memory footprint by 32x at the cost of accuracy. See [Automatically quantize vectors for kNN search](#dense-vector-quantization).
-
+      
       {applies_to}`stack: ga 9.1` `bbq_hnsw` is the default index type for float vectors with greater than or equal to 384 dimensions.
     * `flat` - This utilizes a brute-force search algorithm for exact kNN search. This supports all `element_type` values.
     * `int8_flat` - This utilizes a brute-force search algorithm in addition to automatically scalar quantization. Only supports `element_type` of `float`.
@@ -365,7 +363,6 @@ $$$dense-vector-index-options$$$
 `confidence_interval`
 :   (Optional, float) Only applicable to `int8_hnsw`, `int4_hnsw`, `int8_flat`, and `int4_flat` index types. The confidence interval to use when quantizing the vectors. Can be any value between and including `0.90` and `1.0` or exactly `0`. When the value is `0`, this indicates that dynamic quantiles should be calculated for optimized quantization. When between `0.90` and `1.0`, this value restricts the values used when calculating the quantization thresholds. For example, a value of `0.95` will only use the middle 95% of the values when calculating the quantization thresholds (e.g. the highest and lowest 2.5% of values will be ignored). Defaults to `1/(dims + 1)` for `int8` quantized vectors and `0` for `int4` for dynamic quantile calculation.
 
-
 `rescore_vector` {applies_to}`stack: preview 9.0, ga 9.1`
 :   (Optional, object) An optional section that configures automatic vector rescoring on knn queries for the given field. Only applicable to quantized index types.
 :::::{dropdown} Properties of rescore_vector
@@ -386,7 +383,7 @@ $$$dense-vector-index-options$$$
 `dense_vector` fields support [synthetic `_source`](/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source) .
 
 
-## Indexing & Searching bit vectors [dense-vector-index-bit]
+## Indexing and searching bit vectors [dense-vector-index-bit]
 
 When using `element_type: bit`, this will treat all vectors as bit vectors. Bit vectors utilize only a single bit per dimension and are internally encoded as bytes. This can be useful for very high-dimensional vectors or models.