Merge branch 'main' into esql_move_limit_in_docs

Author: nik9000

.gitattributes

@@ -13,6 +13,9 @@ x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/EsqlBasePar
 x-pack/plugin/esql/src/main/generated/** linguist-generated=true
 x-pack/plugin/esql/src/main/generated-src/** linguist-generated=true
 
-# ESQL functions docs are autogenerated. More information at `docs/reference/esql/functions/README.md`
-docs/reference/esql/functions/*/** linguist-generated=true
+# ESQL functions docs are autogenerated. More information at `docs/reference/query-languages/esql/README.md`
+docs/reference/query-languages/esql/_snippets/functions/*/** linguist-generated=true
+#docs/reference/query-languages/esql/_snippets/operators/*/** linguist-generated=true
+docs/reference/query-languages/esql/images/** linguist-generated=true
+docs/reference/query-languages/esql/kibana/** linguist-generated=true
 

docs/changelog/124581.yaml

@@ -0,0 +1,5 @@
+pr: 124581
+summary: New `vector_rescore` parameter as a quantized index type option
+area: Vector Search
+type: enhancement
+issues: []

docs/changelog/124676.yaml

@@ -0,0 +1,6 @@
+pr: 124676
+summary: TO_LOWER processes all values
+area: ES|QL
+type: bug
+issues:
+ - 124002

docs/changelog/124739.yaml

@@ -0,0 +1,5 @@
+pr: 124739
+summary: Improve rolling up metrics
+area: Downsampling
+type: enhancement
+issues: []

docs/docset.yml

@@ -2,8 +2,8 @@ project: 'Elasticsearch'
 exclude:
   - README.md
   - internal/*
-  - reference/esql/functions/kibana/docs/*
-  - reference/esql/functions/README.md
+  - reference/query-languages/esql/kibana/docs/**
+  - reference/query-languages/esql/README.md
 cross_links:
   - beats
   - cloud

docs/reference/data-analysis/aggregations/_snippets/search-aggregations-metrics-cardinality-aggregation-explanation.md

@@ -0,0 +1,17 @@
+* configurable precision, which decides on how to trade memory for accuracy,
+* excellent accuracy on low-cardinality sets,
+* fixed memory usage: no matter if there are tens or billions of unique values, memory usage only depends on the configured precision.
+
+For a precision threshold of `c`, the implementation that we are using requires about `c * 8` bytes.
+
+The following chart shows how the error varies before and after the threshold:
+
+![cardinality error](/images/cardinality_error.png "")
+
+For all 3 thresholds, counts have been accurate up to the configured threshold. Although not guaranteed,
+this is likely to be the case. Accuracy in practice depends on the dataset in question. In general,
+most datasets show consistently good accuracy. Also note that even with a threshold as low as 100,
+the error remains very low (1-6% as seen in the above graph) even when counting millions of items.
+
+The HyperLogLog++ algorithm depends on the leading zeros of hashed values, the exact distributions of
+hashes in a dataset can affect the accuracy of the cardinality.

docs/reference/query-languages/esql/_snippets/functions/percentile.md renamed to docs/reference/data-analysis/aggregations/_snippets/search-aggregations-metrics-percentile-aggregation-approximate.md

@@ -1,60 +1,3 @@
-## `PERCENTILE` [esql-percentile]
-
-**Syntax**
-
-:::{image} ../../../../../images/percentile.svg
-:alt: Embedded
-:class: text-center
-:::
-
-**Parameters**
-
-true
-**Description**
-
-Returns the value at which a certain percentage of observed values occur. For example, the 95th percentile is the value which is greater than 95% of the observed values and the 50th percentile is the `MEDIAN`.
-
-**Supported types**
-
-| number | percentile | result |
-| --- | --- | --- |
-| double | double | double |
-| double | integer | double |
-| double | long | double |
-| integer | double | double |
-| integer | integer | double |
-| integer | long | double |
-| long | double | double |
-| long | integer | double |
-| long | long | double |
-
-**Examples**
-
-```esql
-FROM employees
-| STATS p0 = PERCENTILE(salary,  0)
-     , p50 = PERCENTILE(salary, 50)
-     , p99 = PERCENTILE(salary, 99)
-```
-
-| p0:double | p50:double | p99:double |
-| --- | --- | --- |
-| 25324 | 47003 | 74970.29 |
-
-The expression can use inline functions. For example, to calculate a percentile of the maximum values of a multivalued column, first use `MV_MAX` to get the maximum value per row, and use the result with the `PERCENTILE` function
-
-```esql
-FROM employees
-| STATS p80_max_salary_change = PERCENTILE(MV_MAX(salary_change), 80)
-```
-
-| p80_max_salary_change:double |
-| --- |
-| 12.132 |
-
-
-### `PERCENTILE` is (usually) approximate [esql-percentile-approximate]
-
 There are many different algorithms to calculate percentiles. The naive implementation simply stores all the values in a sorted array. To find the 50th percentile, you simply find the value that is at `my_array[count(my_array) * 0.5]`.
 
 Clearly, the naive implementation does not scale — the sorted array grows linearly with the number of values in your dataset. To calculate percentiles across potentially billions of values in an Elasticsearch cluster, *approximate* percentiles are calculated.
@@ -72,11 +15,3 @@ The following chart shows the relative error on a uniform distribution depending
 ![percentiles error](/images/percentiles_error.png "")
 
 It shows how precision is better for extreme percentiles. The reason why error diminishes for large number of values is that the law of large numbers makes the distribution of values more and more uniform and the t-digest tree can do a better job at summarizing it. It would not be the case on more skewed distributions.
-
-::::{warning}
-`PERCENTILE` is also [non-deterministic](https://en.wikipedia.org/wiki/Nondeterministic_algorithm). This means you can get slightly different results using the same data.
-
-::::
-
-
-

docs/reference/data-analysis/aggregations/search-aggregations-metrics-cardinality-aggregation.md

@@ -65,19 +65,8 @@ Computing exact counts requires loading values into a hash set and returning its
 
 This `cardinality` aggregation is based on the [HyperLogLog++](https://static.googleusercontent.com/media/research.google.com/fr//pubs/archive/40671.pdf) algorithm, which counts based on the hashes of the values with some interesting properties:
 
-* configurable precision, which decides on how to trade memory for accuracy,
-* excellent accuracy on low-cardinality sets,
-* fixed memory usage: no matter if there are tens or billions of unique values, memory usage only depends on the configured precision.
-
-For a precision threshold of `c`, the implementation that we are using requires about `c * 8` bytes.
-
-The following chart shows how the error varies before and after the threshold:
-
-![cardinality error](../../../images/cardinality_error.png "")
-
-For all 3 thresholds, counts have been accurate up to the configured threshold. Although not guaranteed, this is likely to be the case. Accuracy in practice depends on the dataset in question. In general, most datasets show consistently good accuracy. Also note that even with a threshold as low as 100, the error remains very low (1-6% as seen in the above graph) even when counting millions of items.
-
-The HyperLogLog++ algorithm depends on the leading zeros of hashed values, the exact distributions of hashes in a dataset can affect the accuracy of the cardinality.
+:::{include} _snippets/search-aggregations-metrics-cardinality-aggregation-explanation.md
+:::
 
 
 ## Pre-computed hashes [_pre_computed_hashes]

docs/reference/data-analysis/aggregations/search-aggregations-metrics-percentile-aggregation.md

@@ -175,31 +175,14 @@ GET latency/_search
 
 ## Percentiles are (usually) approximate [search-aggregations-metrics-percentile-aggregation-approximation]
 
-There are many different algorithms to calculate percentiles. The naive implementation simply stores all the values in a sorted array. To find the 50th percentile, you simply find the value that is at `my_array[count(my_array) * 0.5]`.
-
-Clearly, the naive implementation does not scale — the sorted array grows linearly with the number of values in your dataset. To calculate percentiles across potentially billions of values in an Elasticsearch cluster, *approximate* percentiles are calculated.
-
-The algorithm used by the `percentile` metric is called TDigest (introduced by Ted Dunning in [Computing Accurate Quantiles using T-Digests](https://github.com/tdunning/t-digest/blob/master/docs/t-digest-paper/histo.pdf)).
-
-When using this metric, there are a few guidelines to keep in mind:
-
-* Accuracy is proportional to `q(1-q)`. This means that extreme percentiles (e.g. 99%) are more accurate than less extreme percentiles, such as the median
-* For small sets of values, percentiles are highly accurate (and potentially 100% accurate if the data is small enough).
-* As the quantity of values in a bucket grows, the algorithm begins to approximate the percentiles. It is effectively trading accuracy for memory savings. The exact level of inaccuracy is difficult to generalize, since it depends on your data distribution and volume of data being aggregated
-
-The following chart shows the relative error on a uniform distribution depending on the number of collected values and the requested percentile:
-
-![percentiles error](../../../images/percentiles_error.png "")
-
-It shows how precision is better for extreme percentiles. The reason why error diminishes for large number of values is that the law of large numbers makes the distribution of values more and more uniform and the t-digest tree can do a better job at summarizing it. It would not be the case on more skewed distributions.
+:::{include} /reference/data-analysis/aggregations/_snippets/search-aggregations-metrics-percentile-aggregation-approximate.md
+:::
 
 ::::{warning}
 Percentile aggregations are also [non-deterministic](https://en.wikipedia.org/wiki/Nondeterministic_algorithm). This means you can get slightly different results using the same data.
-
 ::::
 
 
-
 ## Compression [search-aggregations-metrics-percentile-aggregation-compression]
 
 Approximate algorithms must balance memory utilization with estimation accuracy. This balance can be controlled using a `compression` parameter:

docs/reference/elasticsearch/mapping-reference/dense-vector.md

@@ -287,6 +287,14 @@ $$$dense-vector-index-options$$$
     `confidence_interval`
     :   (Optional, float) Only applicable to `int8_hnsw`, `int4_hnsw`, `int8_flat`, and `int4_flat` index types. The confidence interval to use when quantizing the vectors. Can be any value between and including `0.90` and `1.0` or exactly `0`. When the value is `0`, this indicates that dynamic quantiles should be calculated for optimized quantization. When between `0.90` and `1.0`, this value restricts the values used when calculating the quantization thresholds. For example, a value of `0.95` will only use the middle 95% of the values when calculating the quantization thresholds (e.g. the highest and lowest 2.5% of values will be ignored). Defaults to `1/(dims + 1)` for `int8` quantized vectors and `0` for `int4` for dynamic quantile calculation.
 
+    `rescore_vector`
+    :   (Optional, object) Functionality in [preview]. An optional section that configures automatic vector rescoring on knn queries for the given field. Only applicable to quantized index types.
+    :::::{dropdown} Properties of `rescore_vector`
+    `oversample`
+    :   (required, float) The amount to oversample the search results by. This value should be greater than `1.0` and less than `10.0`. The higher the value, the more vectors will be gathered and rescored with the raw values per shard.
+        :   In case a knn query specifies a `rescore_vector` parameter, the query `rescore_vector` parameter will be used instead.
+        :   See [oversampling and rescoring quantized vectors](docs-content://solutions/search/vector/knn.md#dense-vector-knn-search-rescoring) for details.
+        :::::
     ::::