Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
7 changes: 5 additions & 2 deletions .gitattributes
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,9 @@ x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/EsqlBasePar
x-pack/plugin/esql/src/main/generated/** linguist-generated=true
x-pack/plugin/esql/src/main/generated-src/** linguist-generated=true

# ESQL functions docs are autogenerated. More information at `docs/reference/esql/functions/README.md`
docs/reference/esql/functions/*/** linguist-generated=true
# ESQL functions docs are autogenerated. More information at `docs/reference/query-languages/esql/README.md`
docs/reference/query-languages/esql/_snippets/functions/*/** linguist-generated=true
#docs/reference/query-languages/esql/_snippets/operators/*/** linguist-generated=true
docs/reference/query-languages/esql/images/** linguist-generated=true
docs/reference/query-languages/esql/kibana/** linguist-generated=true

4 changes: 2 additions & 2 deletions docs/docset.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@ project: 'Elasticsearch'
exclude:
- README.md
- internal/*
- reference/esql/functions/kibana/docs/*
- reference/esql/functions/README.md
- reference/query-languages/esql/kibana/docs/**
- reference/query-languages/esql/README.md
cross_links:
- beats
- cloud
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
* configurable precision, which decides on how to trade memory for accuracy,
* excellent accuracy on low-cardinality sets,
* fixed memory usage: no matter if there are tens or billions of unique values, memory usage only depends on the configured precision.

For a precision threshold of `c`, the implementation that we are using requires about `c * 8` bytes.

The following chart shows how the error varies before and after the threshold:

![cardinality error](/images/cardinality_error.png "")

For all 3 thresholds, counts have been accurate up to the configured threshold. Although not guaranteed,
this is likely to be the case. Accuracy in practice depends on the dataset in question. In general,
most datasets show consistently good accuracy. Also note that even with a threshold as low as 100,
the error remains very low (1-6% as seen in the above graph) even when counting millions of items.

The HyperLogLog++ algorithm depends on the leading zeros of hashed values, the exact distributions of
hashes in a dataset can affect the accuracy of the cardinality.
Original file line number Diff line number Diff line change
@@ -1,60 +1,3 @@
## `PERCENTILE` [esql-percentile]

**Syntax**

:::{image} ../../../../../images/percentile.svg
:alt: Embedded
:class: text-center
:::

**Parameters**

true
**Description**

Returns the value at which a certain percentage of observed values occur. For example, the 95th percentile is the value which is greater than 95% of the observed values and the 50th percentile is the `MEDIAN`.

**Supported types**

| number | percentile | result |
| --- | --- | --- |
| double | double | double |
| double | integer | double |
| double | long | double |
| integer | double | double |
| integer | integer | double |
| integer | long | double |
| long | double | double |
| long | integer | double |
| long | long | double |

**Examples**

```esql
FROM employees
| STATS p0 = PERCENTILE(salary, 0)
, p50 = PERCENTILE(salary, 50)
, p99 = PERCENTILE(salary, 99)
```

| p0:double | p50:double | p99:double |
| --- | --- | --- |
| 25324 | 47003 | 74970.29 |

The expression can use inline functions. For example, to calculate a percentile of the maximum values of a multivalued column, first use `MV_MAX` to get the maximum value per row, and use the result with the `PERCENTILE` function

```esql
FROM employees
| STATS p80_max_salary_change = PERCENTILE(MV_MAX(salary_change), 80)
```

| p80_max_salary_change:double |
| --- |
| 12.132 |


### `PERCENTILE` is (usually) approximate [esql-percentile-approximate]

There are many different algorithms to calculate percentiles. The naive implementation simply stores all the values in a sorted array. To find the 50th percentile, you simply find the value that is at `my_array[count(my_array) * 0.5]`.

Clearly, the naive implementation does not scale — the sorted array grows linearly with the number of values in your dataset. To calculate percentiles across potentially billions of values in an Elasticsearch cluster, *approximate* percentiles are calculated.
Expand All @@ -72,11 +15,3 @@ The following chart shows the relative error on a uniform distribution depending
![percentiles error](/images/percentiles_error.png "")

It shows how precision is better for extreme percentiles. The reason why error diminishes for large number of values is that the law of large numbers makes the distribution of values more and more uniform and the t-digest tree can do a better job at summarizing it. It would not be the case on more skewed distributions.

::::{warning}
`PERCENTILE` is also [non-deterministic](https://en.wikipedia.org/wiki/Nondeterministic_algorithm). This means you can get slightly different results using the same data.

::::



Original file line number Diff line number Diff line change
Expand Up @@ -65,19 +65,8 @@ Computing exact counts requires loading values into a hash set and returning its

This `cardinality` aggregation is based on the [HyperLogLog++](https://static.googleusercontent.com/media/research.google.com/fr//pubs/archive/40671.pdf) algorithm, which counts based on the hashes of the values with some interesting properties:

* configurable precision, which decides on how to trade memory for accuracy,
* excellent accuracy on low-cardinality sets,
* fixed memory usage: no matter if there are tens or billions of unique values, memory usage only depends on the configured precision.

For a precision threshold of `c`, the implementation that we are using requires about `c * 8` bytes.

The following chart shows how the error varies before and after the threshold:

![cardinality error](../../../images/cardinality_error.png "")

For all 3 thresholds, counts have been accurate up to the configured threshold. Although not guaranteed, this is likely to be the case. Accuracy in practice depends on the dataset in question. In general, most datasets show consistently good accuracy. Also note that even with a threshold as low as 100, the error remains very low (1-6% as seen in the above graph) even when counting millions of items.

The HyperLogLog++ algorithm depends on the leading zeros of hashed values, the exact distributions of hashes in a dataset can affect the accuracy of the cardinality.
:::{include} _snippets/search-aggregations-metrics-cardinality-aggregation-explanation.md
:::


## Pre-computed hashes [_pre_computed_hashes]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -175,31 +175,14 @@ GET latency/_search

## Percentiles are (usually) approximate [search-aggregations-metrics-percentile-aggregation-approximation]

There are many different algorithms to calculate percentiles. The naive implementation simply stores all the values in a sorted array. To find the 50th percentile, you simply find the value that is at `my_array[count(my_array) * 0.5]`.

Clearly, the naive implementation does not scale — the sorted array grows linearly with the number of values in your dataset. To calculate percentiles across potentially billions of values in an Elasticsearch cluster, *approximate* percentiles are calculated.

The algorithm used by the `percentile` metric is called TDigest (introduced by Ted Dunning in [Computing Accurate Quantiles using T-Digests](https://github.com/tdunning/t-digest/blob/master/docs/t-digest-paper/histo.pdf)).

When using this metric, there are a few guidelines to keep in mind:

* Accuracy is proportional to `q(1-q)`. This means that extreme percentiles (e.g. 99%) are more accurate than less extreme percentiles, such as the median
* For small sets of values, percentiles are highly accurate (and potentially 100% accurate if the data is small enough).
* As the quantity of values in a bucket grows, the algorithm begins to approximate the percentiles. It is effectively trading accuracy for memory savings. The exact level of inaccuracy is difficult to generalize, since it depends on your data distribution and volume of data being aggregated

The following chart shows the relative error on a uniform distribution depending on the number of collected values and the requested percentile:

![percentiles error](../../../images/percentiles_error.png "")

It shows how precision is better for extreme percentiles. The reason why error diminishes for large number of values is that the law of large numbers makes the distribution of values more and more uniform and the t-digest tree can do a better job at summarizing it. It would not be the case on more skewed distributions.
:::{include} /reference/data-analysis/aggregations/_snippets/search-aggregations-metrics-percentile-aggregation-approximate.md
:::

::::{warning}
Percentile aggregations are also [non-deterministic](https://en.wikipedia.org/wiki/Nondeterministic_algorithm). This means you can get slightly different results using the same data.

::::



## Compression [search-aggregations-metrics-percentile-aggregation-compression]

Approximate algorithms must balance memory utilization with estimation accuracy. This balance can be controlled using a `compression` parameter:
Expand Down
23 changes: 0 additions & 23 deletions docs/reference/esql/functions/README.md

This file was deleted.

17 changes: 0 additions & 17 deletions docs/reference/esql/functions/kibana/definition/pi.json

This file was deleted.

17 changes: 0 additions & 17 deletions docs/reference/esql/functions/kibana/definition/tau.json

This file was deleted.

11 changes: 0 additions & 11 deletions docs/reference/esql/functions/kibana/docs/abs.md

This file was deleted.

11 changes: 0 additions & 11 deletions docs/reference/esql/functions/kibana/docs/avg.md

This file was deleted.

12 changes: 0 additions & 12 deletions docs/reference/esql/functions/kibana/docs/cidr_match.md

This file was deleted.

11 changes: 0 additions & 11 deletions docs/reference/esql/functions/kibana/docs/cos.md

This file was deleted.

11 changes: 0 additions & 11 deletions docs/reference/esql/functions/kibana/docs/cosh.md

This file was deleted.

10 changes: 0 additions & 10 deletions docs/reference/esql/functions/kibana/docs/e.md

This file was deleted.

11 changes: 0 additions & 11 deletions docs/reference/esql/functions/kibana/docs/from_base64.md

This file was deleted.

14 changes: 0 additions & 14 deletions docs/reference/esql/functions/kibana/docs/kql.md

This file was deleted.

14 changes: 0 additions & 14 deletions docs/reference/esql/functions/kibana/docs/left.md

This file was deleted.

11 changes: 0 additions & 11 deletions docs/reference/esql/functions/kibana/docs/max.md

This file was deleted.

13 changes: 0 additions & 13 deletions docs/reference/esql/functions/kibana/docs/md5.md

This file was deleted.

11 changes: 0 additions & 11 deletions docs/reference/esql/functions/kibana/docs/min.md

This file was deleted.

7 changes: 0 additions & 7 deletions docs/reference/esql/functions/kibana/docs/mv_append.md

This file was deleted.

Loading