ESQL autogenerate docs v3 (elastic#124312)

Building on the work started in elastic#123904, we now want to auto-generate most of the small subfiles from the ES|QL functions unit tests.

This work also investigates any remaining discrepancies between the original asciidoc version and the new markdown, and tries to minimize differences so the docs do not look too different.

The kibana json and markdown files are moved to a new location, and the operator docs are a little more generated than before (although still largely manual).

Author: craigtaverner

docs/docset.yml

@@ -2,8 +2,8 @@ project: 'Elasticsearch'
 exclude:
   - README.md
   - internal/*
-  - reference/esql/functions/kibana/docs/*
-  - reference/esql/functions/README.md
+  - reference/query-languages/esql/kibana/docs/**
+  - reference/query-languages/esql/README.md
 cross_links:
   - beats
   - cloud

docs/reference/data-analysis/aggregations/_snippets/search-aggregations-metrics-cardinality-aggregation-explanation.md

@@ -0,0 +1,17 @@
+* configurable precision, which decides on how to trade memory for accuracy,
+* excellent accuracy on low-cardinality sets,
+* fixed memory usage: no matter if there are tens or billions of unique values, memory usage only depends on the configured precision.
+
+For a precision threshold of `c`, the implementation that we are using requires about `c * 8` bytes.
+
+The following chart shows how the error varies before and after the threshold:
+
+![cardinality error](/images/cardinality_error.png "")
+
+For all 3 thresholds, counts have been accurate up to the configured threshold. Although not guaranteed,
+this is likely to be the case. Accuracy in practice depends on the dataset in question. In general,
+most datasets show consistently good accuracy. Also note that even with a threshold as low as 100,
+the error remains very low (1-6% as seen in the above graph) even when counting millions of items.
+
+The HyperLogLog++ algorithm depends on the leading zeros of hashed values, the exact distributions of
+hashes in a dataset can affect the accuracy of the cardinality.

docs/reference/data-analysis/aggregations/_snippets/search-aggregations-metrics-percentile-aggregation-approximate.md

@@ -0,0 +1,17 @@
+There are many different algorithms to calculate percentiles. The naive implementation simply stores all the values in a sorted array. To find the 50th percentile, you simply find the value that is at `my_array[count(my_array) * 0.5]`.
+
+Clearly, the naive implementation does not scale — the sorted array grows linearly with the number of values in your dataset. To calculate percentiles across potentially billions of values in an Elasticsearch cluster, *approximate* percentiles are calculated.
+
+The algorithm used by the `percentile` metric is called TDigest (introduced by Ted Dunning in [Computing Accurate Quantiles using T-Digests](https://github.com/tdunning/t-digest/blob/master/docs/t-digest-paper/histo.pdf)).
+
+When using this metric, there are a few guidelines to keep in mind:
+
+* Accuracy is proportional to `q(1-q)`. This means that extreme percentiles (e.g. 99%) are more accurate than less extreme percentiles, such as the median
+* For small sets of values, percentiles are highly accurate (and potentially 100% accurate if the data is small enough).
+* As the quantity of values in a bucket grows, the algorithm begins to approximate the percentiles. It is effectively trading accuracy for memory savings. The exact level of inaccuracy is difficult to generalize, since it depends on your data distribution and volume of data being aggregated
+
+The following chart shows the relative error on a uniform distribution depending on the number of collected values and the requested percentile:
+
+![percentiles error](/images/percentiles_error.png "")
+
+It shows how precision is better for extreme percentiles. The reason why error diminishes for large number of values is that the law of large numbers makes the distribution of values more and more uniform and the t-digest tree can do a better job at summarizing it. It would not be the case on more skewed distributions.

docs/reference/data-analysis/aggregations/search-aggregations-metrics-cardinality-aggregation.md

@@ -65,19 +65,8 @@ Computing exact counts requires loading values into a hash set and returning its
 
 This `cardinality` aggregation is based on the [HyperLogLog++](https://static.googleusercontent.com/media/research.google.com/fr//pubs/archive/40671.pdf) algorithm, which counts based on the hashes of the values with some interesting properties:
 
-* configurable precision, which decides on how to trade memory for accuracy,
-* excellent accuracy on low-cardinality sets,
-* fixed memory usage: no matter if there are tens or billions of unique values, memory usage only depends on the configured precision.
-
-For a precision threshold of `c`, the implementation that we are using requires about `c * 8` bytes.
-
-The following chart shows how the error varies before and after the threshold:
-
-![cardinality error](../../../images/cardinality_error.png "")
-
-For all 3 thresholds, counts have been accurate up to the configured threshold. Although not guaranteed, this is likely to be the case. Accuracy in practice depends on the dataset in question. In general, most datasets show consistently good accuracy. Also note that even with a threshold as low as 100, the error remains very low (1-6% as seen in the above graph) even when counting millions of items.
-
-The HyperLogLog++ algorithm depends on the leading zeros of hashed values, the exact distributions of hashes in a dataset can affect the accuracy of the cardinality.
+:::{include} _snippets/search-aggregations-metrics-cardinality-aggregation-explanation.md
+:::
 
 
 ## Pre-computed hashes [_pre_computed_hashes]

docs/reference/data-analysis/aggregations/search-aggregations-metrics-percentile-aggregation.md

@@ -175,31 +175,14 @@ GET latency/_search
 
 ## Percentiles are (usually) approximate [search-aggregations-metrics-percentile-aggregation-approximation]
 
-There are many different algorithms to calculate percentiles. The naive implementation simply stores all the values in a sorted array. To find the 50th percentile, you simply find the value that is at `my_array[count(my_array) * 0.5]`.
-
-Clearly, the naive implementation does not scale — the sorted array grows linearly with the number of values in your dataset. To calculate percentiles across potentially billions of values in an Elasticsearch cluster, *approximate* percentiles are calculated.
-
-The algorithm used by the `percentile` metric is called TDigest (introduced by Ted Dunning in [Computing Accurate Quantiles using T-Digests](https://github.com/tdunning/t-digest/blob/master/docs/t-digest-paper/histo.pdf)).
-
-When using this metric, there are a few guidelines to keep in mind:
-
-* Accuracy is proportional to `q(1-q)`. This means that extreme percentiles (e.g. 99%) are more accurate than less extreme percentiles, such as the median
-* For small sets of values, percentiles are highly accurate (and potentially 100% accurate if the data is small enough).
-* As the quantity of values in a bucket grows, the algorithm begins to approximate the percentiles. It is effectively trading accuracy for memory savings. The exact level of inaccuracy is difficult to generalize, since it depends on your data distribution and volume of data being aggregated
-
-The following chart shows the relative error on a uniform distribution depending on the number of collected values and the requested percentile:
-
-![percentiles error](../../../images/percentiles_error.png "")
-
-It shows how precision is better for extreme percentiles. The reason why error diminishes for large number of values is that the law of large numbers makes the distribution of values more and more uniform and the t-digest tree can do a better job at summarizing it. It would not be the case on more skewed distributions.
+:::{include} /reference/data-analysis/aggregations/_snippets/search-aggregations-metrics-percentile-aggregation-approximate.md
+:::
 
 ::::{warning}
 Percentile aggregations are also [non-deterministic](https://en.wikipedia.org/wiki/Nondeterministic_algorithm). This means you can get slightly different results using the same data.
-
 ::::
 
 
-
 ## Compression [search-aggregations-metrics-percentile-aggregation-compression]
 
 Approximate algorithms must balance memory utilization with estimation accuracy. This balance can be controlled using a `compression` parameter:

docs/reference/query-languages/esql/README.md

@@ -0,0 +1,50 @@
+The ES|QL documentation is composed of static content and generated content.
+The static content exists in this directory and can be edited by hand.
+However, the sub-directories `_snippets`, `images` and `kibana` contain mostly
+generated content.
+
+### _snippets
+
+In `_snippets` there are files that can be included within other files
+using the [File Inclusion](https://elastic.github.io/docs-builder/syntax/file_inclusion/)
+feature of the Elastic Docs V3 system.
+Most, but not all, files in this directory are generated.
+In particular the directories `_snippets/functions/*` and `_snippets/operators/*`
+contain subdirectories that are mostly generated:
+
+* `description` - description of each function scraped from `@FunctionInfo#description`
+* `examples` - examples of each function scraped from `@FunctionInfo#examples`
+* `parameters` - description of each function's parameters scraped from `@Param`
+* `signature` - railroad diagram of the syntax to invoke each function
+* `types` - a table of each combination of support type for each parameter. These are generated from tests.
+* `layout` - a fully generated description for each function
+
+Most functions can use the generated docs generated in the `layout` directory.
+If we need something more custom for the function we can make a file in this
+directory that can `include::` any parts of the files above.
+
+To regenerate the files for a function run its tests using gradle.
+For example to generate docs for the `CASE` function:
+```
+./gradlew :x-pack:plugin:esql:test -Dtests.class='CaseTests'
+```
+
+To regenerate the files for all functions run all of ESQL's tests using gradle:
+```
+./gradlew :x-pack:plugin:esql:test
+```
+
+### images
+
+The `images` directory contains `functions` and `operators` sub-directories with
+the `*.svg` files used to describe the syntax of each function or operator.
+These are all generated by the same tests that generate the functions and operators docs above.
+
+### kibana
+
+The `kibana` directory contains `definition` and `docs` sub-directories that are generated:
+
+* `kibana/definition` - function definitions for kibana's ESQL editor
+* `kibana/docs` - the inline docs for kibana
+
+These are also generated as part of the unit tests described above.

docs/reference/query-languages/esql/_snippets/functions/appendix/count_distinct.md

@@ -0,0 +1,24 @@
+% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
+
+### Counts are approximate [esql-agg-count-distinct-approximate]
+
+Computing exact counts requires loading values into a set and returning its
+size. This doesn’t scale when working on high-cardinality sets and/or large
+values as the required memory usage and the need to communicate those
+per-shard sets between nodes would utilize too many resources of the cluster.
+
+This `COUNT_DISTINCT` function is based on the
+[HyperLogLog++](https://static.googleusercontent.com/media/research.google.com/fr//pubs/archive/40671.pdf)
+algorithm, which counts based on the hashes of the values with some interesting
+properties:
+
+:::{include} /reference/data-analysis/aggregations/_snippets/search-aggregations-metrics-cardinality-aggregation-explanation.md
+:::
+
+The `COUNT_DISTINCT` function takes an optional second parameter to configure
+the precision threshold. The `precision_threshold` options allows to trade memory
+for accuracy, and defines a unique count below which counts are expected to be
+close to accurate. Above this value, counts might become a bit more fuzzy. The
+maximum supported value is `40000`, thresholds above this number will have the
+same effect as a threshold of `40000`. The default value is `3000`.
+

docs/reference/query-languages/esql/_snippets/functions/appendix/median.md

@@ -0,0 +1,6 @@
+% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
+
+::::{warning}
+`MEDIAN` is also [non-deterministic](https://en.wikipedia.org/wiki/Nondeterministic_algorithm).
+This means you can get slightly different results using the same data.
+::::

docs/reference/query-languages/esql/_snippets/functions/appendix/median_absolute_deviation.md

@@ -0,0 +1,6 @@
+% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
+
+::::{warning}
+`MEDIAN_ABSOLUTE_DEVIATION` is also [non-deterministic](https://en.wikipedia.org/wiki/Nondeterministic_algorithm).
+This means you can get slightly different results using the same data.
+::::

docs/reference/query-languages/esql/_snippets/functions/appendix/percentile.md

@@ -0,0 +1,11 @@
+% This is generated by ESQL's AbstractFunctionTestCase. Do no edit it. See ../README.md for how to regenerate it.
+
+### `PERCENTILE` is (usually) approximate [esql-percentile-approximate]
+
+:::{include} /reference/data-analysis/aggregations/_snippets/search-aggregations-metrics-percentile-aggregation-approximate.md
+:::
+
+::::{warning}
+`PERCENTILE` is also [non-deterministic](https://en.wikipedia.org/wiki/Nondeterministic_algorithm).
+This means you can get slightly different results using the same data.
+::::