Skip to content

Commit 1e79994

Browse files
committed
ESQL: Add documents_found and values_loaded (elastic#125631)
This adds `documents_found` and `values_loaded` to the to the ESQL response: ```json { "took" : 194, "is_partial" : false, "documents_found" : 100000, "values_loaded" : 200000, "columns" : [ { "name" : "a", "type" : "long" }, { "name" : "b", "type" : "long" } ], "values" : [[10, 1]] } ``` These are cheap enough to collect that we can do it for every query and return it with every response. It's small, but it still gives you a reasonable sense of how much work Elasticsearch had to go through to perform the query. I've also added these two fields to the driver profile and task status: ```json "drivers" : [ { "description" : "data", "cluster_name" : "runTask", "node_name" : "runTask-0", "start_millis" : 1742923173077, "stop_millis" : 1742923173087, "took_nanos" : 9557014, "cpu_nanos" : 9091340, "documents_found" : 5, <---- THESE "values_loaded" : 15, <---- THESE "iterations" : 6, ... ``` These are at a high level and should be easy to reason about. We'd like to extract this into a "show me how difficult this running query is" API one day. But today, just plumbing it into the debugging output is good. Any `Operator` can claim to "find documents" or "load values" by overriding a method on its `Operator.Status` implementation: ```java /** * The number of documents found by this operator. Most operators * don't find documents and will return {@code 0} here. */ default long documentsFound() { return 0; } /** * The number of values loaded by this operator. Most operators * don't load values and will return {@code 0} here. */ default long valuesLoaded() { return 0; } ``` In this PR all of the `LuceneOperator`s declare that each `position` they emit is a "document found" and the `ValuesSourceValuesSourceReaderOperator` says each value it makes is a "value loaded". That's pretty pretty much true. The `LuceneCountOperator` and `LuceneMinMaxOperator` sort of pretend that the count/min/max that they emit is a "document" - but that's good enough to give you a sense of what's going on. It's *like* document.
1 parent 82800f9 commit 1e79994

File tree

55 files changed

+1152
-358
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

55 files changed

+1152
-358
lines changed

docs/changelog/125631.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 125631
2+
summary: Add `documents_found` and `values_loaded`
3+
area: ES|QL
4+
type: enhancement
5+
issues: []

docs/reference/esql/functions/description/knn.asciidoc

Lines changed: 5 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/reference/esql/functions/examples/knn.asciidoc

Lines changed: 21 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/reference/esql/functions/functionNamedParams/knn.asciidoc

Lines changed: 13 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/reference/esql/functions/functionNamedParams/to_ip.asciidoc

Lines changed: 9 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/reference/esql/functions/layout/knn.asciidoc

Lines changed: 18 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/reference/esql/functions/parameters/knn.asciidoc

Lines changed: 12 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/reference/esql/functions/signature/knn.svg

Lines changed: 1 addition & 0 deletions
Loading

docs/reference/esql/functions/types/knn.asciidoc

Lines changed: 9 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

server/src/main/java/org/elasticsearch/TransportVersions.java

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -251,6 +251,7 @@ static TransportVersion def(int id) {
251251
public static final TransportVersion RANDOM_SAMPLER_QUERY_BUILDER_8_19 = def(8_841_0_56);
252252
public static final TransportVersion ML_INFERENCE_SAGEMAKER_ELASTIC_8_19 = def(8_841_0_57);
253253
public static final TransportVersion SPARSE_VECTOR_FIELD_PRUNING_OPTIONS_8_19 = def(8_841_0_58);
254+
public static final TransportVersion ESQL_DOCUMENTS_FOUND_AND_VALUES_LOADED_8_19 = def(8_841_0_59); // NOCOMMIT open PR in main first
254255

255256
/*
256257
* STOP! READ THIS FIRST! No, really,

0 commit comments

Comments
 (0)