Skip to content

Commit 72248e3

Browse files
authored
ESQL: Compute support for filtering grouping aggs (#112476)
Adds support to the compute engine for filtering which positions are processed by grouping aggs. This should allow syntax like ``` | STATS success = COUNT(*) WHERE 200 <= response_code AND response_code < 300, redirect = COUNT(*) WHERE 300 <= response_code AND response_code < 400, client_err = COUNT(*) WHERE 400 <= response_code AND response_code < 500, server_err = COUNT(*) WHERE 500 <= response_code AND response_code < 600, total_count = COUNT(*) BY hostname ``` We could translate the WHERE expression into an `ExpressionEvaluator` and run it, then plug it into the filtering support added in this PR. The actual filtering is done by creating a `FilteredGroupingAggregatorFunction` which runs wraps a regular `GroupingAggregatorFunction` first executing the filter against the incoming `Page` and then `null`ing any positions in the group that don't match. Then passing the resulting groups into the real aggregator. When the real grouping aggregator implementation sees `null` value for groups it skips collecting that position. We had to make two changes to every agg for this to work: 1. Add a method to force local group tracking mode on any aggregator. Previously this was only required if the agg encountered `null` values, but when we're filtering aggs we can no longer trust the `seen` parameter we get when building the result. This local group tracking mode let's us track what we've actually seen locally. 2. Add `Releasable` to the `AddInput` thing we use to handle chunked pages in grouping aggs. This is required because the results of the filter must be closed on completion. Both of these are fairly trivial changes, but require touching every aggregation.
1 parent c946617 commit 72248e3

File tree

68 files changed

+1145
-31
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

68 files changed

+1145
-31
lines changed

x-pack/plugin/esql/compute/gen/src/main/java/org/elasticsearch/compute/gen/GroupingAggregatorImplementer.java

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -182,6 +182,7 @@ private TypeSpec type() {
182182
builder.addMethod(addRawInputLoop(INT_VECTOR, valueVectorType(init, combine)));
183183
builder.addMethod(addRawInputLoop(INT_BLOCK, valueBlockType(init, combine)));
184184
builder.addMethod(addRawInputLoop(INT_BLOCK, valueVectorType(init, combine)));
185+
builder.addMethod(selectedMayContainUnseenGroups());
185186
builder.addMethod(addIntermediateInput());
186187
builder.addMethod(addIntermediateRowInput());
187188
builder.addMethod(evaluateIntermediate());
@@ -338,6 +339,9 @@ private TypeSpec addInput(Consumer<MethodSpec.Builder> addBlock) {
338339
addBlock.accept(vector);
339340
builder.addMethod(vector.build());
340341

342+
MethodSpec.Builder close = MethodSpec.methodBuilder("close").addAnnotation(Override.class).addModifiers(Modifier.PUBLIC);
343+
builder.addMethod(close.build());
344+
341345
return builder.build();
342346
}
343347

@@ -485,6 +489,14 @@ private void combineRawInputForBytesRef(MethodSpec.Builder builder, String block
485489
builder.addStatement("$T.combine(state, groupId, $L.getBytesRef($L, scratch))", declarationType, blockVariable, offsetVariable);
486490
}
487491

492+
private MethodSpec selectedMayContainUnseenGroups() {
493+
MethodSpec.Builder builder = MethodSpec.methodBuilder("selectedMayContainUnseenGroups");
494+
builder.addAnnotation(Override.class).addModifiers(Modifier.PUBLIC);
495+
builder.addParameter(SEEN_GROUP_IDS, "seenGroupIds");
496+
builder.addStatement("state.enableGroupIdTracking(seenGroupIds)");
497+
return builder.build();
498+
}
499+
488500
private MethodSpec addIntermediateInput() {
489501
MethodSpec.Builder builder = MethodSpec.methodBuilder("addIntermediateInput");
490502
builder.addAnnotation(Override.class).addModifiers(Modifier.PUBLIC);

x-pack/plugin/esql/compute/src/main/generated/org/elasticsearch/compute/aggregation/CountDistinctBooleanGroupingAggregatorFunction.java

Lines changed: 13 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

x-pack/plugin/esql/compute/src/main/generated/org/elasticsearch/compute/aggregation/CountDistinctBytesRefGroupingAggregatorFunction.java

Lines changed: 13 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

x-pack/plugin/esql/compute/src/main/generated/org/elasticsearch/compute/aggregation/CountDistinctDoubleGroupingAggregatorFunction.java

Lines changed: 13 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

x-pack/plugin/esql/compute/src/main/generated/org/elasticsearch/compute/aggregation/CountDistinctFloatGroupingAggregatorFunction.java

Lines changed: 13 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

x-pack/plugin/esql/compute/src/main/generated/org/elasticsearch/compute/aggregation/CountDistinctIntGroupingAggregatorFunction.java

Lines changed: 13 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

x-pack/plugin/esql/compute/src/main/generated/org/elasticsearch/compute/aggregation/CountDistinctLongGroupingAggregatorFunction.java

Lines changed: 13 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

x-pack/plugin/esql/compute/src/main/generated/org/elasticsearch/compute/aggregation/MaxBooleanGroupingAggregatorFunction.java

Lines changed: 13 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

x-pack/plugin/esql/compute/src/main/generated/org/elasticsearch/compute/aggregation/MaxBytesRefGroupingAggregatorFunction.java

Lines changed: 13 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

x-pack/plugin/esql/compute/src/main/generated/org/elasticsearch/compute/aggregation/MaxDoubleGroupingAggregatorFunction.java

Lines changed: 13 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)