Skip to content

Commit 148ca7f

Browse files
nik9000sarog
authored andcommitted
ESQL: Track memory in evaluators (elastic#133392) (elastic#133593)
If you write very very large ESQL queries you can spend a lot of memory on the expression evaluators themselves. You can certainly do it in real life, but our tests do something like: ``` FROM foo | EVAL a0001 = n + 1 | EVAL a0002 = a0001 + 1 | EVAL a0003 = a0002 + 1 ... | EVAL a5000 = a4999 + 1 | STATS MAX(a5000) ``` Each evaluator costs like 200 bytes a pop. For thousands of evaluators this adds up. So! We have to track it. Nhat had suggested charging a flat 200 bytes a pop. I thought about it and decided that it'd be pretty easy to get the actual size. Most of the evaluators are generated and it's a fairly small generated change to pick that up. So I did. We *do* build the evaluators before we cost them, but that's fine because they are very very small. So long as we account for them, I think it's safe.
1 parent bcee261 commit 148ca7f

File tree

449 files changed

+4892
-393
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

449 files changed

+4892
-393
lines changed

benchmarks/src/main/java/org/elasticsearch/benchmark/compute/operator/AggregatorBenchmark.java

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -643,6 +643,11 @@ public Block eval(Page page) {
643643
return mask;
644644
}
645645

646+
@Override
647+
public long baseRamBytesUsed() {
648+
return 0;
649+
}
650+
646651
@Override
647652
public void close() {
648653
mask.close();

benchmarks/src/main/java/org/elasticsearch/benchmark/compute/operator/EvalBenchmark.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -132,7 +132,7 @@ public class EvalBenchmark {
132132
public String operation;
133133

134134
private static Operator operator(String operation) {
135-
return new EvalOperator(driverContext.blockFactory(), evaluator(operation));
135+
return new EvalOperator(driverContext, evaluator(operation));
136136
}
137137

138138
private static EvalOperator.ExpressionEvaluator evaluator(String operation) {

docs/changelog/133392.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 133392
2+
summary: Track memory in evaluators
3+
area: ES|QL
4+
type: bug
5+
issues: []

docs/reference/esql/functions/description/date_trunc.asciidoc

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/reference/esql/functions/description/md5.asciidoc

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/reference/esql/functions/examples/to_ip.asciidoc

Lines changed: 27 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/reference/esql/functions/kibana/definition/avg.json

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/reference/esql/functions/kibana/definition/date_trunc.json

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/reference/esql/functions/kibana/definition/md5.json

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/reference/esql/functions/kibana/definition/to_ip.json

Lines changed: 3 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)