Commit e87bd19
committed
ESQL: Track memory in evaluators (#133392)
If you write very very large ESQL queries you can spend a lot of memory
on the expression evaluators themselves. You can certainly do it in real
life, but our tests do something like:
```
FROM foo
| EVAL a0001 = n + 1
| EVAL a0002 = a0001 + 1
| EVAL a0003 = a0002 + 1
...
| EVAL a5000 = a4999 + 1
| STATS MAX(a5000)
```
Each evaluator costs like 200 bytes a pop. For thousands of evaluators
this adds up. So! We have to track it.
Nhat had suggested charging a flat 200 bytes a pop. I thought about it
and decided that it'd be pretty easy to get the actual size. Most of the
evaluators are generated and it's a fairly small generated change to
pick that up. So I did.
We *do* build the evaluators before we cost them, but that's fine
because they are very very small. So long as we account for them, I
think it's safe.1 parent 46a600c commit e87bd19
File tree
449 files changed
+4892
-393
lines changed- benchmarks/src/main/java/org/elasticsearch/benchmark/compute/operator
- docs
- changelog
- reference/esql/functions
- description
- examples
- kibana
- definition
- docs
- layout
- parameters
- signature
- types
- x-pack/plugin
- esql-core/src/main/java/org/elasticsearch/xpack/esql/core/expression
- esql
- compute
- gen/src/main/java/org/elasticsearch/compute/gen
- src
- main/java/org/elasticsearch/compute
- lucene
- operator
- mvdedupe
- test/java/org/elasticsearch/compute
- aggregation
- lucene
- read
- operator
- src
- main
- generated-src/org/elasticsearch/xpack/esql/expression
- function/scalar/nulls
- predicate/operator/comparison
- generated/org/elasticsearch/xpack/esql/expression
- function/scalar
- conditional
- convert
- date
- ip
- math
- multivalue
- spatial
- string
- predicate
- logical
- operator
- arithmetic
- comparison
- java/org/elasticsearch/xpack/esql
- evaluator
- mapper
- expression
- function/scalar
- conditional
- convert
- multivalue
- nulls
- util
- predicate
- nulls
- operator/comparison
- inference
- planner
- test/java/org/elasticsearch/xpack/esql
- expression/function
- scalar
- conditional
- nulls
- inference
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
449 files changed
+4892
-393
lines changedLines changed: 5 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
643 | 643 | | |
644 | 644 | | |
645 | 645 | | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
646 | 651 | | |
647 | 652 | | |
648 | 653 | | |
| |||
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
132 | 132 | | |
133 | 133 | | |
134 | 134 | | |
135 | | - | |
| 135 | + | |
136 | 136 | | |
137 | 137 | | |
138 | 138 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
0 commit comments