ESQL: Block loader for pushing LENGTH #137217
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Creates a `BlockLoader` for pushing the `LENGTH` function down into the loader for `keyword` fields. It takes advantage of the terms dictionary so we only need to calculate the code point count once per unique term loaded. This `BlockLoader` implementation isn't plugged into the infrastructure for emitting it because we're waiting on the infrastructure we've started in elastic#137002. We'll make a follow up PR to plug this in. We're doing this mostly to demonstrate another function that we can push into field loading, in addition to the vector similarity functions we're building in elastic#137002. We don't expect `LENGTH` to be a super hot function. If it happens to be then this'll help. Before we plug this in we'll have to figure out emitting warnings from functions that we've fused to field loading. Because `LENGTH` can emit a warning, specifically when it hits a multivalued field.
elasticsearchmachine
added
the
Team:Analytics
Collaborator
|
Pinging @elastic/es-analytical-engine (Team:Analytics) |
dnhatn
approved these changes
Oct 28, 2025
Member
dnhatn
left a comment
dnhatn
left a comment
There was a problem hiding this comment.
In cases with large indices and many ordinals - such as an index with 10M documents and 10K ordinals - it might be more efficient to look up ordinals in order. However, this isn't a big concern. This looks great. Thank you, Nik!
Member
Author
I think what you are saying is "this is fine, but the |
chrisparrinello
pushed a commit
to chrisparrinello/elasticsearch
that referenced
this pull request
Nov 3, 2025
Creates a `BlockLoader` for pushing the `LENGTH` function down into the loader for `keyword` fields. It takes advantage of the terms dictionary so we only need to calculate the code point count once per unique term loaded. This `BlockLoader` implementation isn't plugged into the infrastructure for emitting it because we're waiting on the infrastructure we've started in elastic#137002. We'll make a follow up PR to plug this in. We're doing this mostly to demonstrate another function that we can push into field loading, in addition to the vector similarity functions we're building in elastic#137002. We don't expect `LENGTH` to be a super hot function. If it happens to be then this'll help. Before we plug this in we'll have to figure out emitting warnings from functions that we've fused to field loading. Because `LENGTH` can emit a warning, specifically when it hits a multivalued field.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Creates a
BlockLoaderfor pushing theLENGTHfunction down into the loader forkeywordfields. It takes advantage of the terms dictionary so we only need to calculate the code point count once per unique term loaded.This
BlockLoaderimplementation isn't plugged into the infrastructure for emitting it because we're waiting on the infrastructure we've started in #137002. We'll make a follow up PR to plug this in.We're doing this mostly to demonstrate another function that we can push into field loading, in addition to the vector similarity functions we're building in #137002. We don't expect
LENGTHto be a super hot function. If it happens to be then this'll help.Before we plug this in we'll have to figure out emitting warnings from functions that we've fused to field loading. Because
LENGTHcan emit a warning, specifically when it hits a multivalued field.