Skip to content

Conversation

kderusso
Copy link
Member

@kderusso kderusso commented Sep 8, 2025

Adds a new function, CHUNK that takes text from a field and returns chunks based on the requested chunking strategy.

For this PR, we're inputting a size which will correspond to the default number of words in a sentence based chunking strategy. Future planned PRs will include the support for explicit chunking settings or an inference ID on top of these defaults. Future optimizations could also include supporting a max chunk size of LIMIT and optimizations to semantic text fields.

Examples of how to call this function:

FROM wikipedia
 | WHERE MATCH(content, \"churchill\") 
 | EVAL chunks = chunk(content, 3, 20) 
 | MV_EXPAND chunks
 | KEEP chunks
 | LIMIT 10

FROM wikipedia
 | WHERE MATCH(content, \"churchill\") 
 | EVAL chunks = chunk(content, 3) 
 | MV_EXPAND chunks
 | KEEP chunks
 | LIMIT 10 

FROM wikipedia
 | WHERE MATCH(content, \"churchill\") 
 | EVAL chunks = chunk(content) 
 | MV_EXPAND chunks
 | KEEP chunks
 | LIMIT 10

@kderusso kderusso changed the title Kderusso/esql chunk function [ES|QL] Add CHUNK function Sep 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants