[ES|QL] Add CHUNK function

[kderusso]

Adds a new function, CHUNK that takes text from a field and returns chunks based on the requested chunking strategy.

For this PR, we're inputting a size which will correspond to the default number of words in a sentence based chunking strategy. Future planned PRs will include the support for explicit chunking settings or an inference ID on top of these defaults. Future optimizations could also include supporting a max chunk size of LIMIT and optimizations to semantic text fields.

Examples of how to call this function:

FROM wikipedia
 | WHERE MATCH(content, \"churchill\") 
 | EVAL chunks = chunk(content, {"num_chunks": 3, "chunk_size": 20}) 
 | MV_EXPAND chunks
 | KEEP chunks
 | LIMIT 10

FROM wikipedia
 | WHERE MATCH(content, \"churchill\") 
 | EVAL chunks = chunk(content) 
 | MV_EXPAND chunks
 | KEEP chunks
 | LIMIT 10

[github-actions]

🔍 Preview links for changed docs

• docs/reference/query-languages/esql/kibana/docs/functions/chunk.md

[github-actions]

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

• Check out the cumulative docs guidelines
• Reach out in the #docs Slack channel

[elasticsearchmachine]

Hi @kderusso, I've created a changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[carlosdelest]

This looks really good! 💯

A couple of minor issues on validation and testing.

I think it would be worth adding a VerifierTests to ensure we catch nulls on the numeric params. I believe CSV tests are getting all other testing I can think of 👍

[carlosdelest]

Why is this needed? Are we using something specific from Lucene on the chunker that will make this not to work?

[kderusso]

I can run the CSV tests via the integration spec test, but CsvTest returns null for all chunked input. Perhaps this is due to the fact that we're using the chunker? I haven't been able to debug the actual CSV test implementation.

[carlosdelest]

LGTM 👍

Minor comments, I think we can simplify the options handling on evaluation.

[carlosdelest]

I'm still confused about why this doesn't work 🤔 . We can do that in a follow up, and we have the EsqlSpecIT fields - but it would be nice to be able to run CSV tests on this.

[kderusso]

There's a good chance that we won't do it as a followup may entail also performing a Lucene query - TBD.