Esql text embedding function

[afoucret]

This PR introduces a new TEXT_EMBEDDING function to ES|QL, enabling users to generate dense vector embeddings for text directly within their queries.

Key Changes

1. Introduction of the TEXT_EMBEDDING Function:

   • Added a new scalar function, TEXT_EMBEDDING, to the ES|QL query language.
   • This function generates dense vector embeddings for text using a specified inference endpoint, enabling advanced semantic search capabilities.
   • Introduced a new InferenceFunction base class to support inference-based functions like TEXT_EMBEDDING.

2. Enhancements to the Inference Framework:

   • Created TextEmbeddingInferenceOperator to execute text embedding inference function
   • Added InferenceFunctionEvaluator for constant folding and optimization of inference functions during query execution
   • BulkInferenceRunner now use the SEARCH thread pool to answer (required to use it in the pre optimizer)

3. Logical plan analysis

   • Enhanced the Analyzer to resolve inference functions and validate compatibility with inference endpoints.

4. Integration with Logical Plan Optimization:

   • Updated the LogicalPlanPreOptimizer to include rules for folding inference functions.

How to use it

The TEXT_EMBEDDING function can be used in the EVAL clause to generate embeddings or directly within the WHERE clause with the KNN function for semantic search.

Example 1: Generating embeddings

FROM my_index
| EVAL embedding = TEXT_EMBEDDING("my text to embed", "my_embedding_model")

Example 2: Semantic search with KNN

FROM my_index
| WHERE KNN(my_vector_field, TEXT_EMBEDDING("my search query", "my_embedding_model"))

This work is part of #131079

TODO:

• More CSV tests

[elasticsearchmachine]

Hi @afoucret, I've created a changelog YAML for you.

[github-actions]

🔍 Preview links for changed docs

• docs/reference/query-languages/esql/kibana/docs/functions/text_embedding.md

[github-actions]

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

• Check out the cumulative docs guidelines
• Reach out in the #docs Slack channel

[elasticsearchmachine]

Hi @afoucret, I've created a changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[elasticsearchmachine]

Hi @afoucret, I've created a changelog YAML for you.

[afoucret]

Changes have been merged through several PR.