-
Notifications
You must be signed in to change notification settings - Fork 25.6k
[DRAFT] ESQL: Add TEXT_EMBEDDING function for dense vector embeddings #131131
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🔍 Preview links for changed docs |
8588320 to
c30c0ec
Compare
6 tasks
c30c0ec to
63c5539
Compare
1e6b1ce to
94fdad1
Compare
# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/vector/Knn.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/inference/InferenceResolver.java # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/inference/InferenceResolverTests.java
Implements the core evaluation logic for the TEXT_EMBEDDING function in ES|QL: - Add InferenceFunctionEvaluator interface for all inference functions - Implement TextEmbeddingFunctionEvaluator with support for float/byte/bit vectors - Integration with InferenceRunner for async model execution - Proper conversion of embedding results to DENSE_VECTOR data type
Integrates the TEXT_EMBEDDING function with the ESQL execution pipeline: - Update PreOptimizer to handle TEXT_EMBEDDING function evaluation - Add TextEmbedding function definition and type validation - Integrate with InferenceServices for model execution - Add comprehensive tests in PreOptimizerTests - Update session and execution components for async function support
94fdad1 to
71d591b
Compare
|
Replaced by #134573 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Implements the
TEXT_EMBEDDINGfunction for ES|QL to generate dense vector embeddings from text using inference model.Function Signature:
TEXT_EMBEDDING(text: string, inference_id: string) -> dense_vectorExample Usage:
Implementation Status
✅ Completed in this PR:
TEXT_EMBEDDING_FUNCTIONcapability with snapshot build gatingCore Function Infrastructure
TextEmbeddingfunction class with proper type validation and serializationInferenceFunctioninterface for inference-based functionsEsqlFunctionRegistryAnalysis of the inference function (validate existence and type of the inference endpoint)
Add a pre-optimizer async phase to the ES|QL query execution
Documentation generated from the annotations
Execute the inference in the pre-optimizer
Integration tests and end-to-end validation
🚧 TODO (Before Merge):
Notes
The function is enabled only in snapshot builds.
TEXT_EMBEDDINGfunction is tracked into #131022