feat: support RAG end-2-end scenario #511
Conversation
Add RAG end-2-end support by leveraging the following features: - use the RAG support from the unitxt - extend the unitxt RAG to call out the MCP tool for information retrieval Signed-off-by: Yihong Wang <[email protected]>
Reviewer's GuideThis PR adds end-to-end RAG support by introducing MCP and RAG types in the API and CRD, extending TaskRecipe serialization, enhancing validation logic for new RAG settings, updating the driver to inject RAG configuration into unitxt commands, and adding tests to verify the full retrieval workflow. Sequence diagram for end-to-end RAG retrieval with MCPsequenceDiagram
actor User
participant LMEvalJob
participant Unitxt
participant MCP
User->>LMEvalJob: Submit evaluation job (with RAG)
LMEvalJob->>Unitxt: Start RAG evaluation
Unitxt->>MCP: Retrieve context information
MCP-->>Unitxt: Return context/docs
Unitxt-->>LMEvalJob: Complete evaluation
LMEvalJob-->>User: Return results
ER diagram for TaskRecipe and RAG/MCP relationship in CRDerDiagram
TASK_RECIPE ||--o| RAG : has
RAG ||--|| MCP : uses
MCP {
string url
string tool
string payloadField
string contextField
string idField
boolean verifyCertificate
}
Class diagram for new RAG and MCP types and TaskRecipe changesclassDiagram
class TaskRecipe {
+int* DemosPoolSize
+RAG* RAG
+String() string
}
class RAG {
+MCP MCP
+String() string
}
class MCP {
+string URL
+string Tool
+string PayloadField
+string ContextField
+string IdField
+bool VerifyCertificate
+String() string
}
TaskRecipe --> RAG : rag
RAG --> MCP : mcp
Class diagram for validation logic changes (ValidateMCP and ValidateURL)classDiagram
class ValidateMCP {
+ValidateMCP(mcp *MCP) error
}
class ValidateURL {
+ValidateURL(url string) error
}
ValidateMCP ..> MCP : validates
ValidateMCP ..> ValidateURL : uses
File-Level Changes
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
|
This change requires an additional change in the unitxt to support the MCP tool calling as the information retrieval mechanism for the RAG scenario. Here is the unitxt PR for that. I believe using the MCP tool for information retrieval is a good approach to support the end-to-end RAG scenario. If we have an agreement on this, we can integrate the unitxt enhancement into the downstream lm-evaluation-harness repo. Then this PR would fulfill the controller change and leverage the new feature. |
|
PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
The improvement/change in the MCP SDK moves swiftly, especially in the security aspect, because there was a lack of security at the beginning. The implementation here aims to support |
2 participants
Add RAG end-to-end support by leveraging the following features:
related: #446
Summary by Sourcery
Add end-to-end RAG support by introducing MCP-based retrieval settings in task recipes, updating CRD schema, validation, driver logic, and tests to incorporate RAG configuration.
New Features:
Enhancements:
Documentation:
Tests: