Interested in Project 21: Improving OpenVINO Test Drive RAG

[Phantom65536]

My name is Elsie, and I’m a third-year Computer Science student at the University of Toronto with experience in LLMs, ML, and robotics. I'm particularly interested in developing more advanced multimodal agents, and I’d love the opportunity to contribute to this project.

I have prior experience working with RAG and both LLMs/VLMs. Below are some of my past and ongoing projects relevant to this work:

• Research in Long-Horizon Robotic Task Planning using Vision-Language Models: Developed a pipeline using Visual Language Models (VLMs) for low-level surgical robotic task planning, leveraging structured language for precise control.
• Voya: AI-Powered Travel Assistant: Built and deployed a RAG pipeline on Google Cloud using LlamaIndex and Llama 3, enabling intelligent travel recommendations with real-time search integration.
• Scalable Resume Processing Chatbot: Built a chatbot capable of processing 20 resumes simultaneously, extracting and comparing candidate qualifications for efficient hiring decisions.

To better understand the current pipeline, I have a few questions:

• Does the current RAG pipeline already include a vector store for embedding documents?
• Is there an existing mechanism for document chunking, such as using sliding windows or sentence-boundary detection?
• Are semantic segmentation or clustering techniques like k-means already applied to group related sections?
• Does the basic implementation perform normalization tasks like data cleaning before processing?

Additionally, here’s a high-level overview of my proposal and the areas I’d like to explore:

1. Agentic Framework Integration

• Iterative Reasoning Loop (LangChain or OpenAI’s API):
Integrate an agentic framework that allows the model to alternate between reasoning and acting. Use the ReAct (Reason+Act) pattern where the model generates an explanation (reasoning) followed by an action (e.g., query refinement or additional retrieval).
• Dynamic Query Refinement:
Implement a module that monitors intermediate outputs. If the model identifies ambiguity or missing context, trigger a secondary retrieval request.

2. Knowledge Graph Utilization

• Knowledge Graph Construction:
Build a domain-specific knowledge graph from curated data sources.
• Entity Linking & Enrichment:
Map document entities to nodes in the graph. Implement entity recognition (via spaCy or Hugging Face Transformers) and linking mechanisms. Use pre-trained models from Hugging Face to extract entities and relate them to the graph.
• Graph-based Retrieval:
When a query is made, perform graph traversal to extract related nodes and relationships. Integrate this enriched context back into the generation pipeline.

3. Hybrid Retrieval Strategies

• Vector-based Semantic Search:
Convert text chunks into embeddings. Store these embeddings in a vector database like FAISS.
• Keyword-based Retrieval:
Implement traditional inverted index search using libraries like Elasticsearch to quickly retrieve relevant documents based on keyword matches.
• Multi-step Retrieval Pipeline:
First perform a broad keyword-based search, then refine the result set using vector similarity to ensure contextual relevance. Combine results from Elasticsearch and FAISS by weighting or ranking algorithms.

Thank you for your time and guidance. I’m genuinely passionate about this project and eager to get involved!

@adrianboguszewski @mlukasze Would you mind connecting me with the mentors? Thank you!

[adrianboguszewski]

The mentors are @RHeckerIntel and @ArendJanKramer

[RHeckerIntel]

Hi @Phantom65536

Thank you for your message. Let me quickly get into the questions first:

1. Does the current RAG pipeline already include a vector store for embedding documents?
   Yes. To expand. We allow the user to store documents in a database and add temporary documents. Both are then queried in langchain and combined.
2. Is there an existing mechanism for document chunking, such as using sliding windows or sentence-boundary detection?
   Yes. But this is rudimentary. It's a sliding window. I did experiment with sentence boundary but I was not satisfied with the results using relatively little time.
3. Are semantic segmentation or clustering techniques like k-means already applied to group related sections?
   No. This is also where a big improvement can be made.
4. Does the basic implementation perform normalization tasks like data cleaning before processing?
   No.

As for the suggestions. I do like the suggestions and there is great freedom in choosing an approach in this project. However, our architecture is a C++ backend for inference using OpenVINO and the Langchain implementation is in dart since we built the user interface in flutter. This means that existing implementations which are usually built in python will not (easily) be accessible for use.

[Phantom65536]

Hello @RHeckerIntel, thank you for getting back to me! I’ve prepared more detailed insights based on your feedback, along with ways on how each approach can be implemented in Dart. Would you be comfortable sharing your email address so I can send you the updated ideas? If you approve, I’ll proceed with experiments for each approach.