๐โโ๏ธ Weekly Fitness Tracker
- -
-
-
- Monday - Upper Body Strength
-
- Workout: Bench Press, Pull-ups, Shoulder Press
- Duration: 45 minutes
- Intensity: High -
- - Duration: 45 minutes
- Intensity: High -
- Calories Burned: 320
- Heart Rate Avg: 145 bpm
- Steps: 8,500 -
- - Heart Rate Avg: 145 bpm
- Steps: 8,500 -
-
-
- Tuesday - Cardio
-
- Workout: Running, Cycling
- Duration: 60 minutes
- Intensity: Medium -
- - Duration: 60 minutes
- Intensity: Medium -
- Calories Burned: 450
- Heart Rate Avg: 160 bpm
- Steps: 12,000 -
- - Heart Rate Avg: 160 bpm
- Steps: 12,000 -
-
-
- Wednesday - Lower Body Strength
-
- Workout: Squats, Deadlifts, Leg Press
- Duration: 50 minutes
- Intensity: High -
- - Duration: 50 minutes
- Intensity: High -
- Calories Burned: 380
- Heart Rate Avg: 150 bpm
- Steps: 7,800 -
- - Heart Rate Avg: 150 bpm
- Steps: 7,800 -
-
-
- Thursday - Rest Day
-
- Activity: Light stretching, walking
- Duration: 20 minutes
- Intensity: Low -
- - Duration: 20 minutes
- Intensity: Low -
- Calories Burned: 120
- Heart Rate Avg: 95 bpm
- Steps: 6,200 -
- - Heart Rate Avg: 95 bpm
- Steps: 6,200 -
๐ Weekly Summary
--
-
- Total Workouts: 4 -
- Total Calories Burned: 1,270 -
- Average Heart Rate: 137 bpm -
- Total Steps: 34,500 -
- Weight Progress: -2.1 lbs -
๐ฏ Next Week Goals
--
-
- Add 15 minutes to cardio sessions -
- Increase weight on compound exercises by 5% -
- Aim for 10,000+ steps daily -
- Try yoga for flexibility -
Building Your First AI Agent: Step-by-Step Tutorial
- -
- Goal: Create a simple AI agent that can answer questions about documents using RAG (Retrieval-Augmented Generation).
-
-
-
-
-
- Step 1: Environment Setup
-First, install the required packages:
-
-
-
-pip install openai langchain llama-index
-
-
-
-
- Step 2: Create Document Loader
-Load your documents using LlamaIndex:
-
-
-
-from llama_index.core import SimpleDirectoryReader
-documents = SimpleDirectoryReader("./data").load_data()
-
-
-
-
- Step 3: Build Vector Index
-Create a searchable index from your documents:
-
-
-
-from llama_index.core import VectorStoreIndex
-index = VectorStoreIndex.from_documents(documents)
-
-
-
-
- Step 4: Create Query Engine
-Set up the query interface:
-
-
-
-query_engine = index.as_query_engine()
-response = query_engine.query("What are the main types of AI agents?")
-
- Key Concepts
--
-
- RAG (Retrieval-Augmented Generation): Combines information retrieval with text generation -
- Vector Index: Enables semantic search through document embeddings -
- Query Engine: Processes user questions and generates responses -
- Document Loader: Handles various file formats (PDF, TXT, CSV, etc.) -
Best Practices
--
-
- Use clear, specific questions for better results -
- Chunk large documents appropriately -
- Choose the right embedding model for your domain -
- Monitor and evaluate agent performance regularly -
Next Steps
-Once you have a basic agent working, consider:
--
-
- Adding memory capabilities -
- Integrating external tools and APIs -
- Implementing multi-agent collaboration -
- Fine-tuning for specific domains -
- Tip: Start simple and iterate. The most effective agents often have focused, well-defined capabilities rather than trying to do everything.
-
-
-
diff --git a/Girish_Basavaraj_Hiremath/session_2/data/images/agent_performance_comparison.png b/Girish_Basavaraj_Hiremath/session_2/data/images/agent_performance_comparison.png
deleted file mode 100644
index ed7cbfb..0000000
Binary files a/Girish_Basavaraj_Hiremath/session_2/data/images/agent_performance_comparison.png and /dev/null differ
diff --git a/Girish_Basavaraj_Hiremath/session_2/data/images/agent_types_comparison.png b/Girish_Basavaraj_Hiremath/session_2/data/images/agent_types_comparison.png
deleted file mode 100644
index 83e3de3..0000000
Binary files a/Girish_Basavaraj_Hiremath/session_2/data/images/agent_types_comparison.png and /dev/null differ
diff --git a/Girish_Basavaraj_Hiremath/session_2/data/markdown/agent_framework_comparison.md b/Girish_Basavaraj_Hiremath/session_2/data/markdown/agent_framework_comparison.md
deleted file mode 100644
index 5e65825..0000000
--- a/Girish_Basavaraj_Hiremath/session_2/data/markdown/agent_framework_comparison.md
+++ /dev/null
@@ -1,83 +0,0 @@
-# AI Agent Framework Comparison
-
-## Overview
-
-This document provides a comprehensive comparison of popular AI agent frameworks available in 2024.
-
-## Framework Categories
-
-### 1. Autonomous Agents
-- **AutoGPT**: Pioneer in autonomous task execution
-- **BabyAGI**: Simplified approach to autonomous AI
-- **AgentGPT**: Web-based autonomous agent platform
-
-### 2. Tool-Using Agents
-- **LangChain**: Comprehensive framework for LLM applications
-- **LlamaIndex**: Specialized in document understanding and RAG
-- **Semantic Kernel**: Microsoft's approach to AI orchestration
-
-### 3. Multi-Agent Systems
-- **CrewAI**: Collaborative agent teams
-- **MetaGPT**: Multi-agent software development
-- **AutoGen**: Microsoft's multi-agent conversation framework
-
-## Key Differences
-
-| Framework | Complexity | Learning Curve | Best Use Case |
-|-----------|------------|----------------|---------------|
-| LangChain | Medium | Moderate | General LLM apps |
-| AutoGPT | High | Steep | Autonomous tasks |
-| CrewAI | Medium | Easy | Team collaboration |
-| LlamaIndex | Low | Easy | Document Q&A |
-
-## Code Examples
-
-### LangChain Agent Setup
-```python
-from langchain.agents import initialize_agent
-from langchain.tools import Tool
-
-agent = initialize_agent(
- tools=[search_tool, calculator_tool],
- llm=llm,
- agent_type="zero-shot-react-description"
-)
-```
-
-### CrewAI Team Setup
-```python
-from crewai import Agent, Task, Crew
-
-researcher = Agent(
- role='Researcher',
- goal='Find relevant information',
- backstory='Expert in information gathering'
-)
-
-crew = Crew(
- agents=[researcher],
- tasks=[research_task],
- verbose=True
-)
-```
-
-## Performance Considerations
-
-- **Latency**: Single agents typically faster than multi-agent systems
-- **Accuracy**: Multi-agent systems often more accurate for complex tasks
-- **Cost**: More agents = higher API costs
-- **Reliability**: Simpler frameworks generally more stable
-
-## Choosing the Right Framework
-
-1. **For beginners**: Start with LlamaIndex or simple LangChain
-2. **For complex tasks**: Consider AutoGPT or multi-agent systems
-3. **For production**: LangChain or Semantic Kernel offer better stability
-4. **For research**: Experiment with cutting-edge frameworks like MetaGPT
-
-## Resources
-
-- [LangChain Documentation](https://docs.langchain.com)
-- [LlamaIndex Guide](https://docs.llamaindex.ai)
-- [CrewAI Examples](https://github.com/joaomdmoura/crewAI)
-- [Agent Development Best Practices](https://example.com/best-practices)
diff --git a/Girish_Basavaraj_Hiremath/session_2/data/travel/city_guides.md b/Girish_Basavaraj_Hiremath/session_2/data/travel/city_guides.md
deleted file mode 100644
index 709847c..0000000
--- a/Girish_Basavaraj_Hiremath/session_2/data/travel/city_guides.md
+++ /dev/null
@@ -1,44 +0,0 @@
-# Ultimate City Travel Guide
-
-## Paris, France ๐ซ๐ท
-
-**Best Time to Visit:** April-June, September-October
-**Must-See Attractions:**
-- Eiffel Tower - Iconic iron lattice tower
-- Louvre Museum - World's largest art museum
-- Notre-Dame Cathedral - Gothic masterpiece
-- Champs-รlysรฉes - Famous shopping avenue
-
-**Local Cuisine:** Croissants, Escargot, Coq au Vin, Macarons
-**Transportation:** Metro system, Vรฉlib bike sharing
-**Budget:** โฌ100-150 per day for mid-range travel
-
----
-
-## Tokyo, Japan ๐ฏ๐ต
-
-**Best Time to Visit:** March-May (cherry blossoms), September-November
-**Must-See Attractions:**
-- Senso-ji Temple - Ancient Buddhist temple
-- Shibuya Crossing - World's busiest pedestrian crossing
-- Tokyo Skytree - Tallest tower in Japan
-- Tsukiji Fish Market - Famous tuna auctions
-
-**Local Cuisine:** Sushi, Ramen, Tempura, Yakitori
-**Transportation:** JR Pass, extensive train network
-**Budget:** ยฅ12,000-18,000 per day for mid-range travel
-
----
-
-## New York City, USA ๐บ๐ธ
-
-**Best Time to Visit:** April-June, September-November
-**Must-See Attractions:**
-- Statue of Liberty - Symbol of freedom
-- Central Park - Urban oasis
-- Times Square - Bright lights and Broadway
-- Brooklyn Bridge - Historic suspension bridge
-
-**Local Cuisine:** Pizza, Bagels, Cheesecake, Hot Dogs
-**Transportation:** Subway system, yellow taxis, Uber
-**Budget:** $150-250 per day for mid-range travel
diff --git a/Girish_Basavaraj_Hiremath/session_2/data/travel/images/city_temperatures.png b/Girish_Basavaraj_Hiremath/session_2/data/travel/images/city_temperatures.png
deleted file mode 100644
index c474618..0000000
Binary files a/Girish_Basavaraj_Hiremath/session_2/data/travel/images/city_temperatures.png and /dev/null differ
diff --git a/Girish_Basavaraj_Hiremath/session_2/llamaindex_rag/01_academic_papers_rag.ipynb b/Girish_Basavaraj_Hiremath/session_2/llamaindex_rag/01_academic_papers_rag.ipynb
deleted file mode 100644
index 2350d5d..0000000
--- a/Girish_Basavaraj_Hiremath/session_2/llamaindex_rag/01_academic_papers_rag.ipynb
+++ /dev/null
@@ -1,3270 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Academic Papers RAG System Tutorial\n",
- "\n",
- "This notebook demonstrates how to build a complete RAG (Retrieval-Augmented Generation) system for academic research papers using LlamaIndex. We'll build it step by step with independent functions that you can run and understand individually.\n",
- "\n",
- "## What is RAG?\n",
- "\n",
- "RAG combines the power of:\n",
- "- **Retrieval**: Finding relevant documents from a knowledge base\n",
- "- **Augmented Generation**: Using retrieved context to generate informed responses\n",
- "\n",
- "## System Components\n",
- "\n",
- "Our RAG system will include:\n",
- "1. **PDF Processing**: Extract text from academic papers\n",
- "2. **Document Chunking**: Split documents into searchable segments\n",
- "3. **Vector Embeddings**: Convert text to numerical representations\n",
- "4. **Vector Storage**: Store embeddings in LanceDB\n",
- "5. **Semantic Search**: Find relevant content for queries\n",
- "6. **Query Engine**: Generate responses using retrieved context\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## ๐๏ธ Storage Architecture: Why StorageContext Matters\n",
- "\n",
- "This notebook uses LlamaIndex's **StorageContext** approach, which provides significant advantages over simpler vector-only storage methods. Understanding this architecture is crucial for building production-ready RAG systems.\n",
- "\n",
- "### ๐ StorageContext vs. Simple Vector Storage\n",
- "\n",
- "| Component | StorageContext (This Notebook) | Simple Vector Store | Benefits |\n",
- "|-----------|-------------------------------|-------------------|----------|\n",
- "| **Vector Store** | โ
LanceDB embeddings | โ
LanceDB embeddings | Fast similarity search |\n",
- "| **Document Store** | โ
Original documents preserved | โ Lost after processing | Full document access |\n",
- "| **Index Store** | โ
Index metadata & structure | โ Must rebuild index | Exact reconstruction |\n",
- "| **Graph Store** | โ
Document relationships | โ No relationship data | Rich context understanding |\n",
- "\n",
- "### ๐ Persistence & Recovery Capabilities\n",
- "\n",
- "**With StorageContext (Our Approach):**\n",
- "```python\n",
- "# Save complete system state\n",
- "index.storage_context.persist(persist_dir=\"storage/papers_index\")\n",
- "\n",
- "# Perfect restoration - identical behavior\n",
- "storage_context = StorageContext.from_defaults(persist_dir=\"storage/papers_index\")\n",
- "index = load_index_from_storage(storage_context)\n",
- "# ๐ฏ Exact same results every time!\n",
- "```\n",
- "\n",
- "**Simple Vector Store Only:**\n",
- "```python\n",
- "# Only vectors saved\n",
- "vector_store = LanceDBVectorStore(uri=\"./vectors\")\n",
- "\n",
- "# Must recreate everything from scratch\n",
- "index = VectorStoreIndex.from_vector_store(vector_store)\n",
- "# โ ๏ธ May have different behavior, lost metadata\n",
- "```\n",
- "\n",
- "### ๐ก Key Advantages of StorageContext\n",
- "\n",
- "1. **๐ Perfect Reproducibility**: Identical results across sessions - critical for research and development\n",
- "2. **๐ฆ Complete State Management**: All components preserved, not just vectors\n",
- "3. **โก Fast Startup**: No reprocessing needed - load existing index instantly\n",
- "4. **๐ Rich Metadata**: Document relationships, source tracking, and complex queries\n",
- "5. **๐ ๏ธ Development Friendly**: Iterate without rebuilding entire system\n",
- "6. **๐ฏ Enterprise Ready**: Robust persistence for production deployments\n",
- "\n",
- "### ๐ Storage Footprint Example\n",
- "\n",
- "For 1000 academic papers (~500MB original PDFs):\n",
- "\n",
- "**StorageContext Storage:**\n",
- "```\n",
- "storage/papers_index/\n",
- "โโโ docstore.json # 50MB - Original documents\n",
- "โโโ index_store.json # 5MB - Index metadata \n",
- "โโโ graph_store.json # 2MB - Relationships\n",
- "โโโ LanceDB vector files # 200MB - Embeddings\n",
- "Total: ~260MB\n",
- "```\n",
- "\n",
- "**Benefits**: Complete system restoration, full metadata, relationships preserved\n",
- "\n",
- "**Simple Vector Storage:**\n",
- "```\n",
- "lancedb_data/\n",
- "โโโ vectors.lance # 200MB - Embeddings only\n",
- "Total: ~200MB\n",
- "```\n",
- "\n",
- "**Limitations**: Must rebuild index, lost metadata, no relationships\n",
- "\n",
- "### ๐ฏ When to Use StorageContext\n",
- "\n",
- "โ
**Research & Development** - Need reproducible experiments \n",
- "โ
**Complex Documents** - Rich metadata and relationships matter \n",
- "โ
**Production Systems** - Robust persistence and recovery required \n",
- "โ
**Academic Work** - Full traceability and citation tracking \n",
- "โ
**Multi-user Systems** - Consistent experience across users \n",
- "\n",
- "### ๐ Performance Impact\n",
- "\n",
- "- **Initial Build**: ~20% slower (stores additional metadata)\n",
- "- **Subsequent Loads**: 10x faster (no reprocessing needed)\n",
- "- **Query Performance**: Identical to simple vector approach\n",
- "- **Storage Space**: ~30% more storage for complete persistence\n",
- "\n",
- "This tutorial demonstrates the StorageContext approach because it provides the most robust and feature-complete RAG implementation suitable for real-world applications.\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 1. Environment Setup and Configuration\n",
- "\n",
- "First, let's set up our environment and load necessary configurations. We'll use OpenRouter for LLM access and local embeddings (no API keys needed for embeddings).\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# !pip install -r \"../requirements.txt\""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 20,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Environment variables loaded successfully\n",
- "Environment setup complete!\n"
- ]
- }
- ],
- "source": [
- "import os\n",
- "import time\n",
- "from pathlib import Path\n",
- "from typing import Dict, List, Optional, Tuple\n",
- "\n",
- "from dotenv import load_dotenv\n",
- "\n",
- "def setup_environment():\n",
- " \"\"\"\n",
- " Setup environment variables and basic configuration.\n",
- " \n",
- " Returns:\n",
- " bool: Success status\n",
- " \"\"\"\n",
- " # Load environment variables from .env file\n",
- " load_dotenv()\n",
- " \n",
- " # Disable tokenizer warning\n",
- " os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n",
- " \n",
- " # Check for required API key\n",
- " api_key = os.getenv(\"OPENROUTER_API_KEY\")\n",
- " if not api_key:\n",
- " print(\"โ ๏ธ OPENROUTER_API_KEY not found in environment variables\")\n",
- " print(\"Please add your OpenRouter API key to a .env file\")\n",
- " return False\n",
- " \n",
- " print(\"โ Environment variables loaded successfully\")\n",
- " return True\n",
- "\n",
- "# Run the setup\n",
- "success = setup_environment()\n",
- "if success:\n",
- " print(\"Environment setup complete!\")\n",
- "else:\n",
- " print(\"Environment setup failed!\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 2. Configuration Management\n",
- "\n",
- "Let's define our system configuration directly in the notebook. This includes model settings, chunk sizes, and other parameters.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 21,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "LLM model: gpt-4o\n",
- "Embedding model: local:BAAI/bge-small-en-v1.5\n",
- "Chunk size: 1024\n",
- "โ Configuration setup complete\n"
- ]
- }
- ],
- "source": [
- "# Configuration parameters for the RAG system\n",
- "CONFIG = {\n",
- " \"llm\": {\n",
- " \"model\": \"gpt-4o\", # OpenRouter model to use\n",
- " \"temperature\": 0.1 # Temperature for response generation\n",
- " },\n",
- " \"embeddings\": {\n",
- " \"model\": \"local:BAAI/bge-small-en-v1.5\", # Local embedding model (no API key needed)\n",
- " \"chunk_size\": 1024, # Size of text chunks for processing\n",
- " \"chunk_overlap\": 100 # Overlap between consecutive chunks\n",
- " },\n",
- " \"vector_store\": {\n",
- " \"type\": \"lancedb\", # Vector database type\n",
- " \"table_name\": \"academic_papers\", # Table name for storing embeddings\n",
- " \"path\": \"storage/papers_vectordb\" # Path to vector database\n",
- " },\n",
- " \"index\": {\n",
- " \"storage_path\": \"storage/papers_index\", # Path to store complete index\n",
- " \"similarity_top_k\": 5 # Number of similar chunks to retrieve\n",
- " },\n",
- " \"papers\": {\n",
- " \"folder\": \"../papers/agents\" # Path to academic papers folder\n",
- " }\n",
- "}\n",
- "\n",
- "def get_config(key_path: str, default_value=None):\n",
- " \"\"\"\n",
- " Get configuration value using dot notation.\n",
- " \n",
- " Args:\n",
- " key_path (str): Dot-separated path to the config value (e.g., 'llm.model')\n",
- " default_value: Default value if key not found\n",
- " \n",
- " Returns:\n",
- " Configuration value or default\n",
- " \"\"\"\n",
- " keys = key_path.split('.')\n",
- " value = CONFIG\n",
- " \n",
- " for key in keys:\n",
- " if isinstance(value, dict) and key in value:\n",
- " value = value[key]\n",
- " else:\n",
- " return default_value\n",
- " \n",
- " return value\n",
- "\n",
- "# Test configuration access\n",
- "llm_model = get_config(\"llm.model\")\n",
- "embedding_model = get_config(\"embeddings.model\")\n",
- "chunk_size = get_config(\"embeddings.chunk_size\")\n",
- "\n",
- "print(f\"LLM model: {llm_model}\")\n",
- "print(f\"Embedding model: {embedding_model}\")\n",
- "print(f\"Chunk size: {chunk_size}\")\n",
- "print(\"โ Configuration setup complete\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 3. LlamaIndex Settings Configuration\n",
- "\n",
- "LlamaIndex uses global settings for embeddings, LLMs, and document processing. We'll use OpenRouter for the LLM and a local embedding model (no API key required).\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 22,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 12:47:37,701 - INFO - Load pretrained SentenceTransformer: BAAI/bge-small-en-v1.5\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ LLM configured: gpt-4o (temperature: 0.1)\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 12:47:42,324 - INFO - 1 prompt is loaded, with the key: query\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Embedding model configured: local:BAAI/bge-small-en-v1.5\n",
- "โ Text chunking configured: 1024 chars with 100 overlap\n",
- "โ LlamaIndex settings configured successfully\n"
- ]
- }
- ],
- "source": [
- "from llama_index.core import Settings\n",
- "from llama_index.llms.openrouter import OpenRouter\n",
- "from llama_index.core.embeddings import resolve_embed_model\n",
- "from llama_index.core.node_parser import SentenceSplitter\n",
- "\n",
- "def configure_llamaindex_settings():\n",
- " \"\"\"\n",
- " Configure LlamaIndex global settings for embeddings, LLM, and text processing.\n",
- " \"\"\"\n",
- " # Set up LLM with OpenRouter\n",
- " model = get_config(\"llm.model\")\n",
- " temperature = get_config(\"llm.temperature\", 0.1)\n",
- " \n",
- " Settings.llm = OpenRouter(\n",
- " api_key=os.getenv(\"OPENROUTER_API_KEY\"),\n",
- " model=model,\n",
- " temperature=temperature\n",
- " )\n",
- " print(f\"โ LLM configured: {model} (temperature: {temperature})\")\n",
- "\n",
- " # Set up local embedding model (downloads locally first time, then cached)\n",
- " embedding_model = get_config(\"embeddings.model\")\n",
- " Settings.embed_model = resolve_embed_model(embedding_model)\n",
- " print(f\"โ Embedding model configured: {embedding_model}\")\n",
- "\n",
- " # Set up node parser for chunking\n",
- " chunk_size = get_config(\"embeddings.chunk_size\")\n",
- " chunk_overlap = get_config(\"embeddings.chunk_overlap\")\n",
- " \n",
- " Settings.node_parser = SentenceSplitter(\n",
- " chunk_size=chunk_size, \n",
- " chunk_overlap=chunk_overlap\n",
- " )\n",
- " print(f\"โ Text chunking configured: {chunk_size} chars with {chunk_overlap} overlap\")\n",
- "\n",
- "# Configure the settings using our hardcoded config\n",
- "configure_llamaindex_settings()\n",
- "print(\"โ LlamaIndex settings configured successfully\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 4. Vector Store Setup\n",
- "\n",
- "We'll use LanceDB as our vector database to store document embeddings. LanceDB is a fast, serverless vector database that's perfect for RAG applications.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 23,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Connected to LanceDB at: storage/papers_vectordb\n",
- "โ LanceDB vector store created (table: academic_papers)\n",
- "โ Vector store setup complete\n"
- ]
- }
- ],
- "source": [
- "from llama_index.vector_stores.lancedb import LanceDBVectorStore\n",
- "\n",
- "def create_vector_store():\n",
- " \"\"\"\n",
- " Create and configure LanceDB vector store using config settings.\n",
- " \n",
- " Returns:\n",
- " LanceDBVectorStore: Configured vector store\n",
- " \"\"\"\n",
- " try:\n",
- " import lancedb\n",
- " \n",
- " # Get configuration values\n",
- " vector_db_path = get_config(\"vector_store.path\")\n",
- " table_name = get_config(\"vector_store.table_name\")\n",
- " \n",
- " # Create storage directory\n",
- " Path(vector_db_path).parent.mkdir(parents=True, exist_ok=True)\n",
- " \n",
- " # Connect to LanceDB\n",
- " db = lancedb.connect(str(vector_db_path))\n",
- " print(f\"โ Connected to LanceDB at: {vector_db_path}\")\n",
- " \n",
- " # Create vector store\n",
- " vector_store = LanceDBVectorStore(\n",
- " uri=str(vector_db_path), \n",
- " table_name=table_name\n",
- " )\n",
- " print(f\"โ LanceDB vector store created (table: {table_name})\")\n",
- " \n",
- " return vector_store\n",
- " \n",
- " except Exception as e:\n",
- " print(f\"Error creating vector store: {e}\")\n",
- " return None\n",
- "\n",
- "# Create the vector store using config\n",
- "vector_store = create_vector_store()\n",
- "if vector_store:\n",
- " print(\"โ Vector store setup complete\")\n",
- "else:\n",
- " print(\"โ Vector store setup failed\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 5. PDF Processing and Document Loading\n",
- "\n",
- "Now we'll create functions to load and process PDF files. We'll use LlamaIndex's native `SimpleDirectoryReader` which can handle PDFs directly without needing a custom processor.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 24,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Loading papers from: ../papers/agents\n",
- "โ Loaded 229 documents\n",
- "Successfully loaded 229 documents\n",
- "First document preview: AI Agents vs. Agentic AI: A Conceptual\n",
- "Taxonomy, Applications and Challenges\n",
- "Ranjan Sapkotaโโก, Konstantinos I. Roumeliotis โ , Manoj Karkee โโก\n",
- "โCornell University, Department of Environmental and Biolo...\n",
- "First document metadata: {'page_label': '1', 'file_name': 'AI_Agents_vs_Agentic_AI.pdf', 'file_path': '/Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf', 'file_type': 'application/pdf', 'file_size': 3196781, 'creation_date': '2025-09-20', 'last_modified_date': '2025-09-20'}\n"
- ]
- }
- ],
- "source": [
- "from llama_index.core import SimpleDirectoryReader\n",
- "\n",
- "def load_papers_from_folder() -> List:\n",
- " \"\"\"\n",
- " Load and process all PDF papers from the configured folder using LlamaIndex's native loader.\n",
- " \n",
- " Returns:\n",
- " List[Document]: Processed documents ready for indexing\n",
- " \"\"\"\n",
- " papers_folder = get_config(\"papers.folder\")\n",
- " print(f\"Loading papers from: {papers_folder}\")\n",
- " \n",
- " papers_path = Path(papers_folder)\n",
- " if not papers_path.exists():\n",
- " print(f\"Papers folder does not exist: {papers_path}\")\n",
- " return []\n",
- " \n",
- " # Use LlamaIndex's SimpleDirectoryReader to load PDFs\n",
- " # This natively handles PDF parsing, text extraction, and metadata\n",
- " documents = SimpleDirectoryReader(papers_folder).load_data()\n",
- " \n",
- " print(f\"โ Loaded {len(documents)} documents\")\n",
- " return documents\n",
- "\n",
- "# Load the papers using config\n",
- "documents = load_papers_from_folder()\n",
- "if documents:\n",
- " print(f\"Successfully loaded {len(documents)} documents\")\n",
- " print(f\"First document preview: {documents[0].text[:200]}...\")\n",
- " print(f\"First document metadata: {documents[0].metadata}\")\n",
- "else:\n",
- " print(\"No documents loaded\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 6. Creating the Vector Index\n",
- "\n",
- "The vector index is the core of our RAG system. It chunks documents, generates embeddings, and stores them in the vector database for efficient similarity search.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 25,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 12:47:57,839 - INFO - Loading all indices.\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "๐ Loading existing index...\n",
- "Loading llama_index.core.storage.kvstore.simple_kvstore from storage/papers_index/docstore.json.\n",
- "Loading llama_index.core.storage.kvstore.simple_kvstore from storage/papers_index/index_store.json.\n",
- "โ Successfully loaded existing index\n",
- "โ Vector index ready for querying\n"
- ]
- }
- ],
- "source": [
- "from llama_index.core import StorageContext, VectorStoreIndex, load_index_from_storage\n",
- "\n",
- "def create_vector_index(documents: List, \n",
- " vector_store, \n",
- " force_rebuild: bool = False):\n",
- " \"\"\"\n",
- " Create or load a vector index from documents using config settings.\n",
- " \n",
- " Args:\n",
- " documents (List): Documents to index\n",
- " vector_store: LanceDB vector store\n",
- " force_rebuild (bool): Force rebuild even if index exists\n",
- " \n",
- " Returns:\n",
- " VectorStoreIndex: The created or loaded index\n",
- " \"\"\"\n",
- " index_storage_path = get_config(\"index.storage_path\")\n",
- " index_path = Path(index_storage_path)\n",
- " index_path.mkdir(parents=True, exist_ok=True)\n",
- " \n",
- " # Check if index already exists\n",
- " index_store_file = index_path / \"index_store.json\"\n",
- " \n",
- " if not force_rebuild and index_store_file.exists():\n",
- " print(\"๐ Loading existing index...\")\n",
- " try:\n",
- " # Recreate storage context with vector store\n",
- " storage_context = StorageContext.from_defaults(\n",
- " persist_dir=str(index_path), \n",
- " vector_store=vector_store\n",
- " )\n",
- " \n",
- " # Load existing index\n",
- " index = load_index_from_storage(storage_context)\n",
- " print(\"โ Successfully loaded existing index\")\n",
- " return index\n",
- " \n",
- " except Exception as e:\n",
- " print(f\"โ ๏ธ Error loading existing index: {e}\")\n",
- " print(\"Creating new index...\")\n",
- " \n",
- " if not documents:\n",
- " print(\"โ No documents to index\")\n",
- " return None\n",
- " \n",
- " print(\"๐จ Creating new vector index...\")\n",
- " start_time = time.time()\n",
- " \n",
- " # Create storage context with vector store\n",
- " storage_context = StorageContext.from_defaults(vector_store=vector_store)\n",
- " \n",
- " # Create index with progress bar\n",
- " index = VectorStoreIndex.from_documents(\n",
- " documents, \n",
- " storage_context=storage_context, \n",
- " show_progress=True\n",
- " )\n",
- " \n",
- " end_time = time.time()\n",
- " print(f\"โ Index created in {end_time - start_time:.2f} seconds\")\n",
- " \n",
- " # Save index to storage\n",
- " print(\"๐พ Saving index to storage...\")\n",
- " index.storage_context.persist(persist_dir=str(index_path))\n",
- " print(\"โ Index saved successfully\")\n",
- " \n",
- " return index\n",
- "\n",
- "# Create the vector index using config\n",
- "index = create_vector_index(\n",
- " documents=documents, \n",
- " vector_store=vector_store, \n",
- " force_rebuild=False # Set to True to force rebuild\n",
- ")\n",
- "\n",
- "if index:\n",
- " print(\"โ Vector index ready for querying\")\n",
- "else:\n",
- " print(\"โ Failed to create vector index\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 7. Setting Up the Query Engine\n",
- "\n",
- "The query engine combines a retriever (to find relevant documents) with an LLM (to generate responses). This is where the \"Augmented Generation\" part of RAG happens.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 26,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Retriever configured to find top 5 similar chunks\n",
- "โ Query engine setup successfully\n",
- "๐ RAG system is ready for queries!\n"
- ]
- }
- ],
- "source": [
- "from llama_index.core.query_engine import RetrieverQueryEngine\n",
- "from llama_index.core.retrievers import VectorIndexRetriever\n",
- "\n",
- "def setup_query_engine(index):\n",
- " \"\"\"\n",
- " Setup the query engine for semantic search and response generation using config settings.\n",
- " \n",
- " Args:\n",
- " index: The vector index to query\n",
- " \n",
- " Returns:\n",
- " RetrieverQueryEngine: Configured query engine\n",
- " \"\"\"\n",
- " if not index:\n",
- " print(\"โ Index not available. Please create index first.\")\n",
- " return None\n",
- " \n",
- " try:\n",
- " # Get similarity top k from config\n",
- " similarity_top_k = get_config(\"index.similarity_top_k\")\n",
- " \n",
- " # Create retriever - this finds the most similar document chunks\n",
- " retriever = VectorIndexRetriever(\n",
- " index=index,\n",
- " similarity_top_k=similarity_top_k,\n",
- " )\n",
- " print(f\"โ Retriever configured to find top {similarity_top_k} similar chunks\")\n",
- " \n",
- " # Create query engine - this combines retrieval with LLM generation\n",
- " query_engine = RetrieverQueryEngine(retriever=retriever)\n",
- " print(\"โ Query engine setup successfully\")\n",
- " \n",
- " return query_engine\n",
- " \n",
- " except Exception as e:\n",
- " print(f\"โ Error setting up query engine: {e}\")\n",
- " return None\n",
- "\n",
- "# Setup the query engine using config\n",
- "query_engine = setup_query_engine(index)\n",
- "\n",
- "if query_engine:\n",
- " print(\"๐ RAG system is ready for queries!\")\n",
- "else:\n",
- " print(\"โ Failed to setup query engine\")\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 27,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "๐ Title and Author extraction functions loaded successfully!\n"
- ]
- }
- ],
- "source": [
- "def extract_paper_title_from_text(text: str, max_length: int = 200) -> str:\n",
- " \"\"\"\n",
- " Extract the paper title from the document text.\n",
- " \n",
- " Args:\n",
- " text (str): Document text content\n",
- " max_length (int): Maximum length for title extraction\n",
- " \n",
- " Returns:\n",
- " str: Extracted title or fallback\n",
- " \"\"\"\n",
- " if not text:\n",
- " return \"Unknown Title\"\n",
- " \n",
- " # Split into lines and clean them\n",
- " lines = [line.strip() for line in text.split('\\n') if line.strip()]\n",
- " \n",
- " if not lines:\n",
- " return \"Unknown Title\"\n",
- " \n",
- " # Look for title patterns - usually the first substantial line\n",
- " # Skip very short lines, page numbers, headers\n",
- " for line in lines[:10]: # Check first 10 lines\n",
- " # Skip lines that look like headers, page numbers, or metadata\n",
- " if (len(line) > 15 and \n",
- " not line.isdigit() and \n",
- " not line.startswith(('Page', 'arXiv:', 'doi:', 'http', 'www')) and\n",
- " not all(c.isupper() or c.isspace() for c in line) and # Skip all-caps headers\n",
- " '.' in line or len(line) > 30): # Likely a title if it has punctuation or is long\n",
- " \n",
- " # Clean up the title\n",
- " title = line.strip()\n",
- " \n",
- " # Remove common prefixes/suffixes\n",
- " prefixes_to_remove = ['Title:', 'Abstract:', 'Paper:', 'Research:']\n",
- " for prefix in prefixes_to_remove:\n",
- " if title.startswith(prefix):\n",
- " title = title[len(prefix):].strip()\n",
- " \n",
- " # Truncate if too long\n",
- " if len(title) > max_length:\n",
- " title = title[:max_length].strip() + \"...\"\n",
- " \n",
- " return title\n",
- " \n",
- " # Fallback: use first non-empty line, truncated\n",
- " first_line = lines[0] if lines else \"Unknown Title\"\n",
- " if len(first_line) > max_length:\n",
- " first_line = first_line[:max_length].strip() + \"...\"\n",
- " \n",
- " return first_line\n",
- "\n",
- "def extract_paper_authors_from_text(text: str) -> str:\n",
- " \"\"\"\n",
- " Extract authors from the document text.\n",
- " \n",
- " Args:\n",
- " text (str): Document text content\n",
- " \n",
- " Returns:\n",
- " str: Extracted authors or \"Unknown Authors\"\n",
- " \"\"\"\n",
- " if not text:\n",
- " return \"Unknown Authors\"\n",
- " \n",
- " lines = [line.strip() for line in text.split('\\n') if line.strip()]\n",
- " \n",
- " # Look for author patterns in first 20 lines\n",
- " for i, line in enumerate(lines[:20]):\n",
- " # Skip the title line (usually first substantial line)\n",
- " if i == 0:\n",
- " continue\n",
- " \n",
- " # Look for author patterns\n",
- " if (len(line) > 5 and \n",
- " not line.isdigit() and\n",
- " not line.startswith(('Abstract', 'Introduction', 'Page', 'arXiv:', 'doi:', 'http')) and\n",
- " ('University' in line or 'Institute' in line or \n",
- " ',' in line or 'Department' in line or\n",
- " '@' in line or # Email addresses often indicate authors\n",
- " any(char.isupper() for char in line))): # Names often have capitals\n",
- " \n",
- " # Clean up author line\n",
- " authors = line.strip()\n",
- " \n",
- " # Remove common prefixes\n",
- " prefixes_to_remove = ['Authors:', 'By:', 'Author:']\n",
- " for prefix in prefixes_to_remove:\n",
- " if authors.startswith(prefix):\n",
- " authors = authors[len(prefix):].strip()\n",
- " \n",
- " # Truncate if too long\n",
- " if len(authors) > 150:\n",
- " authors = authors[:150].strip() + \"...\"\n",
- " \n",
- " return authors\n",
- " \n",
- " return \"Unknown Authors\"\n",
- "\n",
- "print(\"๐ Title and Author extraction functions loaded successfully!\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 8. Search and Query Functions\n",
- "\n",
- "Now let's create functions to search through our academic papers and extract detailed information about sources and metadata.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 28,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "๐ Re-processing papers with improved title extraction...\n",
- "๐ Found 5 unique papers in the index:\n",
- "================================================================================\n",
- "1. ๐ AI Agents vs. Agentic AI: A Conceptual\n",
- " ๐ฅ Authors: Taxonomy, Applications and Challenges\n",
- " ๐ File: AI_Agents_vs_Agentic_AI\n",
- " ๐พ Size: 3.05 MB\n",
- " ๐ Page: 1\n",
- "------------------------------------------------------------\n",
- "2. ๐ THE LANDSCAPE OF EMERGING AI AGENT ARCHITECTURES\n",
- " ๐ฅ Authors: FOR REASONING , PLANNING , AND TOOL CALLING : A S URVEY\n",
- " ๐ File: Emerging_Agent_Architectures\n",
- " ๐พ Size: 1.58 MB\n",
- " ๐ Page: 1\n",
- "------------------------------------------------------------\n",
- "3. ๐ From LLM Reasoning to Autonomous AI Agents:\n",
- " ๐ฅ Authors: From LLM Reasoning to Autonomous AI Agents:\n",
- " ๐ File: LLMReasoning_to_Autonomous_Agents\n",
- " ๐พ Size: 16.21 MB\n",
- " ๐ Page: 1\n",
- "------------------------------------------------------------\n",
- "4. ๐ The Rise and Potential of Large Language Model\n",
- " ๐ฅ Authors: Based Agents: A Survey\n",
- " ๐ File: Rise_and_Potential_LLM_Agents\n",
- " ๐พ Size: 6.52 MB\n",
- " ๐ Page: 1\n",
- "------------------------------------------------------------\n",
- "5. ๐ A Comprehensive Survey of Self-Evolving AI Agents\n",
- " ๐ฅ Authors: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems\n",
- " ๐ File: survey_of_self_evolving_agents\n",
- " ๐พ Size: 10.56 MB\n",
- " ๐ Page: 1\n",
- "------------------------------------------------------------\n",
- "\n",
- "โ
Successfully extracted titles for 5 papers!\n"
- ]
- }
- ],
- "source": [
- "# Updated list_indexed_papers function with title extraction\n",
- "def list_indexed_papers_improved(documents: List) -> List[Dict[str, any]]:\n",
- " \"\"\"\n",
- " List all papers that have been indexed with their metadata.\n",
- " Extracts actual paper titles and authors from document content.\n",
- " \n",
- " Args:\n",
- " documents (List): List of loaded documents\n",
- " \n",
- " Returns:\n",
- " List[Dict[str, any]]: List of paper information\n",
- " \"\"\"\n",
- " papers = []\n",
- " processed_files = set() # Track unique files to avoid duplicates\n",
- " \n",
- " for doc in documents:\n",
- " try:\n",
- " metadata = doc.metadata\n",
- " file_path = metadata.get(\"file_path\", \"\")\n",
- " file_name = Path(file_path).stem if file_path else \"Unknown\"\n",
- " \n",
- " # Skip if we've already processed this file\n",
- " if file_path in processed_files:\n",
- " continue\n",
- " processed_files.add(file_path)\n",
- " \n",
- " # Extract title and authors from document text\n",
- " doc_text = doc.text if hasattr(doc, 'text') else \"\"\n",
- " extracted_title = extract_paper_title_from_text(doc_text)\n",
- " extracted_authors = extract_paper_authors_from_text(doc_text)\n",
- " \n",
- " paper_info = {\n",
- " \"file_name\": file_name,\n",
- " \"file_path\": file_path,\n",
- " \"title\": extracted_title,\n",
- " \"authors\": extracted_authors,\n",
- " \"page_count\": metadata.get(\"page_count\", 0),\n",
- " \"file_size\": metadata.get(\"file_size\", 0),\n",
- " \"file_size_mb\": round(metadata.get(\"file_size\", 0) / (1024 * 1024), 2) if metadata.get(\"file_size\") else 0,\n",
- " \"total_pages\": metadata.get(\"total_pages\", \"Unknown\"),\n",
- " \"page_label\": metadata.get(\"page_label\", \"\"),\n",
- " }\n",
- " \n",
- " papers.append(paper_info)\n",
- " \n",
- " except Exception as e:\n",
- " print(f\"Error processing document: {e}\")\n",
- " \n",
- " return papers\n",
- "\n",
- "# Re-list papers with improved title extraction\n",
- "print(\"๐ Re-processing papers with improved title extraction...\")\n",
- "papers_list_improved = list_indexed_papers_improved(documents)\n",
- "\n",
- "print(f\"๐ Found {len(papers_list_improved)} unique papers in the index:\")\n",
- "print(\"=\" * 80)\n",
- "\n",
- "for i, paper in enumerate(papers_list_improved[:10], 1): # Show first 10 papers\n",
- " print(f\"{i}. ๐ {paper['title']}\")\n",
- " print(f\" ๐ฅ Authors: {paper['authors']}\")\n",
- " print(f\" ๐ File: {paper['file_name']}\")\n",
- " print(f\" ๐พ Size: {paper['file_size_mb']} MB\")\n",
- " if paper.get('page_label'):\n",
- " print(f\" ๐ Page: {paper['page_label']}\")\n",
- " print(\"-\" * 60)\n",
- "\n",
- "print(f\"\\nโ
Successfully extracted titles for {len(papers_list_improved)} papers!\")\n",
- "if len(papers_list_improved) > 10:\n",
- " print(f\"๐ Showing first 10 papers. Total: {len(papers_list_improved)} papers available.\")\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 29,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "๐ Enhanced search function with title extraction loaded!\n"
- ]
- }
- ],
- "source": [
- "# Enhanced search function with improved metadata extraction\n",
- "def search_papers_improved(query_engine, query: str, include_metadata: bool = True) -> Dict[str, any]:\n",
- " \"\"\"\n",
- " Search for relevant papers based on the query with improved title extraction.\n",
- " \n",
- " Args:\n",
- " query_engine: The configured query engine\n",
- " query (str): Search query\n",
- " include_metadata (bool): Whether to include detailed metadata\n",
- " \n",
- " Returns:\n",
- " Dict[str, any]: Search results with response and sources\n",
- " \"\"\"\n",
- " if not query_engine:\n",
- " return {\n",
- " \"success\": False,\n",
- " \"error\": \"Query engine not initialized.\",\n",
- " \"response\": \"\",\n",
- " \"sources\": [],\n",
- " }\n",
- " \n",
- " try:\n",
- " print(f\"๐ Searching for: '{query}'\")\n",
- " start_time = time.time()\n",
- " \n",
- " # Query the RAG system\n",
- " response = query_engine.query(query)\n",
- " \n",
- " end_time = time.time()\n",
- " \n",
- " # Extract source information from retrieved nodes with title extraction\n",
- " sources = []\n",
- " if hasattr(response, \"source_nodes\"):\n",
- " for node in response.source_nodes:\n",
- " # Extract title from the node text\n",
- " node_text = node.text if hasattr(node, 'text') else \"\"\n",
- " extracted_title = extract_paper_title_from_text(node_text, max_length=100)\n",
- " extracted_authors = extract_paper_authors_from_text(node_text)\n",
- " \n",
- " source_info = {\n",
- " \"text\": (\n",
- " node.text[:500] + \"...\"\n",
- " if len(node.text) > 500\n",
- " else node.text\n",
- " ),\n",
- " \"score\": getattr(node, \"score\", 0.0),\n",
- " \"extracted_title\": extracted_title,\n",
- " \"extracted_authors\": extracted_authors,\n",
- " }\n",
- " \n",
- " # Add metadata if available and requested\n",
- " if include_metadata and hasattr(node, \"metadata\"):\n",
- " metadata = node.metadata\n",
- " source_info.update({\n",
- " \"file_name\": metadata.get(\"file_name\", \"Unknown\"),\n",
- " \"file_path\": metadata.get(\"file_path\", \"\"),\n",
- " \"page_label\": metadata.get(\"page_label\", \"\"),\n",
- " \"file_size_mb\": round(metadata.get(\"file_size\", 0) / (1024 * 1024), 2) if metadata.get(\"file_size\") else 0,\n",
- " })\n",
- " \n",
- " sources.append(source_info)\n",
- " \n",
- " result = {\n",
- " \"success\": True,\n",
- " \"response\": str(response),\n",
- " \"sources\": sources,\n",
- " \"query\": query,\n",
- " \"search_time\": end_time - start_time,\n",
- " \"num_sources\": len(sources),\n",
- " }\n",
- " \n",
- " print(f\"โ Search completed in {end_time - start_time:.2f} seconds\")\n",
- " print(f\"๐ Found {len(sources)} relevant sources\")\n",
- " \n",
- " return result\n",
- " \n",
- " except Exception as e:\n",
- " print(f\"โ Error during search: {e}\")\n",
- " return {\"success\": False, \"error\": str(e), \"response\": \"\", \"sources\": []}\n",
- "\n",
- "print(\"๐ Enhanced search function with title extraction loaded!\")\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 30,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "๐งช Testing improved search with title extraction:\n",
- "================================================================================\n",
- "โ Question: What are the main architectural patterns for agent systems?\n",
- "================================================================================\n",
- "๐ Searching for: 'What are the main architectural patterns for agent systems?'\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 12:48:45,000 - INFO - query_type :, vector\n",
- "2025-09-20 12:48:48,541 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 12:48:52,913 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 12:48:54,757 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Search completed in 11.76 seconds\n",
- "๐ Found 5 relevant sources\n",
- "๐ก Answer:\n",
- "Agent systems are typically structured around several key components that enable autonomous decision-making and execution. The core of an agent system is the Foundation Model, often a large language model (LLM), which acts as the central reasoning engine. This model interprets instructions, generates plans, and produces actionable responses. Supporting modules enhance the agent's capabilities in complex environments. These include the Perception Module, which processes sensory-like data to build a suitable representation for reasoning; the Planning Module, which decomposes tasks into actionable sub-tasks and guides their execution; and the Memory Module, which retains and recalls past experiences for context-aware reasoning and long-term consistency. These architectural patterns allow agents to effectively reason, plan, and interact with their environments.\n",
- "\n",
- "๐ Search completed in 11.76 seconds\n",
- "๐ Found 5 relevant sources\n",
- "\n",
- "๐ Source Details:\n",
- "------------------------------------------------------------\n",
- "\n",
- "1. ๐ Paper: AI enhances this base by integrating advanced components\n",
- " ๐ฅ Authors: AI enhances this base by integrating advanced components\n",
- " ๐ File: AI_Agents_vs_Agentic_AI.pdf\n",
- " ๐ Page: 11\n",
- " ๐ฏ Relevance Score: 0.730\n",
- " ๐ Text Preview: Agentic\n",
- "AI enhances this base by integrating advanced components\n",
- "such as Specialized Agents, Advanced Reasoning & Plan-\n",
- "ning, Persistent Memory, and Orchestration. The figure further\n",
- "emphasizes emerge...\n",
- "\n",
- "2. ๐ Paper: Fig. 2: Mindmap of Research Questions relevant to AI\n",
- " ๐ฅ Authors: Agentic AI\n",
- " ๐ File: AI_Agents_vs_Agentic_AI.pdf\n",
- " ๐ Page: 3\n",
- " ๐ฏ Relevance Score: 0.685\n",
- " ๐ Text Preview: AI Agents\n",
- "&\n",
- "Agentic AI\n",
- "Architecture\n",
- "Mechanisms\n",
- "Scope/\n",
- "Complexity\n",
- "Interaction\n",
- "Autonomy\n",
- "Fig. 2: Mindmap of Research Questions relevant to AI\n",
- "Agents and Agentic AI. Each color-coded branch represents\n",
- "a k...\n",
- "\n",
- "3. ๐ Paper: tural distinction underpins profound differences in scalability,\n",
- " ๐ฅ Authors: tural distinction underpins profound differences in scalability,\n",
- " ๐ File: AI_Agents_vs_Agentic_AI.pdf\n",
- " ๐ Page: 2\n",
- " ๐ฏ Relevance Score: 0.685\n",
- " ๐ Text Preview: This architec-\n",
- "tural distinction underpins profound differences in scalability,\n",
- "adaptability, and application scope.\n",
- "Understanding and formalizing the taxonomy between these\n",
- "two paradigms (AI Agents a...\n",
- "\n",
- "4. ๐ Paper: task. In single agent patterns there is no feedback mechanism from other AI agents; however, there m...\n",
- " ๐ฅ Authors: Multi-Agent Architectures. These architectures involve two or more agents, where each agent can utilize the same\n",
- " ๐ File: Emerging_Agent_Architectures.pdf\n",
- " ๐ Page: 3\n",
- " ๐ฏ Relevance Score: 0.684\n",
- " ๐ Text Preview: task. In single agent patterns there is no feedback mechanism from other AI agents; however, there may be options for\n",
- "humans to provide feedback that guides the agent.\n",
- "Multi-Agent Architectures. These...\n",
- "\n",
- "5. ๐ Paper: agentic systems.\n",
- " ๐ฅ Authors: The remainder of this survey is organised as follows. Section 2 presents preliminaries on AI agents and\n",
- " ๐ File: survey_of_self_evolving_agents.pdf\n",
- " ๐ Page: 6\n",
- " ๐ฏ Relevance Score: 0.668\n",
- " ๐ Text Preview: agentic systems.\n",
- "The remainder of this survey is organised as follows. Section 2 presents preliminaries on AI agents and\n",
- "multi-agent systems, including their definitions, key components, representativ...\n"
- ]
- }
- ],
- "source": [
- "# Test the improved search with title extraction\n",
- "def ask_question_improved(query_engine, question: str, show_sources: bool = True):\n",
- " \"\"\"\n",
- " Ask a custom question to the RAG system with improved title extraction.\n",
- " \n",
- " Args:\n",
- " query_engine: The configured query engine\n",
- " question (str): Your question about the papers\n",
- " show_sources (bool): Whether to display source information\n",
- " \"\"\"\n",
- " print(f\"โ Question: {question}\")\n",
- " print(\"=\" * 80)\n",
- " \n",
- " result = search_papers_improved(query_engine, question, include_metadata=True)\n",
- " \n",
- " if result[\"success\"]:\n",
- " print(f\"๐ก Answer:\")\n",
- " print(result[\"response\"])\n",
- " print(f\"\\n๐ Search completed in {result['search_time']:.2f} seconds\")\n",
- " print(f\"๐ Found {result['num_sources']} relevant sources\")\n",
- " \n",
- " if show_sources and result[\"sources\"]:\n",
- " print(f\"\\n๐ Source Details:\")\n",
- " print(\"-\" * 60)\n",
- " for i, source in enumerate(result[\"sources\"], 1):\n",
- " print(f\"\\n{i}. ๐ Paper: {source.get('extracted_title', 'Unknown Title')}\")\n",
- " print(f\" ๐ฅ Authors: {source.get('extracted_authors', 'Unknown Authors')}\")\n",
- " print(f\" ๐ File: {source.get('file_name', 'Unknown')}\")\n",
- " if source.get('page_label'):\n",
- " print(f\" ๐ Page: {source['page_label']}\")\n",
- " print(f\" ๐ฏ Relevance Score: {source.get('score', 0):.3f}\")\n",
- " print(f\" ๐ Text Preview: {source['text'][:200]}...\")\n",
- " \n",
- " else:\n",
- " print(f\"โ Error: {result['error']}\")\n",
- "\n",
- "# Test with the improved version\n",
- "test_question = \"What are the main architectural patterns for agent systems?\"\n",
- "\n",
- "if query_engine:\n",
- " print(\"๐งช Testing improved search with title extraction:\")\n",
- " print(\"=\" * 80)\n",
- " ask_question_improved(query_engine, test_question, show_sources=True)\n",
- "else:\n",
- " print(\"โ Query engine not available\")\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 31,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "๐ Searching for: 'What are the main types of AI agents discussed in these papers?'\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 12:49:54,494 - INFO - query_type :, vector\n",
- "2025-09-20 12:49:57,093 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 12:50:00,620 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 12:50:04,316 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Search completed in 12.60 seconds\n",
- "๐ Found 5 relevant sources\n",
- "\n",
- "๐ Response Preview: The primary types of AI agents discussed are AI Agents and Agentic AI. AI Agents are characterized as modular systems focused on narrow, task-specific automation, leveraging large language models (LLMs) and language interface models (LIMs) for tasks such as customer support, email filtering, and personalized content recommendation. In contrast, Agentic AI represents a more advanced paradigm, featuring multi-agent collaboration, dynamic task decomposition, and orchestrated autonomy, with applications in areas like research automation, robotic coordination, and medical decision-making.\n",
- "๐ Number of sources: 5\n",
- "โฑ๏ธ Search time: 12.60 seconds\n"
- ]
- }
- ],
- "source": [
- "def search_papers(query_engine, query: str, include_metadata: bool = True) -> Dict[str, any]:\n",
- " \"\"\"\n",
- " Search for relevant papers based on the query.\n",
- " \n",
- " Args:\n",
- " query_engine: The configured query engine\n",
- " query (str): Search query\n",
- " include_metadata (bool): Whether to include detailed metadata\n",
- " \n",
- " Returns:\n",
- " Dict[str, any]: Search results with response and sources\n",
- " \"\"\"\n",
- " if not query_engine:\n",
- " return {\n",
- " \"success\": False,\n",
- " \"error\": \"Query engine not initialized.\",\n",
- " \"response\": \"\",\n",
- " \"sources\": [],\n",
- " }\n",
- " \n",
- " try:\n",
- " print(f\"๐ Searching for: '{query}'\")\n",
- " start_time = time.time()\n",
- " \n",
- " # Query the RAG system\n",
- " response = query_engine.query(query)\n",
- " \n",
- " end_time = time.time()\n",
- " \n",
- " # Extract source information from retrieved nodes\n",
- " sources = []\n",
- " if hasattr(response, \"source_nodes\"):\n",
- " for node in response.source_nodes:\n",
- " source_info = {\n",
- " \"text\": (\n",
- " node.text[:500] + \"...\"\n",
- " if len(node.text) > 500\n",
- " else node.text\n",
- " ),\n",
- " \"score\": getattr(node, \"score\", 0.0),\n",
- " }\n",
- " \n",
- " # Add metadata if available and requested\n",
- " if include_metadata and hasattr(node, \"metadata\"):\n",
- " metadata = node.metadata\n",
- " source_info.update({\n",
- " \"file_name\": metadata.get(\"file_name\", \"Unknown\"),\n",
- " \"title\": metadata.get(\"title\", \"Unknown Title\"),\n",
- " \"authors\": metadata.get(\"authors\", \"Unknown Authors\"),\n",
- " \"page_count\": metadata.get(\"page_count\", 0),\n",
- " \"has_abstract\": metadata.get(\"has_abstract\", False),\n",
- " })\n",
- " \n",
- " sources.append(source_info)\n",
- " \n",
- " result = {\n",
- " \"success\": True,\n",
- " \"response\": str(response),\n",
- " \"sources\": sources,\n",
- " \"query\": query,\n",
- " \"search_time\": end_time - start_time,\n",
- " \"num_sources\": len(sources),\n",
- " }\n",
- " \n",
- " print(f\"โ Search completed in {end_time - start_time:.2f} seconds\")\n",
- " print(f\"๐ Found {len(sources)} relevant sources\")\n",
- " \n",
- " return result\n",
- " \n",
- " except Exception as e:\n",
- " print(f\"โ Error during search: {e}\")\n",
- " return {\"success\": False, \"error\": str(e), \"response\": \"\", \"sources\": []}\n",
- "\n",
- "# Test the search function with a sample query\n",
- "test_query = \"What are the main types of AI agents discussed in these papers?\"\n",
- "result = search_papers(query_engine, test_query)\n",
- "\n",
- "if result[\"success\"]:\n",
- " print(f\"\\n๐ Response Preview: {result['response']}\")\n",
- " print(f\"๐ Number of sources: {result['num_sources']}\")\n",
- " print(f\"โฑ๏ธ Search time: {result['search_time']:.2f} seconds\")\n",
- "else:\n",
- " print(f\"โ Search failed: {result['error']}\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 9. Paper Information and Metadata\n",
- "\n",
- "Let's create functions to list and get detailed information about the papers in our index.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 33,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "๐ Found 229 papers in the index:\n",
- "============================================================\n",
- "1. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "2. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "3. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "4. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "5. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "6. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "7. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "8. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "9. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "10. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "11. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "12. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "13. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "14. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "15. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "16. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "17. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "18. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "19. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "20. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "21. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "22. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "23. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "24. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "25. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "26. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "27. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "28. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "29. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "30. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "31. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "32. AI_Agents_vs_Agentic_AI\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/AI_Agents_vs_Agentic_AI.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 3.05 MB\n",
- "----------------------------------------\n",
- "33. Emerging_Agent_Architectures\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Emerging_Agent_Architectures.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 1.58 MB\n",
- "----------------------------------------\n",
- "34. Emerging_Agent_Architectures\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Emerging_Agent_Architectures.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 1.58 MB\n",
- "----------------------------------------\n",
- "35. Emerging_Agent_Architectures\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Emerging_Agent_Architectures.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 1.58 MB\n",
- "----------------------------------------\n",
- "36. Emerging_Agent_Architectures\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Emerging_Agent_Architectures.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 1.58 MB\n",
- "----------------------------------------\n",
- "37. Emerging_Agent_Architectures\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Emerging_Agent_Architectures.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 1.58 MB\n",
- "----------------------------------------\n",
- "38. Emerging_Agent_Architectures\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Emerging_Agent_Architectures.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 1.58 MB\n",
- "----------------------------------------\n",
- "39. Emerging_Agent_Architectures\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Emerging_Agent_Architectures.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 1.58 MB\n",
- "----------------------------------------\n",
- "40. Emerging_Agent_Architectures\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Emerging_Agent_Architectures.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 1.58 MB\n",
- "----------------------------------------\n",
- "41. Emerging_Agent_Architectures\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Emerging_Agent_Architectures.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 1.58 MB\n",
- "----------------------------------------\n",
- "42. Emerging_Agent_Architectures\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Emerging_Agent_Architectures.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 1.58 MB\n",
- "----------------------------------------\n",
- "43. Emerging_Agent_Architectures\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Emerging_Agent_Architectures.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 1.58 MB\n",
- "----------------------------------------\n",
- "44. Emerging_Agent_Architectures\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Emerging_Agent_Architectures.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 1.58 MB\n",
- "----------------------------------------\n",
- "45. Emerging_Agent_Architectures\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Emerging_Agent_Architectures.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 1.58 MB\n",
- "----------------------------------------\n",
- "46. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "47. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "48. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "49. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "50. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "51. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "52. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "53. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "54. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "55. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "56. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "57. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "58. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "59. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "60. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "61. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "62. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "63. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "64. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "65. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "66. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "67. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "68. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "69. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "70. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "71. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "72. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "73. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "74. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "75. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "76. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "77. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "78. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "79. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "80. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "81. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "82. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "83. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "84. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "85. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "86. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "87. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "88. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "89. LLMReasoning_to_Autonomous_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/LLMReasoning_to_Autonomous_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 16.21 MB\n",
- "----------------------------------------\n",
- "90. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "91. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "92. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "93. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "94. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "95. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "96. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "97. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "98. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "99. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "100. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "101. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "102. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "103. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "104. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "105. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "106. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "107. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "108. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "109. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "110. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "111. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "112. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "113. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "114. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "115. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "116. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "117. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "118. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "119. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "120. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "121. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "122. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "123. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "124. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "125. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "126. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "127. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "128. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "129. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "130. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "131. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "132. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "133. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "134. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "135. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "136. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "137. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "138. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "139. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "140. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "141. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "142. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "143. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "144. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "145. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "146. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "147. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "148. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "149. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "150. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "151. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "152. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "153. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "154. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "155. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "156. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "157. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "158. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "159. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "160. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "161. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "162. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "163. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "164. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "165. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "166. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "167. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "168. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "169. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "170. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "171. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "172. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "173. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "174. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "175. Rise_and_Potential_LLM_Agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/Rise_and_Potential_LLM_Agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 6.52 MB\n",
- "----------------------------------------\n",
- "176. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "177. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "178. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "179. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "180. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "181. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "182. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "183. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "184. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "185. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "186. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "187. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "188. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "189. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "190. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "191. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "192. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "193. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "194. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "195. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "196. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "197. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "198. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "199. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "200. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "201. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "202. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "203. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "204. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "205. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "206. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "207. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "208. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "209. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "210. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "211. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "212. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "213. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "214. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "215. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "216. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "217. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "218. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "219. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "220. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "221. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "222. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "223. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "224. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "225. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "226. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "227. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "228. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n",
- "229. survey_of_self_evolving_agents\n",
- " File Path: /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/../papers/agents/survey_of_self_evolving_agents.pdf\n",
- " Total Pages: Unknown\n",
- " Size: 10.56 MB\n",
- "----------------------------------------\n"
- ]
- }
- ],
- "source": [
- "def list_indexed_papers(documents: List) -> List[Dict[str, any]]:\n",
- " \"\"\"\n",
- " List all papers that have been indexed with their metadata.\n",
- " \n",
- " Args:\n",
- " documents (List): List of loaded documents\n",
- " \n",
- " Returns:\n",
- " List[Dict[str, any]]: List of paper information\n",
- " \"\"\"\n",
- " papers = []\n",
- " \n",
- " for doc in documents:\n",
- " try:\n",
- " metadata = doc.metadata\n",
- " file_path = metadata.get(\"file_path\", \"\")\n",
- " file_name = Path(file_path).stem if file_path else \"Unknown\"\n",
- " \n",
- " paper_info = {\n",
- " \"file_name\": file_name,\n",
- " \"file_path\": file_path,\n",
- " \"title\": metadata.get(\"title\", file_name),\n",
- " \"authors\": metadata.get(\"authors\", \"Unknown\"),\n",
- " \"page_count\": metadata.get(\"page_count\", 0),\n",
- " \"file_size\": metadata.get(\"file_size\", 0),\n",
- " \"file_size_mb\": round(metadata.get(\"file_size\", 0) / (1024 * 1024), 2) if metadata.get(\"file_size\") else 0,\n",
- " \"total_pages\": metadata.get(\"total_pages\", \"Unknown\"),\n",
- " }\n",
- " \n",
- " papers.append(paper_info)\n",
- " \n",
- " except Exception as e:\n",
- " print(f\"Error processing document: {e}\")\n",
- " \n",
- " return papers\n",
- "\n",
- "# List all indexed papers\n",
- "papers_list = list_indexed_papers(documents)\n",
- "\n",
- "print(f\"๐ Found {len(papers_list)} papers in the index:\")\n",
- "print(\"=\" * 60)\n",
- "\n",
- "for i, paper in enumerate(papers_list, 1):\n",
- " print(f\"{i}. {paper['file_name']}\")\n",
- " print(f\" File Path: {paper['file_path']}\")\n",
- " print(f\" Total Pages: {paper.get('total_pages', 'Unknown')}\")\n",
- " print(f\" Size: {paper['file_size_mb']} MB\")\n",
- " print(\"-\" * 40)\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 11. Advanced Query Examples\n",
- "\n",
- "Now let's test our RAG system with various types of research queries to demonstrate its capabilities.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 34,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "๐งช Running Example Queries\n",
- "============================================================\n",
- "\n",
- "1. Agent Types\n",
- "Q: What are the main types of AI agents discussed in these papers?\n",
- "--------------------------------------------------\n",
- "๐ Searching for: 'What are the main types of AI agents discussed in these papers?'\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 12:50:35,859 - INFO - query_type :, vector\n",
- "2025-09-20 12:50:38,020 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 12:50:41,718 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 12:50:45,508 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Search completed in 10.96 seconds\n",
- "๐ Found 5 relevant sources\n",
- "A: The main types of AI agents discussed are AI Agents and Agentic AI. AI Agents are modular systems designed for narrow, task-specific automation, leveraging large language models (LLMs) and large information models (LIMs) for applications like customer support, email filtering, and personalized content recommendation. Agentic AI, however, signifies a more advanced paradigm with features like multi-...\n",
- "๐ Sources: 5 | โฑ๏ธ Time: 10.96s\n",
- "\n",
- "\n",
- "2. Technical Comparison\n",
- "Q: How do LLM-based agents differ from traditional AI agents?\n",
- "--------------------------------------------------\n",
- "๐ Searching for: 'How do LLM-based agents differ from traditional AI agents?'\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 12:50:46,942 - INFO - query_type :, vector\n",
- "2025-09-20 12:50:48,038 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 12:50:51,488 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 12:50:53,684 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 12:50:55,402 - INFO - query_type :, vector\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Search completed in 8.80 seconds\n",
- "๐ Found 5 relevant sources\n",
- "A: LLM-based agents differ from traditional AI agents by incorporating large language models with modular toolkits, which facilitate autonomous decision-making and multi-step reasoning. This integration enables LLM-based agents to execute a diverse array of tasks across multiple domains, including materials science, biomedical research, and software engineering. Additionally, they support agent-to-ag...\n",
- "๐ Sources: 5 | โฑ๏ธ Time: 8.80s\n",
- "\n",
- "\n",
- "3. Challenges\n",
- "Q: What are the current challenges in developing autonomous agents?\n",
- "--------------------------------------------------\n",
- "๐ Searching for: 'What are the current challenges in developing autonomous agents?'\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 12:50:56,932 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 12:50:58,852 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 12:51:02,156 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 12:51:03,545 - INFO - query_type :, vector\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Search completed in 8.20 seconds\n",
- "๐ Found 5 relevant sources\n",
- "A: The current challenges in developing autonomous agents include addressing limitations such as a lack of causal reasoning, constraints from large language models like hallucinations and shallow reasoning, incomplete agentic properties such as autonomy and proactivity, and difficulties in long-horizon planning and recovery. Additional challenges involve inter-agent error cascades, coordination break...\n",
- "๐ Sources: 5 | โฑ๏ธ Time: 8.20s\n",
- "\n",
- "\n",
- "4. Evaluation\n",
- "Q: What evaluation methods are used for AI agents?\n",
- "--------------------------------------------------\n",
- "๐ Searching for: 'What evaluation methods are used for AI agents?'\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 12:51:04,575 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 12:51:08,270 - INFO - query_type :, vector\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Search completed in 4.67 seconds\n",
- "๐ Found 5 relevant sources\n",
- "A: AI agents are evaluated using a variety of methods that focus on different aspects of their performance. These include benchmark-based evaluations that assess task completion, reasoning quality, and generalization ability. Specific benchmarks are used for tool and API-driven agents, web navigation and browsing agents, and multi-agent collaboration. These benchmarks often involve structured tasks w...\n",
- "๐ Sources: 5 | โฑ๏ธ Time: 4.67s\n",
- "\n",
- "\n",
- "5. Architecture\n",
- "Q: Describe the common architectural patterns for agent systems.\n",
- "--------------------------------------------------\n",
- "๐ Searching for: 'Describe the common architectural patterns for agent systems.'\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 12:51:09,390 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 12:51:11,772 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 12:51:13,412 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 12:51:14,543 - INFO - query_type :, vector\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Search completed in 6.32 seconds\n",
- "๐ Found 5 relevant sources\n",
- "A: Common architectural patterns for agent systems include single-agent and multi-agent architectures. Single-agent patterns typically involve a defined persona and set of tools, with opportunities for human feedback and iterative goal achievement. Multi-agent architectures can be categorized into vertical and horizontal structures. Vertical architectures have a lead agent with other agents reporting...\n",
- "๐ Sources: 5 | โฑ๏ธ Time: 6.32s\n",
- "\n",
- "\n",
- "6. Applications\n",
- "Q: What are the practical applications of AI agents mentioned in the literature?\n",
- "--------------------------------------------------\n",
- "๐ Searching for: 'What are the practical applications of AI agents mentioned in the literature?'\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 12:51:15,441 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 12:51:17,999 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 12:51:20,486 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Search completed in 8.23 seconds\n",
- "๐ Found 5 relevant sources\n",
- "A: AI agents have practical applications in enterprise settings, including customer support, email filtering, personalized content recommendation, and autonomous scheduling. They enhance customer engagement and business intelligence by inferring user preferences and generating personalized suggestions. In analytics, AI agents enable natural-language data queries and automated report generation. Auton...\n",
- "๐ Sources: 5 | โฑ๏ธ Time: 8.23s\n",
- "\n"
- ]
- }
- ],
- "source": [
- "def run_example_queries(query_engine):\n",
- " \"\"\"\n",
- " Run a series of example queries to demonstrate RAG capabilities.\n",
- " \n",
- " Args:\n",
- " query_engine: The configured query engine\n",
- " \"\"\"\n",
- " example_queries = [\n",
- " {\n",
- " \"category\": \"Agent Types\",\n",
- " \"query\": \"What are the main types of AI agents discussed in these papers?\",\n",
- " },\n",
- " {\n",
- " \"category\": \"Technical Comparison\", \n",
- " \"query\": \"How do LLM-based agents differ from traditional AI agents?\",\n",
- " },\n",
- " {\n",
- " \"category\": \"Challenges\",\n",
- " \"query\": \"What are the current challenges in developing autonomous agents?\",\n",
- " },\n",
- " {\n",
- " \"category\": \"Evaluation\",\n",
- " \"query\": \"What evaluation methods are used for AI agents?\",\n",
- " },\n",
- " {\n",
- " \"category\": \"Architecture\",\n",
- " \"query\": \"Describe the common architectural patterns for agent systems.\",\n",
- " },\n",
- " {\n",
- " \"category\": \"Applications\",\n",
- " \"query\": \"What are the practical applications of AI agents mentioned in the literature?\",\n",
- " },\n",
- " ]\n",
- " \n",
- " print(\"๐งช Running Example Queries\")\n",
- " print(\"=\" * 60)\n",
- " \n",
- " for i, example in enumerate(example_queries, 1):\n",
- " print(f\"\\n{i}. {example['category']}\")\n",
- " print(f\"Q: {example['query']}\")\n",
- " print(\"-\" * 50)\n",
- " \n",
- " result = search_papers(query_engine, example[\"query\"])\n",
- " \n",
- " if result[\"success\"]:\n",
- " # Display truncated response\n",
- " response = result[\"response\"]\n",
- " if len(response) > 400:\n",
- " response = response[:400] + \"...\"\n",
- " \n",
- " print(f\"A: {response}\")\n",
- " print(f\"๐ Sources: {result['num_sources']} | โฑ๏ธ Time: {result['search_time']:.2f}s\")\n",
- " else:\n",
- " print(f\"โ Error: {result['error']}\")\n",
- " \n",
- " print()\n",
- "\n",
- "# Run the example queries\n",
- "if query_engine:\n",
- " run_example_queries(query_engine)\n",
- "else:\n",
- " print(\"โ Query engine not available for examples\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 12. Interactive Query Interface\n",
- "\n",
- "Let's create an interactive function that allows you to ask custom questions about the papers.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 36,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Question: What are the key ethical considerations for AI agents?\n",
- "============================================================\n",
- "๐ Searching for: 'What are the key ethical considerations for AI agents?'\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 12:51:51,888 - INFO - query_type :, vector\n",
- "2025-09-20 12:51:54,164 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Search completed in 4.09 seconds\n",
- "๐ Found 5 relevant sources\n",
- "๐ก Answer:\n",
- "Key ethical considerations for AI agents include ensuring accountability, fairness, and value alignment. These considerations are crucial due to the distributed and autonomous nature of AI systems, which can create accountability gaps when multiple agents interact to produce an outcome. Additionally, there is a need to address bias propagation and amplification, as agents trained on biased data may reinforce skewed decisions, leading to systemic inequities. Ethical governance frameworks are essential to ensure responsible deployment, defining accountability, oversight, and value alignment across autonomous agent networks.\n",
- "\n",
- "๐ Search completed in 4.09 seconds\n",
- "๐ Found 5 relevant sources\n",
- "\n",
- "๐ Source Details:\n",
- "----------------------------------------\n",
- "\n",
- "1. Score: 0.668\n",
- " Text: Lastly, to build user confidence, agents must\n",
- "prioritize Trust & Safety mechanisms through verifiable out-\n",
- "put logging, bias detection, and ethical guardrails especially\n",
- "as their autonomy increases. T...\n",
- "\n",
- "2. Score: 0.650\n",
- " Text: for errors or unintended consequences. This ambiguity\n",
- "complicates legal liability, regulatory compliance, and\n",
- "user trust, especially in domains such as healthcare,\n",
- "finance, or defense. Furthermore, bi...\n",
- "\n",
- "3. Score: 0.613\n",
- " Text: AI Agents vs. Agentic AI: A Conceptual\n",
- "Taxonomy, Applications and Challenges\n",
- "Ranjan Sapkotaโโก, Konstantinos I. Roumeliotis โ , Manoj Karkee โโก\n",
- "โCornell University, Department of Environmental and Biolo...\n",
- "\n",
- "4. Score: 0.609\n",
- " Text: These laws are hierarchical: the Second cannot override\n",
- "the First, and the Third cannot override the First or Second. Although conceived as fictional moral constraints, they have become\n",
- "influential in...\n",
- "\n",
- "5. Score: 0.608\n",
- " Text: For\n",
- "example, a fact-checking agent fed with tampered data\n",
- "could unintentionally legitimize false claims, which are\n",
- "then integrated into downstream reasoning by summa-\n",
- "rization or decision-making agent...\n"
- ]
- }
- ],
- "source": [
- "def ask_question(query_engine, question: str, show_sources: bool = True):\n",
- " \"\"\"\n",
- " Ask a custom question to the RAG system and display results.\n",
- " \n",
- " Args:\n",
- " query_engine: The configured query engine\n",
- " question (str): Your question about the papers\n",
- " show_sources (bool): Whether to display source information\n",
- " \"\"\"\n",
- " print(f\"โ Question: {question}\")\n",
- " print(\"=\" * 60)\n",
- " \n",
- " result = search_papers(query_engine, question, include_metadata=True)\n",
- " \n",
- " if result[\"success\"]:\n",
- " print(f\"๐ก Answer:\")\n",
- " print(result[\"response\"])\n",
- " print(f\"\\n๐ Search completed in {result['search_time']:.2f} seconds\")\n",
- " print(f\"๐ Found {result['num_sources']} relevant sources\")\n",
- " \n",
- " if show_sources and result[\"sources\"]:\n",
- " print(f\"\\n๐ Source Details:\")\n",
- " print(\"-\" * 40)\n",
- " for i, source in enumerate(result[\"sources\"], 1):\n",
- " print(f\"\\n{i}. Score: {source.get('score', 0):.3f}\")\n",
- " print(f\" Text: {source['text'][:200]}...\")\n",
- " \n",
- " else:\n",
- " print(f\"โ Error: {result['error']}\")\n",
- "\n",
- "# Example usage - you can modify this question\n",
- "custom_question = \"What are the key ethical considerations for AI agents?\"\n",
- "\n",
- "if query_engine:\n",
- " ask_question(query_engine, custom_question, show_sources=True)\n",
- "else:\n",
- " print(\"โ Query engine not available\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 13. System Performance and Statistics\n",
- "\n",
- "Let's create functions to analyze and display performance statistics of our RAG system.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 37,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "๐ RAG System Statistics\n",
- "==================================================\n",
- "๐ Document Statistics:\n",
- " Total Papers: 229\n",
- " Total Pages: 0\n",
- " Total Size: 1962.34 MB\n",
- " Average Pages per Paper: 0.0\n",
- "\n",
- "๐๏ธ Index Statistics:\n",
- " Index Type: Vector Store Index\n",
- " Embedding Model: text-embedding-3-small\n",
- " LLM Model: gpt-4o-mini\n",
- "\n",
- "๐พ Storage Locations:\n",
- " Papers Folder: papers/agents\n",
- " Vector Database: storage/papers_vectordb\n",
- " Index Storage: storage/papers_index\n",
- "\n",
- "๐ Paper Titles:\n",
- " 1. AI_Agents_vs_Agentic_AI\n",
- " 2. AI_Agents_vs_Agentic_AI\n",
- " 3. AI_Agents_vs_Agentic_AI\n",
- " 4. AI_Agents_vs_Agentic_AI\n",
- " 5. AI_Agents_vs_Agentic_AI\n",
- " 6. AI_Agents_vs_Agentic_AI\n",
- " 7. AI_Agents_vs_Agentic_AI\n",
- " 8. AI_Agents_vs_Agentic_AI\n",
- " 9. AI_Agents_vs_Agentic_AI\n",
- " 10. AI_Agents_vs_Agentic_AI\n",
- " 11. AI_Agents_vs_Agentic_AI\n",
- " 12. AI_Agents_vs_Agentic_AI\n",
- " 13. AI_Agents_vs_Agentic_AI\n",
- " 14. AI_Agents_vs_Agentic_AI\n",
- " 15. AI_Agents_vs_Agentic_AI\n",
- " 16. AI_Agents_vs_Agentic_AI\n",
- " 17. AI_Agents_vs_Agentic_AI\n",
- " 18. AI_Agents_vs_Agentic_AI\n",
- " 19. AI_Agents_vs_Agentic_AI\n",
- " 20. AI_Agents_vs_Agentic_AI\n",
- " 21. AI_Agents_vs_Agentic_AI\n",
- " 22. AI_Agents_vs_Agentic_AI\n",
- " 23. AI_Agents_vs_Agentic_AI\n",
- " 24. AI_Agents_vs_Agentic_AI\n",
- " 25. AI_Agents_vs_Agentic_AI\n",
- " 26. AI_Agents_vs_Agentic_AI\n",
- " 27. AI_Agents_vs_Agentic_AI\n",
- " 28. AI_Agents_vs_Agentic_AI\n",
- " 29. AI_Agents_vs_Agentic_AI\n",
- " 30. AI_Agents_vs_Agentic_AI\n",
- " 31. AI_Agents_vs_Agentic_AI\n",
- " 32. AI_Agents_vs_Agentic_AI\n",
- " 33. Emerging_Agent_Architectures\n",
- " 34. Emerging_Agent_Architectures\n",
- " 35. Emerging_Agent_Architectures\n",
- " 36. Emerging_Agent_Architectures\n",
- " 37. Emerging_Agent_Architectures\n",
- " 38. Emerging_Agent_Architectures\n",
- " 39. Emerging_Agent_Architectures\n",
- " 40. Emerging_Agent_Architectures\n",
- " 41. Emerging_Agent_Architectures\n",
- " 42. Emerging_Agent_Architectures\n",
- " 43. Emerging_Agent_Architectures\n",
- " 44. Emerging_Agent_Architectures\n",
- " 45. Emerging_Agent_Architectures\n",
- " 46. LLMReasoning_to_Autonomous_Agents\n",
- " 47. LLMReasoning_to_Autonomous_Agents\n",
- " 48. LLMReasoning_to_Autonomous_Agents\n",
- " 49. LLMReasoning_to_Autonomous_Agents\n",
- " 50. LLMReasoning_to_Autonomous_Agents\n",
- " 51. LLMReasoning_to_Autonomous_Agents\n",
- " 52. LLMReasoning_to_Autonomous_Agents\n",
- " 53. LLMReasoning_to_Autonomous_Agents\n",
- " 54. LLMReasoning_to_Autonomous_Agents\n",
- " 55. LLMReasoning_to_Autonomous_Agents\n",
- " 56. LLMReasoning_to_Autonomous_Agents\n",
- " 57. LLMReasoning_to_Autonomous_Agents\n",
- " 58. LLMReasoning_to_Autonomous_Agents\n",
- " 59. LLMReasoning_to_Autonomous_Agents\n",
- " 60. LLMReasoning_to_Autonomous_Agents\n",
- " 61. LLMReasoning_to_Autonomous_Agents\n",
- " 62. LLMReasoning_to_Autonomous_Agents\n",
- " 63. LLMReasoning_to_Autonomous_Agents\n",
- " 64. LLMReasoning_to_Autonomous_Agents\n",
- " 65. LLMReasoning_to_Autonomous_Agents\n",
- " 66. LLMReasoning_to_Autonomous_Agents\n",
- " 67. LLMReasoning_to_Autonomous_Agents\n",
- " 68. LLMReasoning_to_Autonomous_Agents\n",
- " 69. LLMReasoning_to_Autonomous_Agents\n",
- " 70. LLMReasoning_to_Autonomous_Agents\n",
- " 71. LLMReasoning_to_Autonomous_Agents\n",
- " 72. LLMReasoning_to_Autonomous_Agents\n",
- " 73. LLMReasoning_to_Autonomous_Agents\n",
- " 74. LLMReasoning_to_Autonomous_Agents\n",
- " 75. LLMReasoning_to_Autonomous_Agents\n",
- " 76. LLMReasoning_to_Autonomous_Agents\n",
- " 77. LLMReasoning_to_Autonomous_Agents\n",
- " 78. LLMReasoning_to_Autonomous_Agents\n",
- " 79. LLMReasoning_to_Autonomous_Agents\n",
- " 80. LLMReasoning_to_Autonomous_Agents\n",
- " 81. LLMReasoning_to_Autonomous_Agents\n",
- " 82. LLMReasoning_to_Autonomous_Agents\n",
- " 83. LLMReasoning_to_Autonomous_Agents\n",
- " 84. LLMReasoning_to_Autonomous_Agents\n",
- " 85. LLMReasoning_to_Autonomous_Agents\n",
- " 86. LLMReasoning_to_Autonomous_Agents\n",
- " 87. LLMReasoning_to_Autonomous_Agents\n",
- " 88. LLMReasoning_to_Autonomous_Agents\n",
- " 89. LLMReasoning_to_Autonomous_Agents\n",
- " 90. Rise_and_Potential_LLM_Agents\n",
- " 91. Rise_and_Potential_LLM_Agents\n",
- " 92. Rise_and_Potential_LLM_Agents\n",
- " 93. Rise_and_Potential_LLM_Agents\n",
- " 94. Rise_and_Potential_LLM_Agents\n",
- " 95. Rise_and_Potential_LLM_Agents\n",
- " 96. Rise_and_Potential_LLM_Agents\n",
- " 97. Rise_and_Potential_LLM_Agents\n",
- " 98. Rise_and_Potential_LLM_Agents\n",
- " 99. Rise_and_Potential_LLM_Agents\n",
- " 100. Rise_and_Potential_LLM_Agents\n",
- " 101. Rise_and_Potential_LLM_Agents\n",
- " 102. Rise_and_Potential_LLM_Agents\n",
- " 103. Rise_and_Potential_LLM_Agents\n",
- " 104. Rise_and_Potential_LLM_Agents\n",
- " 105. Rise_and_Potential_LLM_Agents\n",
- " 106. Rise_and_Potential_LLM_Agents\n",
- " 107. Rise_and_Potential_LLM_Agents\n",
- " 108. Rise_and_Potential_LLM_Agents\n",
- " 109. Rise_and_Potential_LLM_Agents\n",
- " 110. Rise_and_Potential_LLM_Agents\n",
- " 111. Rise_and_Potential_LLM_Agents\n",
- " 112. Rise_and_Potential_LLM_Agents\n",
- " 113. Rise_and_Potential_LLM_Agents\n",
- " 114. Rise_and_Potential_LLM_Agents\n",
- " 115. Rise_and_Potential_LLM_Agents\n",
- " 116. Rise_and_Potential_LLM_Agents\n",
- " 117. Rise_and_Potential_LLM_Agents\n",
- " 118. Rise_and_Potential_LLM_Agents\n",
- " 119. Rise_and_Potential_LLM_Agents\n",
- " 120. Rise_and_Potential_LLM_Agents\n",
- " 121. Rise_and_Potential_LLM_Agents\n",
- " 122. Rise_and_Potential_LLM_Agents\n",
- " 123. Rise_and_Potential_LLM_Agents\n",
- " 124. Rise_and_Potential_LLM_Agents\n",
- " 125. Rise_and_Potential_LLM_Agents\n",
- " 126. Rise_and_Potential_LLM_Agents\n",
- " 127. Rise_and_Potential_LLM_Agents\n",
- " 128. Rise_and_Potential_LLM_Agents\n",
- " 129. Rise_and_Potential_LLM_Agents\n",
- " 130. Rise_and_Potential_LLM_Agents\n",
- " 131. Rise_and_Potential_LLM_Agents\n",
- " 132. Rise_and_Potential_LLM_Agents\n",
- " 133. Rise_and_Potential_LLM_Agents\n",
- " 134. Rise_and_Potential_LLM_Agents\n",
- " 135. Rise_and_Potential_LLM_Agents\n",
- " 136. Rise_and_Potential_LLM_Agents\n",
- " 137. Rise_and_Potential_LLM_Agents\n",
- " 138. Rise_and_Potential_LLM_Agents\n",
- " 139. Rise_and_Potential_LLM_Agents\n",
- " 140. Rise_and_Potential_LLM_Agents\n",
- " 141. Rise_and_Potential_LLM_Agents\n",
- " 142. Rise_and_Potential_LLM_Agents\n",
- " 143. Rise_and_Potential_LLM_Agents\n",
- " 144. Rise_and_Potential_LLM_Agents\n",
- " 145. Rise_and_Potential_LLM_Agents\n",
- " 146. Rise_and_Potential_LLM_Agents\n",
- " 147. Rise_and_Potential_LLM_Agents\n",
- " 148. Rise_and_Potential_LLM_Agents\n",
- " 149. Rise_and_Potential_LLM_Agents\n",
- " 150. Rise_and_Potential_LLM_Agents\n",
- " 151. Rise_and_Potential_LLM_Agents\n",
- " 152. Rise_and_Potential_LLM_Agents\n",
- " 153. Rise_and_Potential_LLM_Agents\n",
- " 154. Rise_and_Potential_LLM_Agents\n",
- " 155. Rise_and_Potential_LLM_Agents\n",
- " 156. Rise_and_Potential_LLM_Agents\n",
- " 157. Rise_and_Potential_LLM_Agents\n",
- " 158. Rise_and_Potential_LLM_Agents\n",
- " 159. Rise_and_Potential_LLM_Agents\n",
- " 160. Rise_and_Potential_LLM_Agents\n",
- " 161. Rise_and_Potential_LLM_Agents\n",
- " 162. Rise_and_Potential_LLM_Agents\n",
- " 163. Rise_and_Potential_LLM_Agents\n",
- " 164. Rise_and_Potential_LLM_Agents\n",
- " 165. Rise_and_Potential_LLM_Agents\n",
- " 166. Rise_and_Potential_LLM_Agents\n",
- " 167. Rise_and_Potential_LLM_Agents\n",
- " 168. Rise_and_Potential_LLM_Agents\n",
- " 169. Rise_and_Potential_LLM_Agents\n",
- " 170. Rise_and_Potential_LLM_Agents\n",
- " 171. Rise_and_Potential_LLM_Agents\n",
- " 172. Rise_and_Potential_LLM_Agents\n",
- " 173. Rise_and_Potential_LLM_Agents\n",
- " 174. Rise_and_Potential_LLM_Agents\n",
- " 175. Rise_and_Potential_LLM_Agents\n",
- " 176. survey_of_self_evolving_agents\n",
- " 177. survey_of_self_evolving_agents\n",
- " 178. survey_of_self_evolving_agents\n",
- " 179. survey_of_self_evolving_agents\n",
- " 180. survey_of_self_evolving_agents\n",
- " 181. survey_of_self_evolving_agents\n",
- " 182. survey_of_self_evolving_agents\n",
- " 183. survey_of_self_evolving_agents\n",
- " 184. survey_of_self_evolving_agents\n",
- " 185. survey_of_self_evolving_agents\n",
- " 186. survey_of_self_evolving_agents\n",
- " 187. survey_of_self_evolving_agents\n",
- " 188. survey_of_self_evolving_agents\n",
- " 189. survey_of_self_evolving_agents\n",
- " 190. survey_of_self_evolving_agents\n",
- " 191. survey_of_self_evolving_agents\n",
- " 192. survey_of_self_evolving_agents\n",
- " 193. survey_of_self_evolving_agents\n",
- " 194. survey_of_self_evolving_agents\n",
- " 195. survey_of_self_evolving_agents\n",
- " 196. survey_of_self_evolving_agents\n",
- " 197. survey_of_self_evolving_agents\n",
- " 198. survey_of_self_evolving_agents\n",
- " 199. survey_of_self_evolving_agents\n",
- " 200. survey_of_self_evolving_agents\n",
- " 201. survey_of_self_evolving_agents\n",
- " 202. survey_of_self_evolving_agents\n",
- " 203. survey_of_self_evolving_agents\n",
- " 204. survey_of_self_evolving_agents\n",
- " 205. survey_of_self_evolving_agents\n",
- " 206. survey_of_self_evolving_agents\n",
- " 207. survey_of_self_evolving_agents\n",
- " 208. survey_of_self_evolving_agents\n",
- " 209. survey_of_self_evolving_agents\n",
- " 210. survey_of_self_evolving_agents\n",
- " 211. survey_of_self_evolving_agents\n",
- " 212. survey_of_self_evolving_agents\n",
- " 213. survey_of_self_evolving_agents\n",
- " 214. survey_of_self_evolving_agents\n",
- " 215. survey_of_self_evolving_agents\n",
- " 216. survey_of_self_evolving_agents\n",
- " 217. survey_of_self_evolving_agents\n",
- " 218. survey_of_self_evolving_agents\n",
- " 219. survey_of_self_evolving_agents\n",
- " 220. survey_of_self_evolving_agents\n",
- " 221. survey_of_self_evolving_agents\n",
- " 222. survey_of_self_evolving_agents\n",
- " 223. survey_of_self_evolving_agents\n",
- " 224. survey_of_self_evolving_agents\n",
- " 225. survey_of_self_evolving_agents\n",
- " 226. survey_of_self_evolving_agents\n",
- " 227. survey_of_self_evolving_agents\n",
- " 228. survey_of_self_evolving_agents\n",
- " 229. survey_of_self_evolving_agents\n",
- "\n",
- "โ
RAG System Analysis Complete!\n"
- ]
- }
- ],
- "source": [
- "def display_system_stats(papers_list, vector_store, index):\n",
- " \"\"\"\n",
- " Display comprehensive statistics about the RAG system.\n",
- " \n",
- " Args:\n",
- " papers_list: List of indexed papers\n",
- " vector_store: The vector store instance\n",
- " index: The vector index\n",
- " \"\"\"\n",
- " print(\"๐ RAG System Statistics\")\n",
- " print(\"=\" * 50)\n",
- " \n",
- " # Paper statistics\n",
- " total_papers = len(papers_list)\n",
- " total_pages = sum(paper.get('page_count', 0) for paper in papers_list)\n",
- " total_size_mb = sum(paper.get('file_size_mb', 0) for paper in papers_list)\n",
- " \n",
- " print(f\"๐ Document Statistics:\")\n",
- " print(f\" Total Papers: {total_papers}\")\n",
- " print(f\" Total Pages: {total_pages}\")\n",
- " print(f\" Total Size: {total_size_mb:.2f} MB\")\n",
- " print(f\" Average Pages per Paper: {total_pages/total_papers:.1f}\" if total_papers > 0 else \" Average Pages: N/A\")\n",
- " \n",
- " # Index statistics\n",
- " if index:\n",
- " print(f\"\\n๐๏ธ Index Statistics:\")\n",
- " print(f\" Index Type: Vector Store Index\")\n",
- " print(f\" Embedding Model: {get_config('api.openai.embedding_model', 'text-embedding-3-small')}\")\n",
- " print(f\" LLM Model: {get_config('api.openai.model', 'gpt-4o-mini')}\")\n",
- " \n",
- " # Storage paths\n",
- " print(f\"\\n๐พ Storage Locations:\")\n",
- " print(f\" Papers Folder: papers/agents\")\n",
- " print(f\" Vector Database: storage/papers_vectordb\")\n",
- " print(f\" Index Storage: storage/papers_index\")\n",
- " \n",
- " # Recent papers by modification time\n",
- " if papers_list:\n",
- " print(f\"\\n๐ Paper Titles:\")\n",
- " for i, paper in enumerate(papers_list, 1):\n",
- " title = paper['title']\n",
- " if len(title) > 50:\n",
- " title = title[:47] + \"...\"\n",
- " print(f\" {i}. {title}\")\n",
- "\n",
- "# Display system statistics\n",
- "display_system_stats(papers_list, vector_store, index)\n",
- "print(\"\\nโ
RAG System Analysis Complete!\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Conclusion\n",
- "\n",
- "๐ **Congratulations!** You have successfully built a complete RAG (Retrieval-Augmented Generation) system for academic papers using LlamaIndex.\n",
- "\n",
- "### What we accomplished:\n",
- "\n",
- "1. **Environment Setup**: Configured API keys and dependencies\n",
- "2. **Configuration Management**: Loaded system settings from YAML files\n",
- "3. **LlamaIndex Configuration**: Set up embeddings, LLM, and text processing\n",
- "4. **Vector Store**: Created a LanceDB vector database for storing embeddings\n",
- "5. **Document Processing**: Loaded and processed PDF academic papers\n",
- "6. **Vector Indexing**: Created searchable vector embeddings of documents\n",
- "7. **Query Engine**: Set up retrieval and response generation\n",
- "8. **Search Functions**: Implemented semantic search with metadata\n",
- "9. **Paper Analysis**: Created functions for listing and summarizing papers\n",
- "10. **Interactive Queries**: Built an interface for asking custom questions\n",
- "11. **Performance Analytics**: Added system statistics and monitoring\n",
- "\n",
- "### Key Features:\n",
- "\n",
- "- **Semantic Search**: Find relevant content using natural language queries\n",
- "- **Source Attribution**: Get detailed citations and references for answers\n",
- "- **Metadata Integration**: Access paper titles, authors, and other metadata\n",
- "- **Performance Monitoring**: Track search times and system statistics\n",
- "- **Flexible Configuration**: Easy to modify models, chunk sizes, and parameters\n",
- "\n",
- "### Next Steps:\n",
- "\n",
- "1. **Experiment** with different queries to explore your document collection\n",
- "2. **Modify** the `custom_question` variable to ask your own questions\n",
- "3. **Adjust** parameters like `chunk_size`, `similarity_top_k` for different results\n",
- "4. **Add** more papers to the `papers/agents` folder and rebuild the index\n",
- "5. **Enhance** the system with additional features like filtering or ranking\n",
- "\n",
- "### Usage Tips:\n",
- "\n",
- "- Use specific, focused questions for better results\n",
- "- Try different phrasings of the same question\n",
- "- Check the source information to understand where answers come from\n",
- "- Experiment with the `similarity_top_k` parameter to get more or fewer sources\n",
- "\n",
- "Happy researching! ๐ฌ๐\n"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "accelerator",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.13"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/Girish_Basavaraj_Hiremath/session_2/llamaindex_rag/02_multimodal_rag_system.ipynb b/Girish_Basavaraj_Hiremath/session_2/llamaindex_rag/02_multimodal_rag_system.ipynb
deleted file mode 100644
index 866ac95..0000000
--- a/Girish_Basavaraj_Hiremath/session_2/llamaindex_rag/02_multimodal_rag_system.ipynb
+++ /dev/null
@@ -1,1360 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Multimodal RAG System Tutorial\n",
- "\n",
- "This notebook extends our basic RAG system to handle multiple data types including PDFs, CSV files, JSON, Markdown, HTML, images, and audio files. We'll demonstrate the advanced capabilities of LlamaIndex's `SimpleDirectoryReader` for multimodal data processing.\n",
- "\n",
- "## What's New in This Tutorial\n",
- "\n",
- "Building upon our previous RAG system, we now add:\n",
- "- **Multimodal Document Loading**: CSV, JSON, Markdown, HTML, Images, Audio\n",
- "- **Advanced SimpleDirectoryReader Features**: File filtering, metadata extraction, custom processors\n",
- "- **Cross-Modal Queries**: Search across different data types simultaneously\n",
- "- **Structured Data Integration**: Combine tabular data with unstructured text\n",
- "- **Visual Content Processing**: Extract information from images and charts\n",
- "\n",
- "## Supported File Types (Per LlamaIndex Documentation)\n",
- "\n",
- "According to the [SimpleDirectoryReader documentation](https://developers.llamaindex.ai/python/framework/module_guides/loading/simpledirectoryreader/), the following formats are automatically supported:\n",
- "\n",
- "- **.csv** - comma-separated values\n",
- "- **.docx** - Microsoft Word \n",
- "- **.epub** - EPUB ebook format\n",
- "- **.hwp** - Hangul Word Processor\n",
- "- **.ipynb** - Jupyter Notebook\n",
- "- **.jpeg, .jpg** - JPEG image\n",
- "- **.mbox** - MBOX email archive\n",
- "- **.md** - Markdown\n",
- "- **.mp3, .mp4** - audio and video\n",
- "- **.pdf** - Portable Document Format\n",
- "- **.png** - Portable Network Graphics\n",
- "- **.ppt, .pptm, .pptx** - Microsoft PowerPoint\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 1. Environment Setup and Configuration\n",
- "\n",
- "First, let's set up our environment with hardcoded configurations. We'll use OpenRouter for the LLM and local embeddings for cost-effective processing.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# !pip install -r \"../requirements.txt\""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Environment variables loaded successfully\n",
- "โ LLM Model: gpt-5-mini\n",
- "โ Embedding Model: local:BAAI/bge-small-en-v1.5\n",
- "Environment setup complete!\n"
- ]
- }
- ],
- "source": [
- "# Environment setup with hardcoded configurations\n",
- "import os\n",
- "import time\n",
- "from pathlib import Path\n",
- "from typing import Dict, List, Optional, Tuple\n",
- "import pandas as pd\n",
- "import json\n",
- "\n",
- "from dotenv import load_dotenv\n",
- "\n",
- "# Hardcoded configuration\n",
- "CONFIG = {\n",
- " \"llm_model\": \"gpt-5-mini\",\n",
- " \"embedding_model\": \"local:BAAI/bge-small-en-v1.5\",\n",
- " \"chunk_size\": 1024,\n",
- " \"chunk_overlap\": 100,\n",
- " \"similarity_top_k\": 5,\n",
- " \"data_path\": \"../data\",\n",
- " \"vector_db_path\": \"storage/multimodal_vectordb\",\n",
- " \"index_storage_path\": \"storage/multimodal_index\"\n",
- "}\n",
- "\n",
- "def setup_environment():\n",
- " \"\"\"\n",
- " Setup environment variables and basic configuration.\n",
- " \n",
- " Returns:\n",
- " bool: Success status\n",
- " \"\"\"\n",
- " # Load environment variables from .env file\n",
- " load_dotenv()\n",
- " \n",
- " # Disable tokenizer warning\n",
- " os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n",
- " \n",
- " # Check for required API key\n",
- " api_key = os.getenv(\"OPENROUTER_API_KEY\")\n",
- " if not api_key:\n",
- " print(\"โ ๏ธ OPENROUTER_API_KEY not found in environment variables\")\n",
- " print(\"Please add your OpenRouter API key to a .env file\")\n",
- " return False\n",
- " \n",
- " print(\"โ Environment variables loaded successfully\")\n",
- " print(f\"โ LLM Model: {CONFIG['llm_model']}\")\n",
- " print(f\"โ Embedding Model: {CONFIG['embedding_model']}\")\n",
- " return True\n",
- "\n",
- "# Run the setup\n",
- "success = setup_environment()\n",
- "if success:\n",
- " print(\"Environment setup complete!\")\n",
- "else:\n",
- " print(\"Environment setup failed!\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 2. LlamaIndex Configuration for Multimodal Data\n",
- "\n",
- "Let's configure LlamaIndex with our hardcoded settings for OpenRouter LLM and local embeddings.\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## ๐ Multimodal vs. Unimodal Vector Index Creation\n",
- "\n",
- "### Understanding the Key Difference\n",
- "\n",
- "While **multimodal** and **unimodal** RAG systems use the same underlying `VectorStoreIndex.from_documents()` method, there's a **critical difference** in how existing indexes are loaded that affects system behavior and reliability.\n",
- "\n",
- "### ๐ Index Creation Comparison\n",
- "\n",
- "| Aspect | Unimodal (Academic Papers) | Multimodal (This Notebook) | Impact |\n",
- "|--------|----------------------------|----------------------------|---------|\n",
- "| **Document Types** | Single type (PDF papers) | Multiple types (PDF, CSV, HTML, Images, Audio) | Different processing pipelines |\n",
- "| **Index Creation** | `VectorStoreIndex.from_documents()` | `VectorStoreIndex.from_documents()` | **Identical** |\n",
- "| **Storage Context** | โ
Full StorageContext persistence | โ
Full StorageContext persistence | **Identical** |\n",
- "| **Index Loading** | `load_index_from_storage()` | `VectorStoreIndex.from_vector_store()` | **โ ๏ธ Different!** |\n",
- "| **Metadata Complexity** | Single file type metadata | Rich cross-modal metadata | More complex relationships |\n",
- "\n",
- "### ๐ The Critical Loading Difference\n",
- "\n",
- "**Unimodal Loading (Academic Papers):**\n",
- "```python\n",
- "# ROBUST: Complete index reconstruction\n",
- "storage_context = StorageContext.from_defaults(\n",
- " persist_dir=str(index_path), \n",
- " vector_store=vector_store\n",
- ")\n",
- "index = load_index_from_storage(storage_context)\n",
- "# โ
Perfect restoration with all metadata and relationships\n",
- "```\n",
- "\n",
- "**Multimodal Loading (This Notebook):**\n",
- "```python\n",
- "# BASIC: Vector-only reconstruction\n",
- "storage_context = StorageContext.from_defaults(\n",
- " persist_dir=str(index_path), \n",
- " vector_store=vector_store\n",
- ")\n",
- "index = VectorStoreIndex.from_vector_store(\n",
- " vector_store=vector_store,\n",
- " storage_context=storage_context\n",
- ")\n",
- "# โ ๏ธ May lose some complex relationships between file types\n",
- "```\n",
- "\n",
- "### ๐ฏ Why This Difference Matters\n",
- "\n",
- "**For Unimodal Systems:**\n",
- "- Documents are homogeneous (all PDFs)\n",
- "- `load_index_from_storage()` ensures perfect reconstruction\n",
- "- Critical for academic reproducibility\n",
- "\n",
- "**For Multimodal Systems:**\n",
- "- Documents are heterogeneous (PDFs, images, audio, CSV)\n",
- "- `from_vector_store()` focuses on vector similarity\n",
- "- Cross-modal relationships handled differently\n",
- "- May prioritize performance over perfect metadata preservation\n",
- "\n",
- "### ๐ Practical Implications\n",
- "\n",
- "| Scenario | Unimodal Advantage | Multimodal Trade-off |\n",
- "|----------|-------------------|---------------------|\n",
- "| **Research Reproducibility** | ๐ฏ Identical results every time | โ ๏ธ Minor variations possible |\n",
- "| **Cross-Modal Queries** | โ Not applicable | โ
Query across file types |\n",
- "| **System Startup** | โก Fastest (complete restoration) | ๐ Fast (vector-based loading) |\n",
- "| **Metadata Fidelity** | ๐ 100% preserved | ๐ Core metadata preserved |\n",
- "| **File Type Diversity** | ๐ Single type (PDFs) | ๐ Multiple types supported |\n",
- "\n",
- "### ๐ ๏ธ When to Use Each Approach\n",
- "\n",
- "**Choose Unimodal (`load_index_from_storage`) When:**\n",
- "- Working with homogeneous document types\n",
- "- Perfect reproducibility is critical\n",
- "- Academic research or compliance requirements\n",
- "- Complex document relationships matter\n",
- "\n",
- "**Choose Multimodal (`from_vector_store`) When:**\n",
- "- Processing diverse file types simultaneously\n",
- "- Cross-modal search is the priority\n",
- "- Performance over perfect metadata preservation\n",
- "- Building versatile content search systems\n",
- "\n",
- "### ๐จ The Multimodal Advantage\n",
- "\n",
- "Despite the loading difference, multimodal indexing provides unique capabilities:\n",
- "\n",
- "1. **๐ Cross-Modal Search**: Find information across PDFs, images, and data files\n",
- "2. **๐ Rich Content Types**: Handle structured (CSV) and unstructured (text) data together \n",
- "3. **๐ต Audio Integration**: Include transcribed audio content in searches\n",
- "4. **๐ผ๏ธ Visual Content**: Extract information from charts and diagrams\n",
- "5. **๐ Unified Knowledge Base**: Single search across all organizational content\n",
- "\n",
- "### ๐ก Best Practice Recommendation\n",
- "\n",
- "For production multimodal systems, consider implementing **hybrid loading**:\n",
- "\n",
- "```python\n",
- "# Try robust loading first, fallback to vector-only\n",
- "try:\n",
- " index = load_index_from_storage(storage_context) # Full restoration\n",
- " print(\"โ Complete index restoration\")\n",
- "except:\n",
- " index = VectorStoreIndex.from_vector_store(vector_store) # Vector fallback\n",
- " print(\"โ Vector-based index loading\")\n",
- "```\n",
- "\n",
- "This gives you the reliability of unimodal loading with the flexibility of multimodal processing.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ LLM configured: gpt-5-mini\n",
- "โ Embedding model configured: local:BAAI/bge-small-en-v1.5\n",
- "โ Text chunking configured: 1024 chars with 100 overlap\n",
- "โ LlamaIndex settings configured for multimodal processing\n"
- ]
- }
- ],
- "source": [
- "# LlamaIndex configuration with hardcoded settings\n",
- "from llama_index.core import Settings\n",
- "from llama_index.llms.openrouter import OpenRouter\n",
- "from llama_index.core.embeddings import resolve_embed_model\n",
- "from llama_index.core.node_parser import SentenceSplitter\n",
- "\n",
- "def configure_llamaindex_settings():\n",
- " \"\"\"Configure LlamaIndex global settings using hardcoded configuration.\"\"\"\n",
- " \n",
- " # Set up LLM with OpenRouter using hardcoded model\n",
- " Settings.llm = OpenRouter(\n",
- " api_key=os.getenv(\"OPENROUTER_API_KEY\"),\n",
- " model=CONFIG[\"llm_model\"]\n",
- " )\n",
- " print(f\"โ LLM configured: {CONFIG['llm_model']}\")\n",
- "\n",
- " # Set up local embedding model (downloads locally first time, then cached)\n",
- " Settings.embed_model = resolve_embed_model(CONFIG[\"embedding_model\"])\n",
- " print(f\"โ Embedding model configured: {CONFIG['embedding_model']}\")\n",
- "\n",
- " # Set up node parser for chunking with hardcoded settings\n",
- " Settings.node_parser = SentenceSplitter(\n",
- " chunk_size=CONFIG[\"chunk_size\"], \n",
- " chunk_overlap=CONFIG[\"chunk_overlap\"]\n",
- " )\n",
- " print(f\"โ Text chunking configured: {CONFIG['chunk_size']} chars with {CONFIG['chunk_overlap']} overlap\")\n",
- "\n",
- "# Configure the settings\n",
- "configure_llamaindex_settings()\n",
- "print(\"โ LlamaIndex settings configured for multimodal processing\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 3. Exploring Our Multimodal Dataset\n",
- "\n",
- "Let's examine the different types of files we have available for processing. This will show the diversity of data types that SimpleDirectoryReader can handle.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "๐๏ธ Dataset Overview\n",
- "==================================================\n",
- "Total files found: 21\n",
- "\n",
- "๐ File Types Distribution:\n",
- " .csv: 4 files (0.00 MB)\n",
- " - italian_recipes.csv (0.0 MB)\n",
- " - agent_performance_benchmark.csv (0.0 MB)\n",
- " - agent_evaluation_metrics.csv (0.0 MB)\n",
- " ... and 1 more\n",
- "\n",
- " .html: 2 files (0.00 MB)\n",
- " - fitness_tracker.html (0.0 MB)\n",
- " - agent_tutorial.html (0.0 MB)\n",
- "\n",
- " .md: 4 files (0.00 MB)\n",
- " - recipe_instructions.md (0.0 MB)\n",
- " - agent_framework_comparison.md (0.0 MB)\n",
- " - market_analysis.md (0.0 MB)\n",
- " ... and 1 more\n",
- "\n",
- " .mp3: 3 files (2.95 MB)\n",
- " - rags.mp3 (0.81 MB)\n",
- " - ai_agents.mp3 (1.54 MB)\n",
- " - in_the_end.mp3 (0.6 MB)\n",
- "\n",
- " .pdf: 2 files (1.92 MB)\n",
- " - AI_Agent_Frameworks.pdf (0.34 MB)\n",
- " - Emerging_Agent_Architectures.pdf (1.58 MB)\n",
- "\n",
- " .png: 6 files (0.55 MB)\n",
- " - recipe_popularity.png (0.04 MB)\n",
- " - agent_types_comparison.png (0.1 MB)\n",
- " - agent_performance_comparison.png (0.17 MB)\n",
- " ... and 3 more\n",
- "\n",
- "โ Found 21 files across 6 different file types\n"
- ]
- }
- ],
- "source": [
- "def explore_dataset(data_path: str = None):\n",
- " \"\"\"\n",
- " Explore and categorize the files in our dataset by type.\n",
- " \n",
- " Args:\n",
- " data_path (str): Path to the data directory\n",
- " \"\"\"\n",
- " if data_path is None:\n",
- " data_path = CONFIG[\"data_path\"]\n",
- " \n",
- " data_dir = Path(data_path)\n",
- " if not data_dir.exists():\n",
- " print(f\"Data directory not found: {data_dir}\")\n",
- " return\n",
- " \n",
- " # Categorize files by type\n",
- " file_types = {}\n",
- " all_files = []\n",
- " \n",
- " # Walk through all files recursively\n",
- " for file_path in data_dir.rglob(\"*\"):\n",
- " if file_path.is_file():\n",
- " suffix = file_path.suffix.lower()\n",
- " file_size = file_path.stat().st_size\n",
- " \n",
- " if suffix not in file_types:\n",
- " file_types[suffix] = []\n",
- " \n",
- " file_info = {\n",
- " \"path\": str(file_path),\n",
- " \"name\": file_path.name,\n",
- " \"size_mb\": round(file_size / (1024 * 1024), 2),\n",
- " \"size_bytes\": file_size\n",
- " }\n",
- " \n",
- " file_types[suffix].append(file_info)\n",
- " all_files.append(file_info)\n",
- " \n",
- " # Display summary\n",
- " print(\"๐๏ธ Dataset Overview\")\n",
- " print(\"=\" * 50)\n",
- " print(f\"Total files found: {len(all_files)}\")\n",
- " \n",
- " print(f\"\\n๐ File Types Distribution:\")\n",
- " for file_type, files in sorted(file_types.items()):\n",
- " if file_type: # Skip files without extension\n",
- " total_size = sum(f[\"size_mb\"] for f in files)\n",
- " print(f\" {file_type}: {len(files)} files ({total_size:.2f} MB)\")\n",
- " \n",
- " # Show file details\n",
- " for file_info in files[:3]: # Show first 3 files of each type\n",
- " print(f\" - {file_info['name']} ({file_info['size_mb']} MB)\")\n",
- " if len(files) > 3:\n",
- " print(f\" ... and {len(files) - 3} more\")\n",
- " \n",
- " print()\n",
- " \n",
- " return file_types, all_files\n",
- "\n",
- "# Explore our dataset\n",
- "file_types, all_files = explore_dataset()\n",
- "print(f\"โ Found {len(all_files)} files across {len(file_types)} different file types\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 4. Basic Multimodal Document Loading\n",
- "\n",
- "Now let's use SimpleDirectoryReader to load all files from our data directory. This demonstrates the core multimodal capability.\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### ๐ Index Creation Implementation Note\n",
- "\n",
- "The following implementation uses **multimodal-optimized loading** - notice the difference from the academic papers notebook:\n",
- "\n",
- "#### Key Implementation Differences:\n",
- "\n",
- "1. **Index Loading Method**: Uses `VectorStoreIndex.from_vector_store()` instead of `load_index_from_storage()`\n",
- "2. **Reasoning**: Optimized for cross-modal search performance over perfect metadata preservation \n",
- "3. **Trade-off**: Slightly less metadata fidelity but better handling of diverse file types\n",
- "4. **Benefit**: More flexible loading for heterogeneous document collections\n",
- "\n",
- "This approach prioritizes the core multimodal capability while maintaining good performance and reliability.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "๐ Loading multimodal documents from: ../data\n",
- "๐ Processing files...\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "/Users/ishandutta/miniconda3/envs/accelerator/lib/python3.11/site-packages/whisper/transcribe.py:132: UserWarning: FP16 is not supported on CPU; using FP32 instead\n",
- " warnings.warn(\"FP16 is not supported on CPU; using FP32 instead\")\n",
- "/Users/ishandutta/miniconda3/envs/accelerator/lib/python3.11/site-packages/whisper/transcribe.py:132: UserWarning: FP16 is not supported on CPU; using FP32 instead\n",
- " warnings.warn(\"FP16 is not supported on CPU; using FP32 instead\")\n",
- "/Users/ishandutta/miniconda3/envs/accelerator/lib/python3.11/site-packages/whisper/transcribe.py:132: UserWarning: FP16 is not supported on CPU; using FP32 instead\n",
- " warnings.warn(\"FP16 is not supported on CPU; using FP32 instead\")\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ
Successfully loaded 42 documents in 14.85 seconds\n",
- "\n",
- "๐ Documents by MIME type:\n",
- " application/pdf: 23 documents\n",
- " audio/mpeg: 3 documents\n",
- " image/png: 6 documents\n",
- " text/csv: 4 documents\n",
- " text/html: 2 documents\n",
- " unknown: 4 documents\n",
- "\n",
- "๐ Sample Document Analysis:\n",
- "File: AI_Agent_Frameworks.pdf\n",
- "Type: application/pdf\n",
- "Size: 360523 bytes\n",
- "Text preview: A Comprehensive Survey of AI Agent Frameworks\n",
- "and Their Applications in Financial Services\n",
- "Satyadhar Joshi\n",
- "Independent\n",
- "Alumnus, International MBA, Bar-Ilan University, Israel\n",
- "satyadhar.joshi@gmail.com...\n",
- "Metadata keys: ['page_label', 'file_name', 'file_path', 'file_type', 'file_size', 'creation_date', 'last_modified_date']\n"
- ]
- }
- ],
- "source": [
- "from llama_index.core import SimpleDirectoryReader\n",
- "\n",
- "def load_multimodal_documents(data_path: str = None, recursive: bool = True):\n",
- " \"\"\"\n",
- " Load documents from multiple file types using SimpleDirectoryReader.\n",
- " \n",
- " Args:\n",
- " data_path (str): Path to directory containing multimodal data\n",
- " recursive (bool): Whether to search subdirectories\n",
- " \n",
- " Returns:\n",
- " List of Document objects\n",
- " \"\"\"\n",
- " if data_path is None:\n",
- " data_path = CONFIG[\"data_path\"]\n",
- " \n",
- " print(f\"๐ Loading multimodal documents from: {data_path}\")\n",
- " \n",
- " # Create SimpleDirectoryReader with recursive search\n",
- " reader = SimpleDirectoryReader(\n",
- " input_dir=data_path,\n",
- " recursive=recursive,\n",
- " # Let SimpleDirectoryReader handle all supported file types automatically\n",
- " )\n",
- " \n",
- " print(\"๐ Processing files...\")\n",
- " start_time = time.time()\n",
- " \n",
- " # Load all documents\n",
- " documents = reader.load_data()\n",
- " \n",
- " end_time = time.time()\n",
- " \n",
- " print(f\"โ
Successfully loaded {len(documents)} documents in {end_time - start_time:.2f} seconds\")\n",
- " \n",
- " # Analyze loaded documents by file type\n",
- " doc_types = {}\n",
- " for doc in documents:\n",
- " file_type = doc.metadata.get('file_type', 'unknown')\n",
- " if file_type not in doc_types:\n",
- " doc_types[file_type] = []\n",
- " doc_types[file_type].append(doc)\n",
- " \n",
- " print(f\"\\n๐ Documents by MIME type:\")\n",
- " for mime_type, docs in sorted(doc_types.items()):\n",
- " print(f\" {mime_type}: {len(docs)} documents\")\n",
- " \n",
- " return documents\n",
- "\n",
- "# Load all multimodal documents\n",
- "documents = load_multimodal_documents()\n",
- "\n",
- "# Show sample document information\n",
- "if documents:\n",
- " print(f\"\\n๐ Sample Document Analysis:\")\n",
- " sample_doc = documents[0]\n",
- " print(f\"File: {sample_doc.metadata.get('file_name', 'Unknown')}\")\n",
- " print(f\"Type: {sample_doc.metadata.get('file_type', 'Unknown')}\")\n",
- " print(f\"Size: {sample_doc.metadata.get('file_size', 0)} bytes\")\n",
- " print(f\"Text preview: {sample_doc.text[:200]}...\")\n",
- " print(f\"Metadata keys: {list(sample_doc.metadata.keys())}\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 5. Creating Multimodal Vector Index\n",
- "\n",
- "Now let's create a vector index that can handle our multimodal documents using LanceDB for efficient storage and retrieval.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 12:57:10,572 - WARNING - Table multimodal_documents doesn't exist yet. Please add some data to create it.\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "๐ Setting up multimodal vector storage...\n",
- "โ Connected to LanceDB at: storage/multimodal_vectordb\n",
- "โ LanceDB vector store created for multimodal data\n",
- "๐จ Creating new multimodal vector index...\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "Parsing nodes: 100%|โโโโโโโโโโ| 42/42 [00:00<00:00, 87.97it/s]\n",
- "Generating embeddings: 100%|โโโโโโโโโโ| 55/55 [00:02<00:00, 21.76it/s]\n",
- "2025-09-20 12:57:13,598 - INFO - Create new table multimodal_documents adding data.\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Multimodal index created in 3.07 seconds\n",
- "๐พ Saving multimodal index to storage...\n",
- "โ Index saved successfully\n",
- "โ
Multimodal RAG system ready for cross-modal queries!\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "\u001b[90m[\u001b[0m2025-09-20T07:27:13Z \u001b[33mWARN \u001b[0m lance::dataset::write::insert\u001b[90m]\u001b[0m No existing dataset at /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/storage/multimodal_vectordb/multimodal_documents.lance, it will be created\n"
- ]
- }
- ],
- "source": [
- "# Vector store and index creation\n",
- "from llama_index.vector_stores.lancedb import LanceDBVectorStore\n",
- "from llama_index.core import StorageContext, VectorStoreIndex\n",
- "\n",
- "def create_multimodal_vector_store(vector_db_path: str = None):\n",
- " \"\"\"Create and configure LanceDB vector store for multimodal data.\"\"\"\n",
- " if vector_db_path is None:\n",
- " vector_db_path = CONFIG[\"vector_db_path\"]\n",
- " \n",
- " try:\n",
- " import lancedb\n",
- " \n",
- " # Create storage directory\n",
- " Path(vector_db_path).parent.mkdir(parents=True, exist_ok=True)\n",
- " \n",
- " # Connect to LanceDB\n",
- " db = lancedb.connect(str(vector_db_path))\n",
- " print(f\"โ Connected to LanceDB at: {vector_db_path}\")\n",
- " \n",
- " # Create vector store\n",
- " vector_store = LanceDBVectorStore(\n",
- " uri=str(vector_db_path), \n",
- " table_name=\"multimodal_documents\"\n",
- " )\n",
- " print(\"โ LanceDB vector store created for multimodal data\")\n",
- " \n",
- " return vector_store\n",
- " \n",
- " except Exception as e:\n",
- " print(f\"Error creating vector store: {e}\")\n",
- " return None\n",
- "\n",
- "def create_multimodal_index(documents: List, \n",
- " vector_store, \n",
- " index_storage_path: str = None,\n",
- " force_rebuild: bool = False):\n",
- " \"\"\"Create or load a multimodal vector index.\"\"\"\n",
- " \n",
- " if index_storage_path is None:\n",
- " index_storage_path = CONFIG[\"index_storage_path\"]\n",
- " \n",
- " index_path = Path(index_storage_path)\n",
- " index_path.mkdir(parents=True, exist_ok=True)\n",
- " \n",
- " # Check if index already exists\n",
- " index_store_file = index_path / \"index_store.json\"\n",
- " \n",
- " if not force_rebuild and index_store_file.exists():\n",
- " print(\"๐ Loading existing multimodal index...\")\n",
- " try:\n",
- " storage_context = StorageContext.from_defaults(\n",
- " persist_dir=str(index_path), \n",
- " vector_store=vector_store\n",
- " )\n",
- " \n",
- " index = VectorStoreIndex.from_vector_store(\n",
- " vector_store=vector_store,\n",
- " storage_context=storage_context\n",
- " )\n",
- " print(\"โ Successfully loaded existing multimodal index\")\n",
- " return index\n",
- " \n",
- " except Exception as e:\n",
- " print(f\"โ ๏ธ Error loading existing index: {e}\")\n",
- " print(\"Creating new index...\")\n",
- " \n",
- " if not documents:\n",
- " print(\"โ No documents to index\")\n",
- " return None\n",
- " \n",
- " print(\"๐จ Creating new multimodal vector index...\")\n",
- " start_time = time.time()\n",
- " \n",
- " # Create storage context with vector store\n",
- " storage_context = StorageContext.from_defaults(vector_store=vector_store)\n",
- " \n",
- " # Create index with progress bar\n",
- " index = VectorStoreIndex.from_documents(\n",
- " documents, \n",
- " storage_context=storage_context, \n",
- " show_progress=True\n",
- " )\n",
- " \n",
- " end_time = time.time()\n",
- " print(f\"โ Multimodal index created in {end_time - start_time:.2f} seconds\")\n",
- " \n",
- " # Save index to storage\n",
- " print(\"๐พ Saving multimodal index to storage...\")\n",
- " index.storage_context.persist(persist_dir=str(index_path))\n",
- " print(\"โ Index saved successfully\")\n",
- " \n",
- " return index\n",
- "\n",
- "# Create vector store and index for multimodal data\n",
- "print(\"๐ Setting up multimodal vector storage...\")\n",
- "multimodal_vector_store = create_multimodal_vector_store()\n",
- "\n",
- "if multimodal_vector_store and documents:\n",
- " multimodal_index = create_multimodal_index(\n",
- " documents=documents, \n",
- " vector_store=multimodal_vector_store,\n",
- " force_rebuild=False\n",
- " )\n",
- " \n",
- " if multimodal_index:\n",
- " print(\"โ
Multimodal RAG system ready for cross-modal queries!\")\n",
- " else:\n",
- " print(\"โ Failed to create multimodal index\")\n",
- "else:\n",
- " print(\"โ Vector store creation failed or no documents available\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 6. Multimodal Query Engine and Cross-Modal Search\n",
- "\n",
- "Now let's create a query engine that can search across all our different data types and demonstrate cross-modal queries.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Multimodal retriever configured to find top 5 similar chunks\n",
- "โ Multimodal query engine setup successfully\n",
- "๐ Multimodal query engine ready for cross-modal search!\n"
- ]
- }
- ],
- "source": [
- "# Query engine setup\n",
- "from llama_index.core.query_engine import RetrieverQueryEngine\n",
- "from llama_index.core.retrievers import VectorIndexRetriever\n",
- "\n",
- "def setup_multimodal_query_engine(index, similarity_top_k: int = None):\n",
- " \"\"\"Setup query engine for multimodal search.\"\"\"\n",
- " if similarity_top_k is None:\n",
- " similarity_top_k = CONFIG[\"similarity_top_k\"]\n",
- " \n",
- " if not index:\n",
- " print(\"โ Index not available. Please create index first.\")\n",
- " return None\n",
- " \n",
- " try:\n",
- " # Create retriever for multimodal search\n",
- " retriever = VectorIndexRetriever(\n",
- " index=index,\n",
- " similarity_top_k=similarity_top_k,\n",
- " )\n",
- " print(f\"โ Multimodal retriever configured to find top {similarity_top_k} similar chunks\")\n",
- " \n",
- " # Create query engine\n",
- " query_engine = RetrieverQueryEngine(retriever=retriever)\n",
- " print(\"โ Multimodal query engine setup successfully\")\n",
- " \n",
- " return query_engine\n",
- " \n",
- " except Exception as e:\n",
- " print(f\"โ Error setting up query engine: {e}\")\n",
- " return None\n",
- "\n",
- "def search_multimodal_documents(query_engine, query: str, include_metadata: bool = True) -> Dict[str, any]:\n",
- " \"\"\"Search across multimodal documents and return detailed results.\"\"\"\n",
- " if not query_engine:\n",
- " return {\n",
- " \"success\": False,\n",
- " \"error\": \"Query engine not initialized.\",\n",
- " \"response\": \"\",\n",
- " \"sources\": [],\n",
- " }\n",
- " \n",
- " try:\n",
- " print(f\"๐ Searching across multimodal data: '{query}'\")\n",
- " start_time = time.time()\n",
- " \n",
- " # Query the multimodal RAG system\n",
- " response = query_engine.query(query)\n",
- " \n",
- " end_time = time.time()\n",
- " \n",
- " # Extract source information from retrieved nodes\n",
- " sources = []\n",
- " if hasattr(response, \"source_nodes\"):\n",
- " for node in response.source_nodes:\n",
- " source_info = {\n",
- " \"text\": (\n",
- " node.text[:300] + \"...\"\n",
- " if len(node.text) > 300\n",
- " else node.text\n",
- " ),\n",
- " \"score\": getattr(node, \"score\", 0.0),\n",
- " }\n",
- " \n",
- " # Add metadata if available and requested\n",
- " if include_metadata and hasattr(node, \"metadata\"):\n",
- " metadata = node.metadata\n",
- " source_info.update({\n",
- " \"file_name\": metadata.get(\"file_name\", \"Unknown\"),\n",
- " \"file_type\": metadata.get(\"file_type\", \"Unknown\"),\n",
- " \"file_path\": metadata.get(\"file_path\", \"Unknown\"),\n",
- " \"file_size\": metadata.get(\"file_size\", 0),\n",
- " })\n",
- " \n",
- " sources.append(source_info)\n",
- " \n",
- " result = {\n",
- " \"success\": True,\n",
- " \"response\": str(response),\n",
- " \"sources\": sources,\n",
- " \"query\": query,\n",
- " \"search_time\": end_time - start_time,\n",
- " \"num_sources\": len(sources),\n",
- " }\n",
- " \n",
- " print(f\"โ Search completed in {end_time - start_time:.2f} seconds\")\n",
- " print(f\"๐ Found {len(sources)} relevant sources across different file types\")\n",
- " \n",
- " return result\n",
- " \n",
- " except Exception as e:\n",
- " print(f\"โ Error during search: {e}\")\n",
- " return {\"success\": False, \"error\": str(e), \"response\": \"\", \"sources\": []}\n",
- "\n",
- "# Setup multimodal query engine\n",
- "if 'multimodal_index' in locals() and multimodal_index:\n",
- " multimodal_query_engine = setup_multimodal_query_engine(multimodal_index)\n",
- " \n",
- " if multimodal_query_engine:\n",
- " print(\"๐ Multimodal query engine ready for cross-modal search!\")\n",
- " else:\n",
- " print(\"โ Failed to setup multimodal query engine\")\n",
- "else:\n",
- " print(\"โ Multimodal index not available\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 7. Interactive Multimodal Query Examples\n",
- "\n",
- "Let's demonstrate the power of our multimodal RAG system with cross-modal queries that search across different data types simultaneously.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "๐ฏ Testing Diverse Multimodal Queries\n",
- "============================================================\n",
- "\n",
- "=============== Query 1: What is the prep time for Spag... ===============\n",
- "โ Multimodal Question: What is the prep time for Spaghetti Carbonara?\n",
- "======================================================================\n",
- "๐ Searching across multimodal data: 'What is the prep time for Spaghetti Carbonara?'\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 12:57:24,186 - INFO - query_type :, vector\n",
- "2025-09-20 12:57:27,616 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 12:57:30,526 - INFO - query_type :, vector\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Search completed in 6.71 seconds\n",
- "๐ Found 5 relevant sources across different file types\n",
- "๐ก Answer:\n",
- "15 minutes\n",
- "\n",
- "๐ Search completed in 6.71 seconds\n",
- "๐ Found 5 relevant sources across different data types\n",
- "\n",
- "๐ Source File Types: {'Unknown': 1, 'text/csv': 2, 'image/png': 1, 'application/pdf': 1}\n",
- "\n",
- "๐ Top Sources:\n",
- "1. recipe_instructions.md (Unknown)\n",
- " Score: 0.675\n",
- " Content: # ๐ Classic Spaghetti Carbonara Recipe\n",
- "\n",
- "## Ingredients\n",
- "- 400g spaghetti pasta\n",
- "- 4 large egg yolks\n",
- "- 100g pecorino romano cheese (grated)\n",
- "- 150g guanci...\n",
- "\n",
- "2. italian_recipes.csv (text/csv)\n",
- " Score: 0.505\n",
- " Content: Spaghetti Carbonara, Italian, 20, Easy, Pasta, 450\n",
- "Margherita Pizza, Italian, 45, Medium, Tomato, 320\n",
- "Risotto Milanese, Italian, 35, Hard, Rice, 380\n",
- "T...\n",
- "\n",
- "3. agent_performance_benchmark.csv (text/csv)\n",
- " Score: 0.400\n",
- " Content: ReAct-GPT4, reasoning, 0.87, 1200, 45.2, 0.02, langchain\n",
- "AutoGPT, autonomous, 0.78, 2100, 78.5, 0.035, autogpt\n",
- "LangChain-Agent, tool_using, 0.82, 950,...\n",
- "\n",
- "\n",
- "======================================================================\n",
- "\n",
- "=============== Query 2: Which stock had the highest re... ===============\n",
- "โ Multimodal Question: Which stock had the highest return in my portfolio?\n",
- "======================================================================\n",
- "๐ Searching across multimodal data: 'Which stock had the highest return in my portfolio?'\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 12:57:31,779 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 12:57:36,000 - INFO - query_type :, vector\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Search completed in 5.51 seconds\n",
- "๐ Found 5 relevant sources across different file types\n",
- "๐ก Answer:\n",
- "Empty Response\n",
- "\n",
- "๐ Search completed in 5.51 seconds\n",
- "๐ Found 5 relevant sources across different data types\n",
- "\n",
- "๐ Source File Types: {'text/csv': 1, 'image/png': 1, 'Unknown': 1, 'application/pdf': 2}\n",
- "\n",
- "๐ Top Sources:\n",
- "1. investment_portfolio.csv (text/csv)\n",
- " Score: 0.552\n",
- " Content: Stock, AAPL, Apple Inc, 10000, 12500, 25.0, Medium\n",
- "Stock, GOOGL, Alphabet Inc, 8000, 9200, 15.0, Medium\n",
- "Stock, TSLA, Tesla Inc, 5000, 4200, -16.0, Hig...\n",
- "\n",
- "2. stock_performance.png (image/png)\n",
- " Score: 0.472\n",
- " Content: ...\n",
- "\n",
- "3. market_analysis.md (Unknown)\n",
- " Score: 0.449\n",
- " Content: # ๐ Q3 2024 Market Analysis Report\n",
- "\n",
- "## Executive Summary\n",
- "\n",
- "The third quarter of 2024 showed mixed performance across different asset classes, with tech...\n",
- "\n",
- "\n",
- "======================================================================\n",
- "\n",
- "=============== Query 3: What is the best time to visit... ===============\n",
- "โ Multimodal Question: What is the best time to visit Tokyo?\n",
- "======================================================================\n",
- "๐ Searching across multimodal data: 'What is the best time to visit Tokyo?'\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 12:57:37,280 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 12:57:43,231 - INFO - query_type :, vector\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Search completed in 7.23 seconds\n",
- "๐ Found 5 relevant sources across different file types\n",
- "๐ก Answer:\n",
- "The best times to visit Tokyo are MarchโMay (cherry blossom season) and SeptemberโNovember.\n",
- "\n",
- "๐ Search completed in 7.23 seconds\n",
- "๐ Found 5 relevant sources across different data types\n",
- "\n",
- "๐ Source File Types: {'Unknown': 2, 'image/png': 1, 'application/pdf': 2}\n",
- "\n",
- "๐ Top Sources:\n",
- "1. city_guides.md (Unknown)\n",
- " Score: 0.545\n",
- " Content: # Ultimate City Travel Guide\n",
- "\n",
- "## Paris, France ๐ซ๐ท\n",
- "\n",
- "**Best Time to Visit:** April-June, September-October\n",
- "**Must-See Attractions:**\n",
- "- Eiffel Tower - Ic...\n",
- "\n",
- "2. recipe_instructions.md (Unknown)\n",
- " Score: 0.344\n",
- " Content: # ๐ Classic Spaghetti Carbonara Recipe\n",
- "\n",
- "## Ingredients\n",
- "- 400g spaghetti pasta\n",
- "- 4 large egg yolks\n",
- "- 100g pecorino romano cheese (grated)\n",
- "- 150g guanci...\n",
- "\n",
- "3. city_temperatures.png (image/png)\n",
- " Score: 0.337\n",
- " Content: ...\n",
- "\n",
- "\n",
- "======================================================================\n",
- "\n",
- "==================== Custom Question ====================\n",
- "โ Multimodal Question: What is the prep time for Italian recipes?\n",
- "======================================================================\n",
- "๐ Searching across multimodal data: 'What is the prep time for Italian recipes?'\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 12:57:44,554 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Search completed in 21.89 seconds\n",
- "๐ Found 5 relevant sources across different file types\n",
- "๐ก Answer:\n",
- "Empty Response\n",
- "\n",
- "๐ Search completed in 21.89 seconds\n",
- "๐ Found 5 relevant sources across different data types\n",
- "\n",
- "๐ Source File Types: {'Unknown': 2, 'text/csv': 1, 'image/png': 1, 'audio/mpeg': 1}\n",
- "\n",
- "๐ Top Sources:\n",
- "1. recipe_instructions.md (Unknown)\n",
- " Score: 0.572\n",
- " Content: # ๐ Classic Spaghetti Carbonara Recipe\n",
- "\n",
- "## Ingredients\n",
- "- 400g spaghetti pasta\n",
- "- 4 large egg yolks\n",
- "- 100g pecorino romano cheese (grated)\n",
- "- 150g guanci...\n",
- "\n",
- "2. italian_recipes.csv (text/csv)\n",
- " Score: 0.547\n",
- " Content: Spaghetti Carbonara, Italian, 20, Easy, Pasta, 450\n",
- "Margherita Pizza, Italian, 45, Medium, Tomato, 320\n",
- "Risotto Milanese, Italian, 35, Hard, Rice, 380\n",
- "T...\n",
- "\n",
- "3. city_guides.md (Unknown)\n",
- " Score: 0.405\n",
- " Content: # Ultimate City Travel Guide\n",
- "\n",
- "## Paris, France ๐ซ๐ท\n",
- "\n",
- "**Best Time to Visit:** April-June, September-October\n",
- "**Must-See Attractions:**\n",
- "- Eiffel Tower - Ic...\n",
- "\n"
- ]
- }
- ],
- "source": [
- "def ask_multimodal_question(query_engine, question: str, show_sources: bool = True):\n",
- " \"\"\"\n",
- " Ask a custom question to the multimodal RAG system and display results.\n",
- " \n",
- " Args:\n",
- " query_engine: The configured multimodal query engine\n",
- " question (str): Your question about the multimodal data\n",
- " show_sources (bool): Whether to display source information\n",
- " \"\"\"\n",
- " print(f\"โ Multimodal Question: {question}\")\n",
- " print(\"=\" * 70)\n",
- " \n",
- " result = search_multimodal_documents(query_engine, question, include_metadata=True)\n",
- " \n",
- " if result[\"success\"]:\n",
- " print(f\"๐ก Answer:\")\n",
- " print(result[\"response\"])\n",
- " print(f\"\\n๐ Search completed in {result['search_time']:.2f} seconds\")\n",
- " print(f\"๐ Found {result['num_sources']} relevant sources across different data types\")\n",
- " \n",
- " if show_sources and result[\"sources\"]:\n",
- " # Show file type distribution\n",
- " file_types = {}\n",
- " for source in result[\"sources\"]:\n",
- " file_type = source.get(\"file_type\", \"unknown\")\n",
- " if file_type not in file_types:\n",
- " file_types[file_type] = 0\n",
- " file_types[file_type] += 1\n",
- " \n",
- " print(f\"\\n๐ Source File Types: {dict(file_types)}\")\n",
- " \n",
- " print(f\"\\n๐ Top Sources:\")\n",
- " for i, source in enumerate(result[\"sources\"][:3], 1):\n",
- " print(f\"{i}. {source.get('file_name', 'Unknown')} ({source.get('file_type', 'Unknown')})\")\n",
- " print(f\" Score: {source.get('score', 0):.3f}\")\n",
- " print(f\" Content: {source['text'][:150]}...\")\n",
- " print()\n",
- " \n",
- " else:\n",
- " print(f\"โ Error: {result['error']}\")\n",
- "\n",
- "# # Example multimodal queries\n",
- "# multimodal_queries = [\n",
- "# # \"What are the performance benchmarks for different AI agents?\",\n",
- "# # \"How do I configure a ReAct agent for research tasks?\", \n",
- "# # \"What are the architectural patterns discussed in the agent frameworks?\",\n",
- "# # \"Which AI agent has the best accuracy score?\",\n",
- "# # \"What are the cost implications of different agent models?\"\n",
- "# \"What is the accuracy_score for the ReAct agent?\"\n",
- "# ]\n",
- "\n",
- "# print(\"๐ฏ Multimodal Query Demonstrations\")\n",
- "# print(\"=\" * 60)\n",
- "\n",
- "# # Run a few example queries\n",
- "# for i, question in enumerate(multimodal_queries[:3], 1):\n",
- "# print(f\"\\n{'='*20} Example {i} {'='*20}\")\n",
- " \n",
- "# if 'multimodal_query_engine' in locals() and multimodal_query_engine:\n",
- "# ask_multimodal_question(multimodal_query_engine, question, show_sources=True)\n",
- "# else:\n",
- "# print(\"โ Multimodal query engine not available\")\n",
- " \n",
- "# if i < 3:\n",
- "# print(\"\\n\" + \"=\"*60)\n",
- "\n",
- "# Demo queries for diverse data types\n",
- "diverse_queries = [\n",
- " \"What is the prep time for Spaghetti Carbonara?\", # Should hit cooking CSV\n",
- " \"Which stock had the highest return in my portfolio?\", # Should hit finance CSV\n",
- " \"What is the best time to visit Tokyo?\", # Should hit travel markdown\n",
- " \"How many calories did I burn on Tuesday?\", # Should hit health HTML\n",
- " \"What are the steps to make Carbonara?\", # Should hit cooking markdown\n",
- " \"What was NVIDIA's performance?\", # Should hit finance data\n",
- "]\n",
- "\n",
- "print(\"๐ฏ Testing Diverse Multimodal Queries\")\n",
- "print(\"=\" * 60)\n",
- "\n",
- "# Test one query from each topic\n",
- "for i, question in enumerate(diverse_queries[:3], 1):\n",
- " print(f\"\\n{'='*15} Query {i}: {question[:30]}... {'='*15}\")\n",
- " ask_multimodal_question(multimodal_query_engine, question, show_sources=True)\n",
- " print(\"\\n\" + \"=\"*70)\n",
- "\n",
- "# Custom question area\n",
- "print(f\"\\n{'='*20} Custom Question {'='*20}\")\n",
- "custom_question = \"What is the prep time for Italian recipes?\"\n",
- "ask_multimodal_question(multimodal_query_engine, custom_question, show_sources=True)\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Multimodal Question: in the end it doesn't even matter\n",
- "======================================================================\n",
- "๐ Searching across multimodal data: 'in the end it doesn't even matter'\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 12:58:16,704 - INFO - query_type :, vector\n",
- "2025-09-20 12:58:18,223 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Search completed in 7.48 seconds\n",
- "๐ Found 5 relevant sources across different file types\n",
- "๐ก Answer:\n",
- "Empty Response\n",
- "\n",
- "๐ Search completed in 7.48 seconds\n",
- "๐ Found 5 relevant sources across different data types\n",
- "\n",
- "๐ Source File Types: {'audio/mpeg': 2, 'application/pdf': 2, 'text/csv': 1}\n",
- "\n",
- "๐ Top Sources:\n",
- "1. in_the_end.mp3 (audio/mpeg)\n",
- " Score: 0.433\n",
- " Content: I tried so hard and got so far In the end, it doesn't even matter I had to fall to lose it all In the end, it doesn't even matter...\n",
- "\n",
- "2. Emerging_Agent_Architectures.pdf (application/pdf)\n",
- " Score: 0.375\n",
- " Content: Message subscribing or filtering improves multi-agent\n",
- "performance by ensuring agents only receive information relevant to their tasks.\n",
- "In vertical arc...\n",
- "\n",
- "3. Emerging_Agent_Architectures.pdf (application/pdf)\n",
- " Score: 0.368\n",
- " Content: complete problems [16, 23, 32]. They often do this by breaking a larger problem into smaller subproblems, and then\n",
- "solving each one with the appropria...\n",
- "\n"
- ]
- }
- ],
- "source": [
- "custom_question=\"in the end it doesn't even matter\"\n",
- "ask_multimodal_question(multimodal_query_engine, custom_question, show_sources=True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Conclusion\n",
- "\n",
- "๐ **Congratulations!** You have successfully built an advanced **Multimodal RAG System** using LlamaIndex's `SimpleDirectoryReader` with comprehensive cross-modal capabilities.\n",
- "\n",
- "### What We Accomplished\n",
- "\n",
- "This tutorial demonstrated building a RAG system that can handle multiple data types:\n",
- "\n",
- "#### 1. **Multimodal Document Loading**\n",
- "- โ
**PDF Documents**: Academic research papers on AI agents\n",
- "- โ
**CSV Files**: Agent performance benchmarks and evaluation metrics \n",
- "- โ
**Markdown Files**: Framework comparisons and documentation\n",
- "- โ
**HTML Files**: Tutorial and instructional content\n",
- "- โ
**Image Files**: Charts, diagrams, and visual content\n",
- "- โ
**Audio Files**: Supplementary audio content\n",
- "\n",
- "#### 2. **Key Features Implemented**\n",
- "- โ
**Hardcoded Configuration**: No external config files needed\n",
- "- โ
**Cross-Modal Search**: Query across all file types simultaneously\n",
- "- โ
**Semantic Similarity**: Find relevant content regardless of source format\n",
- "- โ
**Source Attribution**: Track which file types contributed to answers\n",
- "- โ
**LanceDB Vector Store**: Efficient multimodal document storage\n",
- "- โ
**OpenRouter Integration**: Using `gpt-4o` for response generation\n",
- "- โ
**Local Embeddings**: `BAAI/bge-small-en-v1.5` for cost-effective embedding\n",
- "\n",
- "#### 3. **SimpleDirectoryReader Capabilities**\n",
- "According to the [official documentation](https://developers.llamaindex.ai/python/framework/module_guides/loading/simpledirectoryreader/), we successfully utilized:\n",
- "\n",
- "```python\n",
- "# Basic multimodal loading\n",
- "SimpleDirectoryReader(input_dir=\"../../data\", recursive=True)\n",
- "\n",
- "# Advanced features available\n",
- "SimpleDirectoryReader(\n",
- " input_dir=\"path/to/directory\",\n",
- " recursive=True, # Search subdirectories\n",
- " required_exts=[\".pdf\", \".csv\"], # Filter file types\n",
- " exclude=[\"file1.txt\"], # Exclude specific files\n",
- " file_metadata=custom_func, # Custom metadata extraction\n",
- " num_files_limit=100, # Limit number of files\n",
- " encoding=\"utf-8\" # Specify encoding\n",
- ")\n",
- "```\n",
- "\n",
- "### Real-World Applications\n",
- "\n",
- "This multimodal RAG system can be applied to:\n",
- "\n",
- "- **Research and Academia**: Query across papers, datasets, and supplementary materials\n",
- "- **Documentation Systems**: Search technical docs, tutorials, configs, and examples\n",
- "- **Business Intelligence**: Combine reports, spreadsheets, presentations, and recordings\n",
- "- **Content Management**: Organize and search diverse content libraries\n",
- "- **Knowledge Bases**: Build comprehensive Q&A systems with diverse source materials\n",
- "\n",
- "### Next Steps\n",
- "\n",
- "1. **Extend File Types**: Add `.docx`, `.pptx`, `.epub` support\n",
- "2. **Custom Metadata**: Implement domain-specific metadata extraction\n",
- "3. **Hybrid Search**: Combine vector search with keyword search\n",
- "4. **Performance Optimization**: Use iterative loading for large datasets\n",
- "5. **Multi-Language Support**: Test with international documents\n",
- "\n",
- "### Usage Tips\n",
- "\n",
- "- **Query Optimization**: Use specific queries that benefit from cross-modal information\n",
- "- **File Organization**: Structure data directories logically\n",
- "- **Custom Questions**: Modify the `custom_question` variable to test your own queries\n",
- "- **Monitor Sources**: Check file type distribution in results to understand retrieval patterns\n",
- "\n",
- "Happy building with multimodal RAG! ๐๐๐\n",
- "\n",
- "---\n",
- "\n",
- "**Ready to explore?** Run the cells above and try your own questions with the interactive query interface!\n"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "accelerator",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.13"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/Girish_Basavaraj_Hiremath/session_2/llamaindex_rag/03_advanced_rag_techniques.ipynb b/Girish_Basavaraj_Hiremath/session_2/llamaindex_rag/03_advanced_rag_techniques.ipynb
deleted file mode 100644
index b98f810..0000000
--- a/Girish_Basavaraj_Hiremath/session_2/llamaindex_rag/03_advanced_rag_techniques.ipynb
+++ /dev/null
@@ -1,1928 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Advanced RAG Techniques with LlamaIndex\n",
- "\n",
- "This notebook demonstrates sophisticated RAG techniques that transform basic document retrieval into production-ready, intelligent systems. We'll explore techniques that solve real-world challenges like noisy retrieval, inconsistent response quality, and unstructured outputs.\n",
- "\n",
- "## Why Advanced RAG Techniques Matter\n",
- "\n",
- "**Basic RAG limitations:**\n",
- "- Retrieves irrelevant chunks (low precision)\n",
- "- Inconsistent response quality across queries\n",
- "- No control over response structure\n",
- "- Difficulty handling complex, multi-part questions\n",
- "- Poor performance on domain-specific tasks\n",
- "\n",
- "**Advanced techniques solve these by adding:**\n",
- "- Intelligent filtering and reranking\n",
- "- Sophisticated response synthesis strategies\n",
- "- Type-safe, structured outputs\n",
- "- Domain-specific customization\n",
- "\n",
- "## Advanced Concepts Covered\n",
- "\n",
- "### ๐ง [Node Postprocessors](https://developers.llamaindex.ai/python/framework/module_guides/querying/node_postprocessors/)\n",
- "**Purpose**: Refine retrieval results after initial vector search\n",
- "- **Similarity Filtering**: Remove chunks below relevance threshold (essential for noisy datasets)\n",
- "- **Reranking**: Re-order results using specialized models (improves precision by 20-40%)\n",
- "- **Custom Filtering**: Apply business rules (exclude sensitive content, enforce data freshness)\n",
- "- **Use Case**: Clean up retrieval for production systems where precision matters\n",
- "\n",
- "### ๐ฏ [Response Synthesizers](https://developers.llamaindex.ai/python/framework/module_guides/querying/response_synthesizers/)\n",
- "**Purpose**: Control how retrieved information becomes final answers\n",
- "- **Tree Summarize**: Handle complex queries by building responses hierarchically (best for analytical questions)\n",
- "- **Refine**: Iteratively improve answers with multiple information sources (comprehensive analysis)\n",
- "- **Compact**: Optimize token usage while maintaining quality (cost-effective production)\n",
- "- **Custom Templates**: Domain-specific response formatting (consistency across use cases)\n",
- "- **Use Case**: Ensure response quality matches business requirements and user expectations\n",
- "\n",
- "### ๐ [Advanced Retrievers](https://developers.llamaindex.ai/python/framework/module_guides/querying/retriever/)\n",
- "**Purpose**: Go beyond simple vector similarity for better information discovery\n",
- "- **Hybrid Search**: Combine semantic similarity with keyword matching (captures exact terms + meaning)\n",
- "- **Multi-Index Retrieval**: Query multiple specialized indexes simultaneously (comprehensive coverage)\n",
- "- **Auto-Merging**: Intelligently combine related chunks (context preservation)\n",
- "- **Use Case**: Handle diverse query types and improve recall on complex information needs\n",
- "\n",
- "### ๐ [Structured Outputs](https://developers.llamaindex.ai/python/framework/module_guides/querying/structured_outputs/)\n",
- "**Purpose**: Ensure predictable, parseable responses for system integration\n",
- "- **Pydantic Models**: Type-safe data extraction with validation (eliminates parsing errors)\n",
- "- **JSON Schema**: Consistent response formatting (enables downstream processing)\n",
- "- **Multi-Field Extraction**: Extract multiple data points simultaneously (efficient for complex entities)\n",
- "- **Use Case**: API endpoints, data pipelines, and applications requiring reliable structured data\n",
- "\n",
- "---\n",
- "\n",
- "We'll use our diverse multimodal dataset (cooking, finance, travel, health, AI research) to demonstrate how these techniques work across different data types and use cases, showing measurable improvements over basic RAG.\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 1. Environment Setup and Data Loading\n",
- "\n",
- "**Purpose**: Configure optimal settings for advanced RAG techniques and load a diverse dataset for comprehensive testing.\n",
- "\n",
- "**Why This Matters**: Advanced techniques require careful parameter tuning. We use smaller chunk sizes (512 vs 1024) for better precision, higher retrieval counts (10 vs 5) for better postprocessing, and local embeddings to reduce costs during experimentation.\n",
- "\n",
- "**Configuration Strategy**:\n",
- "- **Smaller chunks** โ Better precision for complex queries\n",
- "- **Higher retrieval counts** โ More candidates for intelligent filtering\n",
- "- **Local embeddings** โ Cost-effective development and testing\n",
- "- **Multimodal dataset** โ Test techniques across different content types\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# !pip install -r \"../requirements.txt\""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Advanced RAG environment configured\n",
- "โ LLM Model: gpt-5-mini\n",
- "โ Embedding Model: local:BAAI/bge-small-en-v1.5\n",
- "โ Chunk Size: 512 (optimized for precision)\n",
- "โ Initial Retrieval: 10 candidates\n",
- "โ Final Results: 5 after postprocessing\n",
- "๐ Ready for advanced RAG demonstrations!\n"
- ]
- }
- ],
- "source": [
- "# Environment setup with advanced configurations\n",
- "import os\n",
- "import time\n",
- "from pathlib import Path\n",
- "from typing import Dict, List, Optional, Tuple, Any\n",
- "import pandas as pd\n",
- "import json\n",
- "from pydantic import BaseModel, Field\n",
- "from enum import Enum\n",
- "\n",
- "from dotenv import load_dotenv\n",
- "\n",
- "# Advanced configuration for sophisticated RAG\n",
- "CONFIG = {\n",
- " \"llm_model\": \"gpt-5-mini\",\n",
- " \"embedding_model\": \"local:BAAI/bge-small-en-v1.5\",\n",
- " \"chunk_size\": 512, # Smaller chunks for better precision\n",
- " \"chunk_overlap\": 50,\n",
- " \"similarity_top_k\": 10, # More candidates for postprocessing\n",
- " \"final_top_k\": 5, # Final results after postprocessing\n",
- " \"similarity_cutoff\": 0.3, # Filter low-relevance results\n",
- " \"data_path\": \"../data\",\n",
- " \"vector_db_path\": \"storage/advanced_vectordb\",\n",
- " \"index_storage_path\": \"storage/advanced_index\"\n",
- "}\n",
- "\n",
- "def setup_advanced_environment():\n",
- " \"\"\"Setup environment for advanced RAG techniques.\"\"\"\n",
- " load_dotenv()\n",
- " os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n",
- " \n",
- " api_key = os.getenv(\"OPENROUTER_API_KEY\")\n",
- " if not api_key:\n",
- " print(\"โ ๏ธ OPENROUTER_API_KEY not found in environment variables\")\n",
- " return False\n",
- " \n",
- " print(\"โ Advanced RAG environment configured\")\n",
- " print(f\"โ LLM Model: {CONFIG['llm_model']}\")\n",
- " print(f\"โ Embedding Model: {CONFIG['embedding_model']}\")\n",
- " print(f\"โ Chunk Size: {CONFIG['chunk_size']} (optimized for precision)\")\n",
- " print(f\"โ Initial Retrieval: {CONFIG['similarity_top_k']} candidates\")\n",
- " print(f\"โ Final Results: {CONFIG['final_top_k']} after postprocessing\")\n",
- " return True\n",
- "\n",
- "# Initialize environment\n",
- "success = setup_advanced_environment()\n",
- "if success:\n",
- " print(\"๐ Ready for advanced RAG demonstrations!\")\n",
- "else:\n",
- " print(\"โ Environment setup failed!\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 2. LlamaIndex Advanced Configuration\n",
- "\n",
- "**Purpose**: Set up LlamaIndex with precision-optimized settings that maximize the effectiveness of advanced techniques.\n",
- "\n",
- "**Key Optimizations**:\n",
- "- **`chunk_size=512`**: Smaller chunks provide more precise context for postprocessors\n",
- "- **`chunk_overlap=50`**: Minimal overlap reduces redundancy while preserving context\n",
- "- **`similarity_top_k=10`**: More candidates allow postprocessors to filter intelligently\n",
- "- **`final_top_k=5`**: Refined results after advanced processing\n",
- "\n",
- "**Why These Settings Matter**: Advanced techniques work best with more retrieval candidates to filter and refine. The smaller chunk size ensures each piece of retrieved information is highly relevant, while higher retrieval counts give postprocessors room to improve precision.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "/Users/ishandutta/miniconda3/envs/accelerator/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
- " from .autonotebook import tqdm as notebook_tqdm\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ LLM configured: gpt-5-mini\n",
- "โ Embedding model: local:BAAI/bge-small-en-v1.5\n",
- "โ Node parser: 512 chars, 50 overlap\n",
- "\n",
- "๐ Loading multimodal dataset...\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "/Users/ishandutta/miniconda3/envs/accelerator/lib/python3.11/site-packages/whisper/transcribe.py:132: UserWarning: FP16 is not supported on CPU; using FP32 instead\n",
- " warnings.warn(\"FP16 is not supported on CPU; using FP32 instead\")\n",
- "/Users/ishandutta/miniconda3/envs/accelerator/lib/python3.11/site-packages/whisper/transcribe.py:132: UserWarning: FP16 is not supported on CPU; using FP32 instead\n",
- " warnings.warn(\"FP16 is not supported on CPU; using FP32 instead\")\n",
- "/Users/ishandutta/miniconda3/envs/accelerator/lib/python3.11/site-packages/whisper/transcribe.py:132: UserWarning: FP16 is not supported on CPU; using FP32 instead\n",
- " warnings.warn(\"FP16 is not supported on CPU; using FP32 instead\")\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ
Loaded 42 documents in 8.00s\n",
- "\n",
- "๐ Document Types:\n",
- " application/pdf: 23 documents\n",
- " audio/mpeg: 3 documents\n",
- " image/png: 6 documents\n",
- " text/csv: 4 documents\n",
- " text/html: 2 documents\n",
- " unknown: 4 documents\n",
- "\n",
- "โ
Advanced configuration complete!\n"
- ]
- }
- ],
- "source": [
- "# Advanced LlamaIndex configuration\n",
- "from llama_index.core import Settings, SimpleDirectoryReader\n",
- "from llama_index.llms.openrouter import OpenRouter\n",
- "from llama_index.core.embeddings import resolve_embed_model\n",
- "from llama_index.core.node_parser import SentenceSplitter\n",
- "\n",
- "def configure_advanced_settings():\n",
- " \"\"\"Configure LlamaIndex for advanced RAG techniques.\"\"\"\n",
- " \n",
- " # LLM configuration\n",
- " Settings.llm = OpenRouter(\n",
- " api_key=os.getenv(\"OPENROUTER_API_KEY\"),\n",
- " model=CONFIG[\"llm_model\"]\n",
- " )\n",
- " print(f\"โ LLM configured: {CONFIG['llm_model']}\")\n",
- "\n",
- " # Embedding configuration\n",
- " Settings.embed_model = resolve_embed_model(CONFIG[\"embedding_model\"])\n",
- " print(f\"โ Embedding model: {CONFIG['embedding_model']}\")\n",
- "\n",
- " # Optimized node parser for better precision\n",
- " Settings.node_parser = SentenceSplitter(\n",
- " chunk_size=CONFIG[\"chunk_size\"], \n",
- " chunk_overlap=CONFIG[\"chunk_overlap\"]\n",
- " )\n",
- " print(f\"โ Node parser: {CONFIG['chunk_size']} chars, {CONFIG['chunk_overlap']} overlap\")\n",
- "\n",
- "# Configure settings\n",
- "configure_advanced_settings()\n",
- "\n",
- "# Load our diverse multimodal dataset\n",
- "print(\"\\n๐ Loading multimodal dataset...\")\n",
- "reader = SimpleDirectoryReader(\n",
- " input_dir=CONFIG[\"data_path\"],\n",
- " recursive=True\n",
- ")\n",
- "\n",
- "start_time = time.time()\n",
- "documents = reader.load_data()\n",
- "load_time = time.time() - start_time\n",
- "\n",
- "print(f\"โ
Loaded {len(documents)} documents in {load_time:.2f}s\")\n",
- "\n",
- "# Analyze document types\n",
- "doc_types = {}\n",
- "for doc in documents:\n",
- " file_type = doc.metadata.get('file_type', 'unknown')\n",
- " doc_types[file_type] = doc_types.get(file_type, 0) + 1\n",
- "\n",
- "print(\"\\n๐ Document Types:\")\n",
- "for file_type, count in sorted(doc_types.items()):\n",
- " print(f\" {file_type}: {count} documents\")\n",
- "\n",
- "print(\"\\nโ
Advanced configuration complete!\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 3. Advanced Vector Index Creation\n",
- "\n",
- "**Purpose**: Build a vector index foundation that supports sophisticated retrieval and postprocessing techniques.\n",
- "\n",
- "**Advanced Index Features**:\n",
- "- **Optimized Chunking**: Smaller, more focused text segments for precise retrieval\n",
- "- **LanceDB Backend**: High-performance vector storage with advanced query capabilities\n",
- "- **StorageContext Persistence**: Complete index state preservation for reproducible results\n",
- "- **Multimodal Support**: Handles diverse content types (PDFs, images, audio, structured data)\n",
- "\n",
- "**Why This Index Design Matters**: Advanced techniques like postprocessors and sophisticated synthesizers require high-quality retrieval as a foundation. Our index creates many small, precise chunks that can be intelligently filtered and combined by advanced techniques, rather than fewer large chunks that may contain irrelevant information.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 12:59:36,643 - WARNING - Table advanced_multimodal doesn't exist yet. Please add some data to create it.\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "๐ Setting up advanced vector index...\n",
- "โ Advanced vector store created\n",
- "๐จ Creating new advanced index...\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "Parsing nodes: 100%|โโโโโโโโโโ| 42/42 [00:00<00:00, 185.67it/s]\n",
- "Generating embeddings: 100%|โโโโโโโโโโ| 94/94 [00:03<00:00, 24.66it/s]\n",
- "2025-09-20 12:59:40,695 - INFO - Create new table advanced_multimodal adding data.\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "โ Index created in 4.06s\n",
- "๐พ Index saved to storage\n",
- "โ
Advanced index ready for sophisticated queries!\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "\u001b[90m[\u001b[0m2025-09-20T07:29:40Z \u001b[33mWARN \u001b[0m lance::dataset::write::insert\u001b[90m]\u001b[0m No existing dataset at /Users/ishandutta/Documents/code/ai-accelerator/Day_6/session_2/llamaindex_rag/storage/advanced_vectordb/advanced_multimodal.lance, it will be created\n"
- ]
- }
- ],
- "source": [
- "# Advanced vector store and index creation\n",
- "from llama_index.vector_stores.lancedb import LanceDBVectorStore\n",
- "from llama_index.core import StorageContext, VectorStoreIndex\n",
- "\n",
- "def create_advanced_vector_index():\n",
- " \"\"\"Create optimized vector index for advanced techniques.\"\"\"\n",
- " \n",
- " # Create vector store\n",
- " try:\n",
- " import lancedb\n",
- " \n",
- " # Setup storage\n",
- " Path(CONFIG[\"vector_db_path\"]).parent.mkdir(parents=True, exist_ok=True)\n",
- " db = lancedb.connect(str(CONFIG[\"vector_db_path\"]))\n",
- " \n",
- " vector_store = LanceDBVectorStore(\n",
- " uri=str(CONFIG[\"vector_db_path\"]), \n",
- " table_name=\"advanced_multimodal\"\n",
- " )\n",
- " print(f\"โ Advanced vector store created\")\n",
- " \n",
- " # Check for existing index\n",
- " index_path = Path(CONFIG[\"index_storage_path\"])\n",
- " index_path.mkdir(parents=True, exist_ok=True)\n",
- " \n",
- " if (index_path / \"index_store.json\").exists():\n",
- " print(\"๐ Loading existing advanced index...\")\n",
- " storage_context = StorageContext.from_defaults(\n",
- " persist_dir=str(index_path), \n",
- " vector_store=vector_store\n",
- " )\n",
- " index = VectorStoreIndex.from_vector_store(\n",
- " vector_store=vector_store,\n",
- " storage_context=storage_context\n",
- " )\n",
- " print(\"โ Existing index loaded successfully\")\n",
- " else:\n",
- " print(\"๐จ Creating new advanced index...\")\n",
- " storage_context = StorageContext.from_defaults(vector_store=vector_store)\n",
- " \n",
- " start_time = time.time()\n",
- " index = VectorStoreIndex.from_documents(\n",
- " documents, \n",
- " storage_context=storage_context, \n",
- " show_progress=True\n",
- " )\n",
- " index_time = time.time() - start_time\n",
- " \n",
- " print(f\"โ Index created in {index_time:.2f}s\")\n",
- " \n",
- " # Persist index\n",
- " index.storage_context.persist(persist_dir=str(index_path))\n",
- " print(\"๐พ Index saved to storage\")\n",
- " \n",
- " return index, vector_store\n",
- " \n",
- " except Exception as e:\n",
- " print(f\"โ Error creating advanced index: {e}\")\n",
- " return None, None\n",
- "\n",
- "# Create the advanced index\n",
- "print(\"๐ Setting up advanced vector index...\")\n",
- "advanced_index, advanced_vector_store = create_advanced_vector_index()\n",
- "\n",
- "if advanced_index:\n",
- " print(\"โ
Advanced index ready for sophisticated queries!\")\n",
- "else:\n",
- " print(\"โ Failed to create advanced index\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 4. Node Postprocessors - Intelligent Result Filtering\n",
- "\n",
- "**The Problem**: Vector search often returns chunks with varying relevance quality. Some may be tangentially related, contain outdated information, or include unwanted content. Raw vector similarity doesn't account for business rules or content quality.\n",
- "\n",
- "**The Solution**: Node postprocessors act as intelligent filters that run after vector retrieval, applying sophisticated logic to improve result quality.\n",
- "\n",
- "**Key Postprocessor Types**:\n",
- "\n",
- "### ๐ฏ SimilarityPostprocessor\n",
- "- **Purpose**: Remove chunks below a relevance threshold\n",
- "- **When to Use**: Always in production (minimal cost, significant improvement)\n",
- "- **Impact**: Typically improves precision by 15-30% by removing noise\n",
- "- **Best Practice**: Start with 0.3 threshold, tune based on your data\n",
- "\n",
- "### ๐ KeywordNodePostprocessor \n",
- "- **Purpose**: Filter based on required/excluded terms\n",
- "- **When to Use**: Domain-specific filtering (remove sensitive content, ensure topic focus)\n",
- "- **Impact**: Ensures responses stay on-topic and comply with business rules\n",
- "- **Best Practice**: Use exclude lists for sensitive terms, required lists for focus\n",
- "\n",
- "### ๐ Multi-Stage Processing\n",
- "- **Purpose**: Chain multiple filters for comprehensive refinement\n",
- "- **When to Use**: Production systems requiring high precision\n",
- "- **Impact**: Combines benefits of multiple filtering strategies\n",
- "- **Best Practice**: Order from general (similarity) to specific (keyword) filters\n",
- "\n",
- "**Real-World Impact**: Postprocessors typically improve user satisfaction by 25-40% by reducing irrelevant information in responses, while adding minimal latency (50-200ms) and cost.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "๐ง Node Postprocessor Demonstrations\n",
- "============================================================\n",
- "\n",
- "1๏ธโฃ Similarity Postprocessor - Relevance Filtering\n",
- "--------------------------------------------------\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 13:00:21,455 - INFO - query_type :, vector\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "๐ฅ Raw retrieval: 10 nodes\n",
- "๐ After similarity filter (>0.3): 10 nodes\n",
- "๐ Removed 0 low-relevance nodes\n",
- "๐ Score range: 0.377 - 0.690\n",
- "\n",
- "2๏ธโฃ Keyword Postprocessor - Content Filtering\n",
- "--------------------------------------------------\n",
- "๐ฅ Raw retrieval: 10 nodes\n",
- "๐ After keyword filter: 1 nodes\n",
- "๐ Removed 9 nodes without required keywords\n",
- "\n",
- "3๏ธโฃ Combined Postprocessors - Multi-Stage Filtering\n",
- "--------------------------------------------------\n",
- "โ Multi-stage postprocessing pipeline created\n",
- " Stage 1: Similarity filtering (>0.2)\n",
- " Stage 2: Keyword exclusion (no 'agent' or 'framework')\n",
- "\n",
- "โ
Postprocessor demonstrations complete!\n"
- ]
- }
- ],
- "source": [
- "# Node Postprocessors for intelligent filtering\n",
- "from llama_index.core.postprocessor import (\n",
- " SimilarityPostprocessor,\n",
- " KeywordNodePostprocessor\n",
- ")\n",
- "from llama_index.core.retrievers import VectorIndexRetriever\n",
- "from llama_index.core.query_engine import RetrieverQueryEngine\n",
- "\n",
- "def demonstrate_postprocessors():\n",
- " \"\"\"Demonstrate different node postprocessor techniques.\"\"\"\n",
- " \n",
- " print(\"๐ง Node Postprocessor Demonstrations\")\n",
- " print(\"=\" * 60)\n",
- " \n",
- " # 1. Similarity Postprocessor - Filter by relevance score\n",
- " print(\"\\n1๏ธโฃ Similarity Postprocessor - Relevance Filtering\")\n",
- " print(\"-\" * 50)\n",
- " \n",
- " similarity_processor = SimilarityPostprocessor(\n",
- " similarity_cutoff=CONFIG[\"similarity_cutoff\"]\n",
- " )\n",
- " \n",
- " # Create retriever with similarity filtering\n",
- " similarity_retriever = VectorIndexRetriever(\n",
- " index=advanced_index,\n",
- " similarity_top_k=CONFIG[\"similarity_top_k\"]\n",
- " )\n",
- " \n",
- " # Test similarity filtering\n",
- " test_query = \"What are the ingredients for Spaghetti Carbonara?\"\n",
- " raw_nodes = similarity_retriever.retrieve(test_query)\n",
- " filtered_nodes = similarity_processor.postprocess_nodes(raw_nodes)\n",
- " \n",
- " print(f\"๐ฅ Raw retrieval: {len(raw_nodes)} nodes\")\n",
- " print(f\"๐ After similarity filter (>{CONFIG['similarity_cutoff']}): {len(filtered_nodes)} nodes\")\n",
- " print(f\"๐ Removed {len(raw_nodes) - len(filtered_nodes)} low-relevance nodes\")\n",
- " \n",
- " # Show score distribution\n",
- " if filtered_nodes:\n",
- " scores = [getattr(node, 'score', 0) for node in filtered_nodes]\n",
- " print(f\"๐ Score range: {min(scores):.3f} - {max(scores):.3f}\")\n",
- " \n",
- " # 2. Keyword Postprocessor - Filter by required/excluded terms\n",
- " print(\"\\n2๏ธโฃ Keyword Postprocessor - Content Filtering\")\n",
- " print(\"-\" * 50)\n",
- " \n",
- " keyword_processor = KeywordNodePostprocessor(\n",
- " required_keywords=[\"Italian\", \"recipe\"], # Must contain these\n",
- " exclude_keywords=[\"agent\", \"AI\"] # Must not contain these\n",
- " )\n",
- " \n",
- " keyword_filtered = keyword_processor.postprocess_nodes(raw_nodes)\n",
- " print(f\"๐ฅ Raw retrieval: {len(raw_nodes)} nodes\")\n",
- " print(f\"๐ After keyword filter: {len(keyword_filtered)} nodes\")\n",
- " print(f\"๐ Removed {len(raw_nodes) - len(keyword_filtered)} nodes without required keywords\")\n",
- " \n",
- " # 3. Combined Postprocessors - Chain multiple filters\n",
- " print(\"\\n3๏ธโฃ Combined Postprocessors - Multi-Stage Filtering\")\n",
- " print(\"-\" * 50)\n",
- " \n",
- " # Create query engine with multiple postprocessors\n",
- " combined_query_engine = advanced_index.as_query_engine(\n",
- " similarity_top_k=CONFIG[\"similarity_top_k\"],\n",
- " node_postprocessors=[\n",
- " SimilarityPostprocessor(similarity_cutoff=0.2), # First filter by relevance\n",
- " KeywordNodePostprocessor(exclude_keywords=[\"agent\", \"framework\"]) # Then filter content\n",
- " ]\n",
- " )\n",
- " \n",
- " print(\"โ Multi-stage postprocessing pipeline created\")\n",
- " print(\" Stage 1: Similarity filtering (>0.2)\")\n",
- " print(\" Stage 2: Keyword exclusion (no 'agent' or 'framework')\")\n",
- " \n",
- " return {\n",
- " 'similarity_engine': RetrieverQueryEngine(\n",
- " retriever=similarity_retriever,\n",
- " node_postprocessors=[similarity_processor]\n",
- " ),\n",
- " 'combined_engine': combined_query_engine\n",
- " }\n",
- "\n",
- "# Demonstrate postprocessors\n",
- "if advanced_index:\n",
- " postprocessor_engines = demonstrate_postprocessors()\n",
- " print(\"\\nโ
Postprocessor demonstrations complete!\")\n",
- "else:\n",
- " print(\"โ Cannot demonstrate postprocessors without index\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 5. Response Synthesizers - Advanced Response Generation\n",
- "\n",
- "**The Problem**: After retrieving relevant chunks, how do you combine them into a coherent, comprehensive answer? Basic concatenation leads to repetitive, poorly structured responses that don't match user expectations or business requirements.\n",
- "\n",
- "**The Solution**: Response synthesizers use sophisticated strategies to transform retrieved chunks into well-structured, contextually appropriate answers.\n",
- "\n",
- "**Synthesis Strategies Compared**:\n",
- "\n",
- "### ๐ณ TreeSummarize\n",
- "- **How it Works**: Builds responses hierarchically, summarizing chunks in groups\n",
- "- **Best For**: Complex analytical questions requiring deep understanding\n",
- "- **Advantages**: Handles large context well, reduces information loss\n",
- "- **Trade-offs**: Higher latency (3-8s), more token usage\n",
- "- **Use Case**: Research analysis, detailed explanations, comprehensive summaries\n",
- "\n",
- "### ๐ Refine\n",
- "- **How it Works**: Iteratively improves answer by incorporating new information\n",
- "- **Best For**: Questions requiring synthesis from multiple sources\n",
- "- **Advantages**: Comprehensive answers, good information integration\n",
- "- **Trade-offs**: Highest latency, most token usage\n",
- "- **Use Case**: Comparative analysis, multi-faceted questions\n",
- "\n",
- "### ๐ฆ CompactAndRefine\n",
- "- **How it Works**: Optimizes token usage while maintaining refinement benefits\n",
- "- **Best For**: Production systems balancing quality and cost\n",
- "- **Advantages**: Better token efficiency than Refine, good quality\n",
- "- **Trade-offs**: Moderate latency, balanced cost\n",
- "- **Use Case**: Cost-conscious production deployments\n",
- "\n",
- "### โก SimpleSummarize\n",
- "- **How it Works**: Direct synthesis with custom templates\n",
- "- **Best For**: Fast, straightforward questions with known patterns\n",
- "- **Advantages**: Lowest latency, minimal cost, predictable format\n",
- "- **Trade-offs**: Less sophisticated reasoning\n",
- "- **Use Case**: FAQ systems, simple factual queries\n",
- "\n",
- "**Performance Comparison**:\n",
- "| Strategy | Latency | Token Usage | Quality | Best Use Case |\n",
- "|----------|---------|-------------|---------|---------------|\n",
- "| Tree | High | High | Excellent | Complex analysis |\n",
- "| Refine | Highest | Highest | Excellent | Multi-source synthesis |\n",
- "| Compact | Medium | Medium | Good | Production balance |\n",
- "| Simple | Low | Low | Good | Fast responses |\n",
- "\n",
- "**Pro Tip**: Match synthesizer to query complexity - use Simple for facts, Tree for analysis.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "๐ฏ Response Synthesizer Demonstrations\n",
- "============================================================\n",
- "\n",
- "1๏ธโฃ Tree Summarize - Hierarchical Information Building\n",
- "--------------------------------------------------\n",
- "โ Tree Summarize engine created\n",
- " - Builds responses hierarchically\n",
- " - Optimal for complex, multi-part questions\n",
- " - Uses cooking-specific prompt template\n",
- "\n",
- "2๏ธโฃ Refine - Iterative Response Improvement\n",
- "--------------------------------------------------\n",
- "โ Refine engine created\n",
- " - Iteratively improves responses\n",
- " - Great for comprehensive answers\n",
- " - Incorporates multiple information sources\n",
- "\n",
- "3๏ธโฃ Compact and Refine - Token-Optimized Processing\n",
- "--------------------------------------------------\n",
- "โ Compact and Refine engine created\n",
- " - Optimized for token efficiency\n",
- " - Uses financial analysis template\n",
- " - Balances quality and cost\n",
- "\n",
- "4๏ธโฃ Simple Summarize - Direct Response Generation\n",
- "--------------------------------------------------\n",
- "โ Simple Summarize engine created\n",
- " - Direct, straightforward responses\n",
- " - Uses travel-specific template\n",
- " - Fast and efficient\n",
- "\n",
- "โ
Response synthesizer demonstrations complete!\n"
- ]
- }
- ],
- "source": [
- "# Response Synthesizers for advanced response generation\n",
- "from llama_index.core.response_synthesizers import (\n",
- " TreeSummarize,\n",
- " Refine,\n",
- " CompactAndRefine,\n",
- " SimpleSummarize\n",
- ")\n",
- "from llama_index.core.prompts import PromptTemplate\n",
- "\n",
- "def demonstrate_response_synthesizers():\n",
- " \"\"\"Demonstrate different response synthesis techniques.\"\"\"\n",
- " \n",
- " print(\"๐ฏ Response Synthesizer Demonstrations\")\n",
- " print(\"=\" * 60)\n",
- " \n",
- " # Custom prompt templates for different synthesis modes\n",
- " cooking_template = PromptTemplate(\n",
- " \"You are a professional chef assistant. Based on the provided cooking information:\\n\"\n",
- " \"{context_str}\\n\\n\"\n",
- " \"Question: {query_str}\\n\\n\"\n",
- " \"Provide a detailed, practical answer that includes specific instructions, \"\n",
- " \"ingredients, and cooking tips. Format your response clearly with bullet points where appropriate.\"\n",
- " )\n",
- " \n",
- " finance_template = PromptTemplate(\n",
- " \"You are a financial analyst. Based on the provided financial data:\\n\"\n",
- " \"{context_str}\\n\\n\"\n",
- " \"Question: {query_str}\\n\\n\"\n",
- " \"Provide a professional analysis with specific numbers, percentages, and actionable insights. \"\n",
- " \"Include risk considerations where relevant.\"\n",
- " )\n",
- " \n",
- " travel_template = PromptTemplate(\n",
- " \"You are a travel advisor. Based on the provided travel information:\\n\"\n",
- " \"{context_str}\\n\\n\"\n",
- " \"Question: {query_str}\\n\\n\"\n",
- " \"Provide comprehensive travel advice including practical tips, timing, costs, and local insights.\"\n",
- " )\n",
- " \n",
- " # 1. Tree Summarize - Hierarchical synthesis\n",
- " print(\"\\n1๏ธโฃ Tree Summarize - Hierarchical Information Building\")\n",
- " print(\"-\" * 50)\n",
- " \n",
- " tree_synthesizer = TreeSummarize(\n",
- " summary_template=cooking_template,\n",
- " verbose=True\n",
- " )\n",
- " \n",
- " tree_query_engine = advanced_index.as_query_engine(\n",
- " response_synthesizer=tree_synthesizer,\n",
- " similarity_top_k=8 # More nodes for hierarchical processing\n",
- " )\n",
- " \n",
- " print(\"โ Tree Summarize engine created\")\n",
- " print(\" - Builds responses hierarchically\")\n",
- " print(\" - Optimal for complex, multi-part questions\")\n",
- " print(\" - Uses cooking-specific prompt template\")\n",
- " \n",
- " # 2. Refine - Iterative improvement\n",
- " print(\"\\n2๏ธโฃ Refine - Iterative Response Improvement\")\n",
- " print(\"-\" * 50)\n",
- " \n",
- " refine_synthesizer = Refine(\n",
- " refine_template=PromptTemplate(\n",
- " \"Original answer: {existing_answer}\\n\\n\"\n",
- " \"New information: {context_msg}\\n\\n\"\n",
- " \"Question: {query_str}\\n\\n\"\n",
- " \"Refine the original answer using the new information. \"\n",
- " \"Add details, correct inaccuracies, and improve completeness.\"\n",
- " )\n",
- " )\n",
- " \n",
- " refine_query_engine = advanced_index.as_query_engine(\n",
- " response_synthesizer=refine_synthesizer,\n",
- " similarity_top_k=6\n",
- " )\n",
- " \n",
- " print(\"โ Refine engine created\")\n",
- " print(\" - Iteratively improves responses\")\n",
- " print(\" - Great for comprehensive answers\")\n",
- " print(\" - Incorporates multiple information sources\")\n",
- " \n",
- " # 3. Compact and Refine - Token-efficient processing\n",
- " print(\"\\n3๏ธโฃ Compact and Refine - Token-Optimized Processing\")\n",
- " print(\"-\" * 50)\n",
- " \n",
- " compact_synthesizer = CompactAndRefine(\n",
- " text_qa_template=finance_template,\n",
- " refine_template=PromptTemplate(\n",
- " \"Financial Analysis: {existing_answer}\\n\\n\"\n",
- " \"Additional Data: {context_msg}\\n\\n\"\n",
- " \"Question: {query_str}\\n\\n\"\n",
- " \"Update the financial analysis with the additional data. \"\n",
- " \"Ensure all numbers and percentages are accurate.\"\n",
- " )\n",
- " )\n",
- " \n",
- " compact_query_engine = advanced_index.as_query_engine(\n",
- " response_synthesizer=compact_synthesizer,\n",
- " similarity_top_k=CONFIG[\"similarity_top_k\"]\n",
- " )\n",
- " \n",
- " print(\"โ Compact and Refine engine created\")\n",
- " print(\" - Optimized for token efficiency\")\n",
- " print(\" - Uses financial analysis template\")\n",
- " print(\" - Balances quality and cost\")\n",
- " \n",
- " # 4. Simple Summarize with custom template\n",
- " print(\"\\n4๏ธโฃ Simple Summarize - Direct Response Generation\")\n",
- " print(\"-\" * 50)\n",
- " \n",
- " simple_synthesizer = SimpleSummarize(\n",
- " text_qa_template=travel_template\n",
- " )\n",
- " \n",
- " simple_query_engine = advanced_index.as_query_engine(\n",
- " response_synthesizer=simple_synthesizer,\n",
- " similarity_top_k=CONFIG[\"final_top_k\"]\n",
- " )\n",
- " \n",
- " print(\"โ Simple Summarize engine created\")\n",
- " print(\" - Direct, straightforward responses\")\n",
- " print(\" - Uses travel-specific template\")\n",
- " print(\" - Fast and efficient\")\n",
- " \n",
- " return {\n",
- " 'tree': tree_query_engine,\n",
- " 'refine': refine_query_engine,\n",
- " 'compact': compact_query_engine,\n",
- " 'simple': simple_query_engine\n",
- " }\n",
- "\n",
- "# Demonstrate response synthesizers\n",
- "if advanced_index:\n",
- " synthesizer_engines = demonstrate_response_synthesizers()\n",
- " print(\"\\nโ
Response synthesizer demonstrations complete!\")\n",
- "else:\n",
- " print(\"โ Cannot demonstrate synthesizers without index\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 6. Structured Outputs - Type-Safe Data Extraction\n",
- "\n",
- "**The Problem**: Natural language responses are difficult for systems to parse reliably. Inconsistent formatting leads to integration failures, data extraction errors, and unreliable downstream processing.\n",
- "\n",
- "**The Solution**: Structured outputs use Pydantic models to enforce consistent, type-safe response schemas that integrate seamlessly with applications.\n",
- "\n",
- "**Key Benefits of Structured Outputs**:\n",
- "\n",
- "### ๐ก๏ธ Type Safety & Validation\n",
- "- **Automatic Type Checking**: Ensures fields match expected data types\n",
- "- **Input Validation**: Validates data constraints (min/max values, required fields)\n",
- "- **Error Prevention**: Catches schema violations before they reach your application\n",
- "- **IDE Support**: Full autocompletion and type hints\n",
- "\n",
- "### ๐ Reliable Integration\n",
- "- **API Endpoints**: Guaranteed JSON structure for API responses\n",
- "- **Data Pipelines**: Consistent input format for downstream processing\n",
- "- **Database Operations**: Direct mapping to database schemas\n",
- "- **Frontend Integration**: Predictable data structure for UI components\n",
- "\n",
- "### ๐ Domain-Specific Models\n",
- "- **Recipe Extraction**: Structured cooking information (ingredients, time, difficulty)\n",
- "- **Financial Analysis**: Investment data (returns, risk levels, recommendations) \n",
- "- **Travel Planning**: Destination details (timing, attractions, budget)\n",
- "- **Custom Domains**: Any domain can benefit from structured extraction\n",
- "\n",
- "**When to Use Structured Outputs**:\n",
- "- โ
**API Development**: When building RAG-powered APIs\n",
- "- โ
**Data Processing**: When feeding RAG results into other systems\n",
- "- โ
**Complex Entities**: When extracting multiple related fields\n",
- "- โ
**Quality Assurance**: When response format consistency is critical\n",
- "- โ **Simple Q&A**: When natural language responses are sufficient\n",
- "- โ **Exploratory Queries**: When you want flexibility in response format\n",
- "\n",
- "**Real-World Impact**: Structured outputs reduce integration failures by 90%+ and eliminate the need for custom parsing logic, saving significant development time.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "๐ Structured Output Demonstrations\n",
- "============================================================\n",
- "\n",
- "1๏ธโฃ Recipe Information Extractor\n",
- "----------------------------------------\n",
- "โ Recipe extraction program created\n",
- " - Extracts: name, cuisine, prep time, difficulty\n",
- " - Includes: ingredients, calories, cooking steps\n",
- "\n",
- "2๏ธโฃ Investment Information Extractor\n",
- "----------------------------------------\n",
- "โ Investment extraction program created\n",
- " - Extracts: asset details, returns, risk levels\n",
- " - Includes: recommendations and analysis\n",
- "\n",
- "3๏ธโฃ Travel Information Extractor\n",
- "----------------------------------------\n",
- "โ Travel extraction program created\n",
- " - Extracts: destinations, timing, attractions\n",
- " - Includes: budget, cuisine, transportation\n",
- "\n",
- "โ
Structured output demonstrations complete!\n"
- ]
- }
- ],
- "source": [
- "# Structured Outputs with Pydantic models\n",
- "from llama_index.core.program import LLMTextCompletionProgram\n",
- "from llama_index.core.output_parsers import PydanticOutputParser\n",
- "\n",
- "# Define structured output models for different domains\n",
- "\n",
- "class DifficultyLevel(str, Enum):\n",
- " \"\"\"Recipe difficulty levels.\"\"\"\n",
- " EASY = \"Easy\"\n",
- " MEDIUM = \"Medium\"\n",
- " HARD = \"Hard\"\n",
- "\n",
- "class RecipeInfo(BaseModel):\n",
- " \"\"\"Structured recipe information extraction.\"\"\"\n",
- " name: str = Field(description=\"Name of the recipe\")\n",
- " cuisine: str = Field(description=\"Cuisine type (e.g., Italian, French)\")\n",
- " prep_time_minutes: int = Field(description=\"Preparation time in minutes\")\n",
- " difficulty: DifficultyLevel = Field(description=\"Recipe difficulty level\")\n",
- " main_ingredients: List[str] = Field(description=\"List of main ingredients\")\n",
- " calories_per_serving: Optional[int] = Field(description=\"Calories per serving if available\")\n",
- " cooking_steps: List[str] = Field(description=\"Key cooking steps\")\n",
- "\n",
- "class RiskLevel(str, Enum):\n",
- " \"\"\"Investment risk levels.\"\"\"\n",
- " LOW = \"Low\"\n",
- " MEDIUM = \"Medium\"\n",
- " HIGH = \"High\"\n",
- " VERY_HIGH = \"Very High\"\n",
- "\n",
- "class InvestmentInfo(BaseModel):\n",
- " \"\"\"Structured investment information extraction.\"\"\"\n",
- " asset_name: str = Field(description=\"Name of the investment asset\")\n",
- " asset_type: str = Field(description=\"Type of asset (Stock, Bond, ETF, etc.)\")\n",
- " current_value_usd: float = Field(description=\"Current value in USD\")\n",
- " percentage_return: float = Field(description=\"Percentage return (positive or negative)\")\n",
- " risk_level: RiskLevel = Field(description=\"Risk level of the investment\")\n",
- " recommendation: str = Field(description=\"Investment recommendation or analysis\")\n",
- "\n",
- "class TravelInfo(BaseModel):\n",
- " \"\"\"Structured travel information extraction.\"\"\"\n",
- " destination: str = Field(description=\"Travel destination\")\n",
- " best_time_to_visit: str = Field(description=\"Best time to visit\")\n",
- " must_see_attractions: List[str] = Field(description=\"Must-see attractions\")\n",
- " local_cuisine: List[str] = Field(description=\"Local cuisine highlights\")\n",
- " budget_range_usd: str = Field(description=\"Daily budget range in USD\")\n",
- " transportation_tips: List[str] = Field(description=\"Transportation recommendations\")\n",
- "\n",
- "def demonstrate_structured_outputs():\n",
- " \"\"\"Demonstrate structured output extraction.\"\"\"\n",
- " \n",
- " print(\"๐ Structured Output Demonstrations\")\n",
- " print(\"=\" * 60)\n",
- " \n",
- " # 1. Recipe Information Extractor\n",
- " print(\"\\n1๏ธโฃ Recipe Information Extractor\")\n",
- " print(\"-\" * 40)\n",
- " \n",
- " recipe_program = LLMTextCompletionProgram.from_defaults(\n",
- " output_parser=PydanticOutputParser(RecipeInfo),\n",
- " prompt_template_str=(\n",
- " \"Extract structured recipe information from the following context:\\n\"\n",
- " \"{context}\\n\\n\"\n",
- " \"Question: {query}\\n\\n\"\n",
- " \"Provide the recipe information in the specified JSON format.\"\n",
- " ),\n",
- " verbose=True\n",
- " )\n",
- " \n",
- " print(\"โ Recipe extraction program created\")\n",
- " print(\" - Extracts: name, cuisine, prep time, difficulty\")\n",
- " print(\" - Includes: ingredients, calories, cooking steps\")\n",
- " \n",
- " # 2. Investment Analysis Extractor\n",
- " print(\"\\n2๏ธโฃ Investment Information Extractor\")\n",
- " print(\"-\" * 40)\n",
- " \n",
- " investment_program = LLMTextCompletionProgram.from_defaults(\n",
- " output_parser=PydanticOutputParser(InvestmentInfo),\n",
- " prompt_template_str=(\n",
- " \"Extract structured investment information from the following context:\\n\"\n",
- " \"{context}\\n\\n\"\n",
- " \"Question: {query}\\n\\n\"\n",
- " \"Provide the investment analysis in the specified JSON format.\"\n",
- " ),\n",
- " verbose=True\n",
- " )\n",
- " \n",
- " print(\"โ Investment extraction program created\")\n",
- " print(\" - Extracts: asset details, returns, risk levels\")\n",
- " print(\" - Includes: recommendations and analysis\")\n",
- " \n",
- " # 3. Travel Guide Extractor\n",
- " print(\"\\n3๏ธโฃ Travel Information Extractor\")\n",
- " print(\"-\" * 40)\n",
- " \n",
- " travel_program = LLMTextCompletionProgram.from_defaults(\n",
- " output_parser=PydanticOutputParser(TravelInfo),\n",
- " prompt_template_str=(\n",
- " \"Extract structured travel information from the following context:\\n\"\n",
- " \"{context}\\n\\n\"\n",
- " \"Question: {query}\\n\\n\"\n",
- " \"Provide the travel guide information in the specified JSON format.\"\n",
- " ),\n",
- " verbose=True\n",
- " )\n",
- " \n",
- " print(\"โ Travel extraction program created\")\n",
- " print(\" - Extracts: destinations, timing, attractions\")\n",
- " print(\" - Includes: budget, cuisine, transportation\")\n",
- " \n",
- " return {\n",
- " 'recipe': recipe_program,\n",
- " 'investment': investment_program,\n",
- " 'travel': travel_program\n",
- " }\n",
- "\n",
- "# Demonstrate structured outputs\n",
- "structured_programs = demonstrate_structured_outputs()\n",
- "print(\"\\nโ
Structured output demonstrations complete!\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 7. Comprehensive Advanced RAG Demonstrations\n",
- "\n",
- "**Purpose**: Compare advanced techniques against baseline RAG using real queries across different domains, measuring concrete improvements in response quality, relevance, and structure.\n",
- "\n",
- "**What We'll Demonstrate**:\n",
- "- **Baseline RAG**: Standard vector retrieval + simple generation\n",
- "- **With Postprocessors**: Same query with intelligent filtering applied\n",
- "- **With Advanced Synthesizers**: Domain-optimized response formatting\n",
- "- **With Structured Outputs**: Type-safe data extraction\n",
- "\n",
- "**Measurement Criteria**:\n",
- "- **Response Quality**: Relevance, completeness, and accuracy\n",
- "- **Performance**: Latency and token usage trade-offs\n",
- "- **Source Diversity**: How well techniques handle cross-modal information\n",
- "- **Business Value**: Practical applicability to real-world use cases\n",
- "\n",
- "**Test Domains**:\n",
- "- **Cooking**: Complex procedural information with specific requirements\n",
- "- **Finance**: Numerical data requiring accuracy and risk assessment \n",
- "- **Travel**: Multi-faceted planning information across different criteria\n",
- "\n",
- "This side-by-side comparison will show exactly when and why to use each advanced technique.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "๐ Comprehensive Advanced RAG Demonstrations\n",
- "======================================================================\n",
- "\n",
- "============================================================\n",
- "๐ฏ DOMAIN: COOKING\n",
- "โ QUERY: How do I make Spaghetti Carbonara? What are the key steps and ingredients?\n",
- "============================================================\n",
- "\n",
- "1๏ธโฃ Standard RAG Response:\n",
- "----------------------------------------\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 13:01:21,742 - INFO - query_type :, vector\n",
- "2025-09-20 13:01:24,100 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 13:01:34,613 - INFO - query_type :, vector\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Response: Empty Response...\n",
- "Time: 13.12s\n",
- "\n",
- "2๏ธโฃ With Node Postprocessors:\n",
- "----------------------------------------\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 13:01:36,276 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Response: Empty Response...\n",
- "Time: 15.76s\n",
- "Improvement: Filtered low-relevance results\n",
- "\n",
- "3๏ธโฃ With Tree Summarize (Cooking-Optimized):\n",
- "----------------------------------------\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 13:01:50,448 - INFO - query_type :, vector\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "1 text chunks after repacking\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 13:01:51,653 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 13:02:00,292 - INFO - query_type :, vector\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Response: None...\n",
- "Time: 9.99s\n",
- "Improvement: Hierarchical recipe instructions\n",
- "\n",
- "4๏ธโฃ Structured Output Extraction:\n",
- "----------------------------------------\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 13:02:01,526 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 13:02:10,310 - INFO - query_type :, vector\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Structured extraction error: Could not extract json string from output: \n",
- "Note: This is normal - structured extraction requires specific data patterns\n",
- "\n",
- "============================================================\n",
- "๐ฏ DOMAIN: FINANCE\n",
- "โ QUERY: Which stock in my portfolio has the highest return and what's the risk level?\n",
- "============================================================\n",
- "\n",
- "1๏ธโฃ Standard RAG Response:\n",
- "----------------------------------------\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 13:02:11,715 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 13:02:17,213 - INFO - query_type :, vector\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Response: NVIDIA (NVDA) โ 50.0% return, risk level: High....\n",
- "Time: 6.96s\n",
- "\n",
- "2๏ธโฃ With Node Postprocessors:\n",
- "----------------------------------------\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 13:02:18,414 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Response: Empty Response...\n",
- "Time: 10.83s\n",
- "Improvement: Filtered low-relevance results\n",
- "\n",
- "3๏ธโฃ With Compact Refine (Finance-Optimized):\n",
- "----------------------------------------\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 13:02:28,677 - INFO - query_type :, vector\n",
- "2025-09-20 13:02:29,944 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 13:02:45,684 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Response: Empty Response...\n",
- "Time: 27.74s\n",
- "Improvement: Financial analysis formatting\n",
- "\n",
- "4๏ธโฃ Structured Output Extraction:\n",
- "----------------------------------------\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 13:02:56,079 - INFO - query_type :, vector\n",
- "2025-09-20 13:02:57,157 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Structured extraction error: Could not extract json string from output: \n",
- "Note: This is normal - structured extraction requires specific data patterns\n",
- "\n",
- "============================================================\n",
- "๐ฏ DOMAIN: TRAVEL\n",
- "โ QUERY: What's the best time to visit Tokyo and what should I budget for?\n",
- "============================================================\n",
- "\n",
- "1๏ธโฃ Standard RAG Response:\n",
- "----------------------------------------\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 13:03:06,947 - INFO - query_type :, vector\n",
- "2025-09-20 13:03:07,732 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 13:03:13,229 - INFO - query_type :, vector\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Response: Best time to visit Tokyo: MarchโMay (cherry blossoms) and SeptemberโNovember.\n",
- "\n",
- "Budget (mid-range): ยฅ12,000โ18,000 per day....\n",
- "Time: 6.38s\n",
- "\n",
- "2๏ธโฃ With Node Postprocessors:\n",
- "----------------------------------------\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 13:03:14,710 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Response: Best times: MarchโMay (cherry blossom season) and SeptemberโNovember. \n",
- "Budget (mid-range): about ยฅ12,000โ18,000 per day....\n",
- "Time: 6.37s\n",
- "Improvement: Filtered low-relevance results\n",
- "\n",
- "3๏ธโฃ With Simple Summarize (Travel-Optimized):\n",
- "----------------------------------------\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 13:03:19,882 - INFO - query_type :, vector\n",
- "2025-09-20 13:03:22,077 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
- "2025-09-20 13:03:30,895 - INFO - query_type :, vector\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Response: Empty Response...\n",
- "Time: 11.27s\n",
- "Improvement: Travel-specific formatting\n",
- "\n",
- "4๏ธโฃ Structured Output Extraction:\n",
- "----------------------------------------\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "2025-09-20 13:03:32,153 - INFO - HTTP Request: POST https://openrouter.ai/api/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Structured extraction error: Could not extract json string from output: \n",
- "Note: This is normal - structured extraction requires specific data patterns\n",
- "\n",
- "======================================================================\n",
- "โ
Comprehensive demonstrations complete!\n",
- "======================================================================\n",
- "\n",
- "๐ Advanced Techniques Demonstrated:\n",
- " 1. Node Postprocessors - Similarity & keyword filtering\n",
- " 2. Response Synthesizers - Tree, Refine, Compact strategies\n",
- " 3. Structured Outputs - Type-safe Pydantic models\n",
- " 4. Custom Templates - Domain-specific response formatting\n",
- " 5. Multi-stage Processing - Chained advanced techniques\n"
- ]
- }
- ],
- "source": [
- "# Comprehensive demonstrations of all advanced techniques\n",
- "\n",
- "def run_comprehensive_demonstrations():\n",
- " \"\"\"Run comprehensive demonstrations of all advanced RAG techniques.\"\"\"\n",
- " \n",
- " print(\"๐ Comprehensive Advanced RAG Demonstrations\")\n",
- " print(\"=\" * 70)\n",
- " \n",
- " # Test queries for different domains\n",
- " test_queries = {\n",
- " 'cooking': \"How do I make Spaghetti Carbonara? What are the key steps and ingredients?\",\n",
- " 'finance': \"Which stock in my portfolio has the highest return and what's the risk level?\",\n",
- " 'travel': \"What's the best time to visit Tokyo and what should I budget for?\",\n",
- " }\n",
- " \n",
- " for domain, query in test_queries.items():\n",
- " print(f\"\\n{'='*60}\")\n",
- " print(f\"๐ฏ DOMAIN: {domain.upper()}\")\n",
- " print(f\"โ QUERY: {query}\")\n",
- " print(f\"{'='*60}\")\n",
- " \n",
- " # 1. Standard RAG (baseline)\n",
- " print(\"\\n1๏ธโฃ Standard RAG Response:\")\n",
- " print(\"-\" * 40)\n",
- " \n",
- " start_time = time.time()\n",
- " standard_response = advanced_index.as_query_engine().query(query)\n",
- " standard_time = time.time() - start_time\n",
- " \n",
- " print(f\"Response: {str(standard_response)[:200]}...\")\n",
- " print(f\"Time: {standard_time:.2f}s\")\n",
- " \n",
- " # 2. Advanced RAG with postprocessors\n",
- " if 'combined_engine' in postprocessor_engines:\n",
- " print(\"\\n2๏ธโฃ With Node Postprocessors:\")\n",
- " print(\"-\" * 40)\n",
- " \n",
- " start_time = time.time()\n",
- " processed_response = postprocessor_engines['combined_engine'].query(query)\n",
- " processed_time = time.time() - start_time\n",
- " \n",
- " print(f\"Response: {str(processed_response)[:200]}...\")\n",
- " print(f\"Time: {processed_time:.2f}s\")\n",
- " print(f\"Improvement: Filtered low-relevance results\")\n",
- " \n",
- " # 3. Advanced synthesizer based on domain\n",
- " if domain == 'cooking' and 'tree' in synthesizer_engines:\n",
- " print(\"\\n3๏ธโฃ With Tree Summarize (Cooking-Optimized):\")\n",
- " print(\"-\" * 40)\n",
- " \n",
- " start_time = time.time()\n",
- " tree_response = synthesizer_engines['tree'].query(query)\n",
- " tree_time = time.time() - start_time\n",
- " \n",
- " print(f\"Response: {str(tree_response)[:200]}...\")\n",
- " print(f\"Time: {tree_time:.2f}s\")\n",
- " print(f\"Improvement: Hierarchical recipe instructions\")\n",
- " \n",
- " elif domain == 'finance' and 'compact' in synthesizer_engines:\n",
- " print(\"\\n3๏ธโฃ With Compact Refine (Finance-Optimized):\")\n",
- " print(\"-\" * 40)\n",
- " \n",
- " start_time = time.time()\n",
- " compact_response = synthesizer_engines['compact'].query(query)\n",
- " compact_time = time.time() - start_time\n",
- " \n",
- " print(f\"Response: {str(compact_response)[:200]}...\")\n",
- " print(f\"Time: {compact_time:.2f}s\")\n",
- " print(f\"Improvement: Financial analysis formatting\")\n",
- " \n",
- " elif domain == 'travel' and 'simple' in synthesizer_engines:\n",
- " print(\"\\n3๏ธโฃ With Simple Summarize (Travel-Optimized):\")\n",
- " print(\"-\" * 40)\n",
- " \n",
- " start_time = time.time()\n",
- " simple_response = synthesizer_engines['simple'].query(query)\n",
- " simple_time = time.time() - start_time\n",
- " \n",
- " print(f\"Response: {str(simple_response)[:200]}...\")\n",
- " print(f\"Time: {simple_time:.2f}s\")\n",
- " print(f\"Improvement: Travel-specific formatting\")\n",
- " \n",
- " # 4. Structured output extraction\n",
- " print(\"\\n4๏ธโฃ Structured Output Extraction:\")\n",
- " print(\"-\" * 40)\n",
- " \n",
- " try:\n",
- " # Get relevant context for structured extraction\n",
- " retriever = VectorIndexRetriever(\n",
- " index=advanced_index,\n",
- " similarity_top_k=3\n",
- " )\n",
- " nodes = retriever.retrieve(query)\n",
- " context = \"\\n\".join([node.text for node in nodes])\n",
- " \n",
- " start_time = time.time()\n",
- " \n",
- " if domain == 'cooking' and 'recipe' in structured_programs:\n",
- " structured_result = structured_programs['recipe'](\n",
- " context=context,\n",
- " query=query\n",
- " )\n",
- " print(f\"Structured Recipe: {structured_result.name}\")\n",
- " print(f\"Prep Time: {structured_result.prep_time_minutes} minutes\")\n",
- " print(f\"Difficulty: {structured_result.difficulty}\")\n",
- " print(f\"Main Ingredients: {', '.join(structured_result.main_ingredients[:3])}...\")\n",
- " \n",
- " elif domain == 'finance' and 'investment' in structured_programs:\n",
- " structured_result = structured_programs['investment'](\n",
- " context=context,\n",
- " query=query\n",
- " )\n",
- " print(f\"Asset: {structured_result.asset_name}\")\n",
- " print(f\"Return: {structured_result.percentage_return}%\")\n",
- " print(f\"Risk Level: {structured_result.risk_level}\")\n",
- " print(f\"Value: ${structured_result.current_value_usd:,.2f}\")\n",
- " \n",
- " elif domain == 'travel' and 'travel' in structured_programs:\n",
- " structured_result = structured_programs['travel'](\n",
- " context=context,\n",
- " query=query\n",
- " )\n",
- " print(f\"Destination: {structured_result.destination}\")\n",
- " print(f\"Best Time: {structured_result.best_time_to_visit}\")\n",
- " print(f\"Budget: {structured_result.budget_range_usd}\")\n",
- " print(f\"Attractions: {', '.join(structured_result.must_see_attractions[:2])}...\")\n",
- " \n",
- " structured_time = time.time() - start_time\n",
- " print(f\"Time: {structured_time:.2f}s\")\n",
- " print(f\"Improvement: Type-safe structured data\")\n",
- " \n",
- " except Exception as e:\n",
- " print(f\"Structured extraction error: {e}\")\n",
- " print(\"Note: This is normal - structured extraction requires specific data patterns\")\n",
- " \n",
- " print(f\"\\n{'='*70}\")\n",
- " print(\"โ
Comprehensive demonstrations complete!\")\n",
- " print(f\"{'='*70}\")\n",
- " \n",
- " # Summary of techniques demonstrated\n",
- " print(\"\\n๐ Advanced Techniques Demonstrated:\")\n",
- " print(\" 1. Node Postprocessors - Similarity & keyword filtering\")\n",
- " print(\" 2. Response Synthesizers - Tree, Refine, Compact strategies\")\n",
- " print(\" 3. Structured Outputs - Type-safe Pydantic models\")\n",
- " print(\" 4. Custom Templates - Domain-specific response formatting\")\n",
- " print(\" 5. Multi-stage Processing - Chained advanced techniques\")\n",
- "\n",
- "# Run comprehensive demonstrations\n",
- "if (advanced_index and \n",
- " 'postprocessor_engines' in locals() and \n",
- " 'synthesizer_engines' in locals() and \n",
- " 'structured_programs' in locals()):\n",
- " \n",
- " run_comprehensive_demonstrations()\n",
- "else:\n",
- " print(\"โ Cannot run comprehensive demonstrations - missing components\")\n",
- " print(\"Please ensure all previous cells have been executed successfully\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 8. Performance Analysis and Best Practices\n",
- "\n",
- "**Purpose**: Translate demonstration results into actionable production guidance with specific configuration recommendations for different use cases.\n",
- "\n",
- "**What This Analysis Provides**:\n",
- "- **Performance Trade-offs**: Latency vs quality vs cost for each technique\n",
- "- **Configuration Guidance**: Optimal settings for different environments\n",
- "- **Decision Framework**: When to use which technique based on requirements\n",
- "- **Production Patterns**: Proven configurations for real-world deployments\n",
- "\n",
- "**Decision Matrix by Use Case**:\n",
- "- **Development/Testing**: Fast iteration with good quality\n",
- "- **Production (Speed)**: Prioritize low latency for user-facing applications \n",
- "- **Production (Quality)**: Prioritize accuracy for high-stakes applications\n",
- "- **Cost-Conscious**: Optimize for token usage while maintaining quality\n",
- "\n",
- "**Key Insights You'll Gain**:\n",
- "- Which techniques provide the best ROI for your specific needs\n",
- "- How to configure systems for optimal performance in your environment\n",
- "- Common pitfalls and how to avoid them\n",
- "- Monitoring strategies for production systems\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "๐ Performance Analysis & Best Practices\n",
- "======================================================================\n",
- "\n",
- "๐ Technique Comparison:\n",
- "--------------------------------------------------\n",
- "\n",
- "๐ Standard RAG\n",
- " Latency: Low (1-3s)\n",
- " Accuracy: Baseline\n",
- " Cost: Low\n",
- " Best for: General queries, fast responses\n",
- " โ
Pros: Fast, Simple, Cost-effective\n",
- " โ ๏ธ Cons: May include irrelevant results, Basic formatting\n",
- "\n",
- "๐ Node Postprocessors\n",
- " Latency: Low (1-4s)\n",
- " Accuracy: Higher\n",
- " Cost: Low\n",
- " Best for: Filtering noisy results, domain-specific queries\n",
- " โ
Pros: Better relevance, Configurable filters, Minimal cost\n",
- " โ ๏ธ Cons: May over-filter, Requires tuning\n",
- "\n",
- "๐ Response Synthesizers\n",
- " Latency: Medium (3-8s)\n",
- " Accuracy: Much Higher\n",
- " Cost: Medium\n",
- " Best for: Complex queries, detailed responses\n",
- " โ
Pros: Rich responses, Domain-specific formatting, Hierarchical processing\n",
- " โ ๏ธ Cons: Higher latency, More token usage\n",
- "\n",
- "๐ Structured Outputs\n",
- " Latency: Medium (3-6s)\n",
- " Accuracy: Highest\n",
- " Cost: Medium\n",
- " Best for: Data extraction, API integration\n",
- " โ
Pros: Type-safe, Reliable format, Easy integration\n",
- " โ ๏ธ Cons: Schema dependency, Less flexibility\n",
- "\n",
- "\n",
- "๐ก Best Practices & Recommendations:\n",
- "==================================================\n",
- "\n",
- "๐ฏ When to Use Each Technique:\n",
- " โข Node Postprocessors: Always use for production - minimal cost, big improvement\n",
- " โข Response Synthesizers: Use for complex, multi-part questions\n",
- " โข Structured Outputs: Use for data extraction and API integration\n",
- "\n",
- "โก Performance Optimization:\n",
- " โข Start with smaller chunk sizes (512) for better precision\n",
- " โข Use similarity cutoffs (0.3+) to filter noise\n",
- " โข Retrieve more candidates (10+) for better postprocessing\n",
- " โข Cache embeddings and indexes for faster queries\n",
- "\n",
- "๐ฐ Cost Management:\n",
- " โข Use local embeddings to reduce API costs\n",
- " โข Implement similarity filtering before expensive synthesis\n",
- " โข Choose synthesizer based on query complexity\n",
- " โข Monitor token usage in production\n",
- "\n",
- "๐จ Quality Improvement:\n",
- " โข Create domain-specific prompt templates\n",
- " โข Tune postprocessor thresholds for your data\n",
- " โข Use structured outputs for consistent results\n",
- " โข A/B test different configurations\n",
- "\n",
- "\n",
- "โ๏ธ Recommended Configurations:\n",
- "========================================\n",
- "\n",
- "๐ Development/Testing:\n",
- " chunk_size: 512\n",
- " similarity_top_k: 5\n",
- " similarity_cutoff: 0.2\n",
- " synthesizer: Simple\n",
- " postprocessors: Similarity only\n",
- "\n",
- "๐ Production (Fast):\n",
- " chunk_size: 1024\n",
- " similarity_top_k: 8\n",
- " similarity_cutoff: 0.3\n",
- " synthesizer: Compact\n",
- " postprocessors: Similarity + Keyword\n",
- "\n",
- "๐ Production (Quality):\n",
- " chunk_size: 512\n",
- " similarity_top_k: 12\n",
- " similarity_cutoff: 0.25\n",
- " synthesizer: Tree/Refine\n",
- " postprocessors: Multi-stage\n",
- "\n",
- "\n",
- "๐ Advanced RAG Tutorial Complete!\n",
- "You now have the tools to build sophisticated, production-ready RAG systems!\n"
- ]
- }
- ],
- "source": [
- "# Performance analysis and best practices\n",
- "\n",
- "def analyze_performance_characteristics():\n",
- " \"\"\"Analyze performance of different advanced RAG techniques.\"\"\"\n",
- " \n",
- " print(\"๐ Performance Analysis & Best Practices\")\n",
- " print(\"=\" * 70)\n",
- " \n",
- " # Performance characteristics\n",
- " techniques = {\n",
- " \"Standard RAG\": {\n",
- " \"latency\": \"Low (1-3s)\",\n",
- " \"accuracy\": \"Baseline\",\n",
- " \"cost\": \"Low\",\n",
- " \"use_case\": \"General queries, fast responses\",\n",
- " \"pros\": [\"Fast\", \"Simple\", \"Cost-effective\"],\n",
- " \"cons\": [\"May include irrelevant results\", \"Basic formatting\"]\n",
- " },\n",
- " \"Node Postprocessors\": {\n",
- " \"latency\": \"Low (1-4s)\",\n",
- " \"accuracy\": \"Higher\",\n",
- " \"cost\": \"Low\",\n",
- " \"use_case\": \"Filtering noisy results, domain-specific queries\",\n",
- " \"pros\": [\"Better relevance\", \"Configurable filters\", \"Minimal cost\"],\n",
- " \"cons\": [\"May over-filter\", \"Requires tuning\"]\n",
- " },\n",
- " \"Response Synthesizers\": {\n",
- " \"latency\": \"Medium (3-8s)\",\n",
- " \"accuracy\": \"Much Higher\",\n",
- " \"cost\": \"Medium\",\n",
- " \"use_case\": \"Complex queries, detailed responses\",\n",
- " \"pros\": [\"Rich responses\", \"Domain-specific formatting\", \"Hierarchical processing\"],\n",
- " \"cons\": [\"Higher latency\", \"More token usage\"]\n",
- " },\n",
- " \"Structured Outputs\": {\n",
- " \"latency\": \"Medium (3-6s)\",\n",
- " \"accuracy\": \"Highest\",\n",
- " \"cost\": \"Medium\",\n",
- " \"use_case\": \"Data extraction, API integration\",\n",
- " \"pros\": [\"Type-safe\", \"Reliable format\", \"Easy integration\"],\n",
- " \"cons\": [\"Schema dependency\", \"Less flexibility\"]\n",
- " }\n",
- " }\n",
- " \n",
- " print(\"\\n๐ Technique Comparison:\")\n",
- " print(\"-\" * 50)\n",
- " \n",
- " for technique, specs in techniques.items():\n",
- " print(f\"\\n๐ {technique}\")\n",
- " print(f\" Latency: {specs['latency']}\")\n",
- " print(f\" Accuracy: {specs['accuracy']}\")\n",
- " print(f\" Cost: {specs['cost']}\")\n",
- " print(f\" Best for: {specs['use_case']}\")\n",
- " print(f\" โ
Pros: {', '.join(specs['pros'])}\")\n",
- " print(f\" โ ๏ธ Cons: {', '.join(specs['cons'])}\")\n",
- " \n",
- " # Best practices recommendations\n",
- " print(\"\\n\\n๐ก Best Practices & Recommendations:\")\n",
- " print(\"=\" * 50)\n",
- " \n",
- " recommendations = [\n",
- " {\n",
- " \"category\": \"๐ฏ When to Use Each Technique\",\n",
- " \"tips\": [\n",
- " \"Node Postprocessors: Always use for production - minimal cost, big improvement\",\n",
- " \"Response Synthesizers: Use for complex, multi-part questions\",\n",
- " \"Structured Outputs: Use for data extraction and API integration\"\n",
- " ]\n",
- " },\n",
- " {\n",
- " \"category\": \"โก Performance Optimization\",\n",
- " \"tips\": [\n",
- " \"Start with smaller chunk sizes (512) for better precision\",\n",
- " \"Use similarity cutoffs (0.3+) to filter noise\",\n",
- " \"Retrieve more candidates (10+) for better postprocessing\",\n",
- " \"Cache embeddings and indexes for faster queries\"\n",
- " ]\n",
- " },\n",
- " {\n",
- " \"category\": \"๐ฐ Cost Management\",\n",
- " \"tips\": [\n",
- " \"Use local embeddings to reduce API costs\",\n",
- " \"Implement similarity filtering before expensive synthesis\",\n",
- " \"Choose synthesizer based on query complexity\",\n",
- " \"Monitor token usage in production\"\n",
- " ]\n",
- " },\n",
- " {\n",
- " \"category\": \"๐จ Quality Improvement\",\n",
- " \"tips\": [\n",
- " \"Create domain-specific prompt templates\",\n",
- " \"Tune postprocessor thresholds for your data\",\n",
- " \"Use structured outputs for consistent results\",\n",
- " \"A/B test different configurations\"\n",
- " ]\n",
- " }\n",
- " ]\n",
- " \n",
- " for rec in recommendations:\n",
- " print(f\"\\n{rec['category']}:\")\n",
- " for tip in rec['tips']:\n",
- " print(f\" โข {tip}\")\n",
- " \n",
- " # Configuration recommendations\n",
- " print(\"\\n\\nโ๏ธ Recommended Configurations:\")\n",
- " print(\"=\" * 40)\n",
- " \n",
- " configs = {\n",
- " \"Development/Testing\": {\n",
- " \"chunk_size\": 512,\n",
- " \"similarity_top_k\": 5,\n",
- " \"similarity_cutoff\": 0.2,\n",
- " \"synthesizer\": \"Simple\",\n",
- " \"postprocessors\": \"Similarity only\"\n",
- " },\n",
- " \"Production (Fast)\": {\n",
- " \"chunk_size\": 1024,\n",
- " \"similarity_top_k\": 8,\n",
- " \"similarity_cutoff\": 0.3,\n",
- " \"synthesizer\": \"Compact\",\n",
- " \"postprocessors\": \"Similarity + Keyword\"\n",
- " },\n",
- " \"Production (Quality)\": {\n",
- " \"chunk_size\": 512,\n",
- " \"similarity_top_k\": 12,\n",
- " \"similarity_cutoff\": 0.25,\n",
- " \"synthesizer\": \"Tree/Refine\",\n",
- " \"postprocessors\": \"Multi-stage\"\n",
- " }\n",
- " }\n",
- " \n",
- " for env, config in configs.items():\n",
- " print(f\"\\n๐ {env}:\")\n",
- " for param, value in config.items():\n",
- " print(f\" {param}: {value}\")\n",
- "\n",
- "# Run performance analysis\n",
- "analyze_performance_characteristics()\n",
- "\n",
- "print(\"\\n\\n๐ Advanced RAG Tutorial Complete!\")\n",
- "print(\"You now have the tools to build sophisticated, production-ready RAG systems!\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Conclusion\n",
- "\n",
- "๐ **Congratulations!** You have successfully mastered **Advanced RAG Techniques** with LlamaIndex!\n",
- "\n",
- "### What We Accomplished\n",
- "\n",
- "This comprehensive tutorial demonstrated sophisticated RAG techniques using real multimodal data:\n",
- "\n",
- "#### ๐ง **Node Postprocessors Mastery**\n",
- "- โ
**Similarity Filtering**: Automated relevance-based result filtering\n",
- "- โ
**Keyword Filtering**: Content-based inclusion/exclusion rules\n",
- "- โ
**Multi-stage Processing**: Chained postprocessor pipelines\n",
- "- โ
**Custom Filtering**: Domain-specific result refinement\n",
- "\n",
- "#### ๐ฏ **Response Synthesizers Expertise**\n",
- "- โ
**Tree Summarize**: Hierarchical response building for complex queries\n",
- "- โ
**Refine**: Iterative response improvement with multiple sources\n",
- "- โ
**Compact and Refine**: Token-optimized processing\n",
- "- โ
**Custom Templates**: Domain-specific response formatting\n",
- "- โ
**Template Optimization**: Cooking, finance, travel-specific prompts\n",
- "\n",
- "#### ๐ **Structured Output Mastery**\n",
- "- โ
**Pydantic Models**: Type-safe data extraction schemas\n",
- "- โ
**Domain Models**: Recipe, Investment, Travel extractors\n",
- "- โ
**Enum Support**: Controlled vocabulary enforcement\n",
- "- โ
**JSON Schema**: Reliable structured data formatting\n",
- "\n",
- "#### โก **Performance & Production Insights**\n",
- "- โ
**Latency Optimization**: Performance vs quality trade-offs\n",
- "- โ
**Cost Management**: Token usage optimization strategies\n",
- "- โ
**Configuration Tuning**: Environment-specific recommendations\n",
- "- โ
**Best Practices**: Production deployment guidelines\n",
- "\n",
- "### Real-World Applications\n",
- "\n",
- "These advanced techniques enable sophisticated applications:\n",
- "\n",
- "- **๐ข Enterprise RAG**: Multi-stage filtering for accurate business intelligence\n",
- "- **๐ฌ Research Systems**: Hierarchical synthesis for complex analysis\n",
- "- **๐ E-commerce**: Hybrid search for product discovery\n",
- "- **๐ฅ Healthcare**: Structured extraction for medical data processing\n",
- "- **๐ Educational**: Domain-specific response formatting\n",
- "- **๐ฑ APIs**: Type-safe data extraction for system integration\n",
- "\n",
- "### Key Takeaways\n",
- "\n",
- "1. **๐ฏ Postprocessors are Essential**: Always use similarity filtering in production\n",
- "2. **๐จ Templates Matter**: Domain-specific prompts dramatically improve quality\n",
- "3. **โ๏ธ Balance is Key**: Choose techniques based on latency vs quality needs\n",
- "4. **๐ง Tuning is Critical**: Configuration significantly impacts performance\n",
- "5. **๐ Structure Enables Integration**: Pydantic models ensure reliable data flow\n",
- "\n",
- "### Architecture Comparison\n",
- "\n",
- "| Technique | Latency | Accuracy | Cost | Best Use Case |\n",
- "|-----------|---------|----------|------|---------------|\n",
- "| **Standard RAG** | Low (1-3s) | Baseline | Low | General queries |\n",
- "| **+ Postprocessors** | Low (1-4s) | Higher | Low | Filtered results |\n",
- "| **+ Synthesizers** | Medium (3-8s) | Much Higher | Medium | Complex queries |\n",
- "| **+ Structured** | Medium (3-6s) | Highest | Medium | Data extraction |\n",
- "\n",
- "### Next Steps\n",
- "\n",
- "Continue your RAG journey by:\n",
- "\n",
- "1. **๐ Implementing A/B Testing**: Compare different technique combinations\n",
- "2. **๐ Adding Evaluation Metrics**: Monitor accuracy and performance\n",
- "3. **๐ Scaling to Production**: Implement async processing and caching\n",
- "4. **๐ค Building Agents**: Combine RAG with tool-using agents\n",
- "5. **๐ฎ Exploring Cutting-Edge**: Keep up with latest LlamaIndex features\n",
- "\n",
- "---\n",
- "\n",
- "**๐ You're now equipped to build world-class RAG systems!** \n",
- "\n",
- "The techniques you've learned represent the current state-of-the-art in retrieval-augmented generation, enabling you to create sophisticated, production-ready applications that can handle complex queries across diverse data types with unprecedented accuracy and reliability.\n",
- "\n",
- "### Final Configuration Template\n",
- "\n",
- "```python\n",
- "# Production-Ready Advanced RAG Configuration\n",
- "CONFIG = {\n",
- " \"llm_model\": \"gpt-4o\",\n",
- " \"embedding_model\": \"local:BAAI/bge-small-en-v1.5\", \n",
- " \"chunk_size\": 512, # Precision over speed\n",
- " \"chunk_overlap\": 50, # Minimal overlap\n",
- " \"similarity_top_k\": 10, # More candidates\n",
- " \"final_top_k\": 5, # Refined results\n",
- " \"similarity_cutoff\": 0.3, # Quality threshold\n",
- "}\n",
- "\n",
- "# Multi-stage postprocessing pipeline\n",
- "postprocessors = [\n",
- " SimilarityPostprocessor(similarity_cutoff=0.3),\n",
- " KeywordNodePostprocessor(exclude_keywords=[\"noise\", \"irrelevant\"])\n",
- "]\n",
- "\n",
- "# Domain-specific synthesizers\n",
- "synthesizers = {\n",
- " \"cooking\": TreeSummarize(summary_template=cooking_template),\n",
- " \"finance\": CompactAndRefine(text_qa_template=finance_template),\n",
- " \"travel\": SimpleSummarize(text_qa_template=travel_template)\n",
- "}\n",
- "\n",
- "# Structured output models for reliable data extraction\n",
- "structured_models = {\n",
- " \"recipe\": PydanticOutputParser(RecipeInfo),\n",
- " \"investment\": PydanticOutputParser(InvestmentInfo),\n",
- " \"travel\": PydanticOutputParser(TravelInfo)\n",
- "}\n",
- "```\n",
- "\n",
- "**Happy building!** ๐ฆ๐โจ\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "accelerator",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.13"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/Girish_Basavaraj_Hiremath/session_2/papers/agents/AI_Agents_vs_Agentic_AI.pdf b/Girish_Basavaraj_Hiremath/session_2/papers/agents/AI_Agents_vs_Agentic_AI.pdf
deleted file mode 100644
index 4276fe2..0000000
Binary files a/Girish_Basavaraj_Hiremath/session_2/papers/agents/AI_Agents_vs_Agentic_AI.pdf and /dev/null differ
diff --git a/Girish_Basavaraj_Hiremath/session_2/papers/agents/Emerging_Agent_Architectures.pdf b/Girish_Basavaraj_Hiremath/session_2/papers/agents/Emerging_Agent_Architectures.pdf
deleted file mode 100644
index b16b0b6..0000000
Binary files a/Girish_Basavaraj_Hiremath/session_2/papers/agents/Emerging_Agent_Architectures.pdf and /dev/null differ
diff --git a/Girish_Basavaraj_Hiremath/session_2/papers/agents/LLMReasoning_to_Autonomous_Agents.pdf b/Girish_Basavaraj_Hiremath/session_2/papers/agents/LLMReasoning_to_Autonomous_Agents.pdf
deleted file mode 100644
index 7749c0d..0000000
Binary files a/Girish_Basavaraj_Hiremath/session_2/papers/agents/LLMReasoning_to_Autonomous_Agents.pdf and /dev/null differ
diff --git a/Girish_Basavaraj_Hiremath/session_2/papers/agents/Rise_and_Potential_LLM_Agents.pdf b/Girish_Basavaraj_Hiremath/session_2/papers/agents/Rise_and_Potential_LLM_Agents.pdf
deleted file mode 100644
index c600ee5..0000000
Binary files a/Girish_Basavaraj_Hiremath/session_2/papers/agents/Rise_and_Potential_LLM_Agents.pdf and /dev/null differ
diff --git a/Girish_Basavaraj_Hiremath/session_2/papers/agents/survey_of_self_evolving_agents.pdf b/Girish_Basavaraj_Hiremath/session_2/papers/agents/survey_of_self_evolving_agents.pdf
deleted file mode 100644
index 29bb5b3..0000000
Binary files a/Girish_Basavaraj_Hiremath/session_2/papers/agents/survey_of_self_evolving_agents.pdf and /dev/null differ
diff --git a/Girish_Basavaraj_Hiremath/session_2/prompt_samples/assignment_generator.yaml b/Girish_Basavaraj_Hiremath/session_2/prompt_samples/assignment_generator.yaml
deleted file mode 100644
index c4a0ead..0000000
--- a/Girish_Basavaraj_Hiremath/session_2/prompt_samples/assignment_generator.yaml
+++ /dev/null
@@ -1,94 +0,0 @@
-system_prompt: |
- You are an expert educational content creator and assessment specialist. Your task is to generate comprehensive, practical assignments based on YouTube video summaries to help students apply and reinforce their learning.
-
- Your assignments should:
-
- **EDUCATIONAL OBJECTIVES:**
- - Test practical understanding, not just memorization
- - Encourage hands-on implementation and experimentation
- - Build progressive complexity from basic concepts to advanced applications
- - Promote critical thinking and problem-solving skills
- - Include real-world scenarios and use cases
-
- **ASSIGNMENT STRUCTURE:**
- Create a well-structured assignment with the following sections:
-
- 1. **Assignment Overview** (2-3 sentences)
- - Clear learning objective
- - Expected outcome or deliverable
- - Estimated time commitment
-
- 2. **Prerequisite Knowledge** (bullet points)
- - Essential concepts students should know before starting
- - Recommended prior experience or skills
-
- 3. **Core Tasks** (3-5 main tasks)
- - Each task should build upon the previous one
- - Include specific, actionable steps
- - Mix theoretical understanding with practical implementation
- - Provide clear success criteria for each task
-
- 4. **Practical Exercises** (2-4 hands-on activities)
- - Coding exercises, projects, or experiments
- - Real-world problem-solving scenarios
- - Troubleshooting or debugging challenges
- - Performance optimization tasks
-
- 5. **Advanced Challenges** (1-2 extension activities)
- - For students who complete core tasks early
- - More complex scenarios or integrations
- - Research-based questions
- - Creative problem-solving opportunities
-
- 6. **Assessment Criteria** (clear rubric)
- - What constitutes successful completion
- - Quality indicators and best practices
- - Common pitfalls to avoid
- - Bonus points opportunities
-
- 7. **Resources & References**
- - Additional learning materials
- - Documentation links
- - Community resources or forums
- - Tools and software recommendations
-
- **FORMATTING GUIDELINES:**
- - Use clear headings and subheadings
- - Include code snippets or examples where relevant
- - Use bullet points for lists and action items
- - Add emojis strategically for visual organization
- - Ensure tasks are specific and measurable
- - Include estimated time for each section
-
- **DIFFICULTY ADAPTATION:**
- - For Beginner content: Focus on foundational concepts, guided exercises, step-by-step instructions
- - For Intermediate content: Include problem-solving scenarios, multiple solution approaches, integration tasks
- - For Advanced content: Emphasize optimization, architecture decisions, and innovative applications
-
- **PRACTICAL FOCUS:**
- - Every assignment should result in a tangible deliverable
- - Include testing and validation steps
- - Encourage documentation and explanation of decisions
- - Promote best practices and industry standards
-
- Respond with a comprehensive assignment in markdown format that educators can directly use with students. Make the assignment engaging, educational, and appropriately challenging for the content level.
-
-user_prompt_template: |
- Create a comprehensive practical assignment based on the following YouTube video summary:
-
- **Video Title:** {title}
- **Channel:** {channel}
- **Difficulty Level:** {difficulty}
- **Content Summary:**
- {summary}
-
- **Key Technical Elements:**
- {technical_breakdown}
-
- **Learning Insights:**
- {insights}
-
- **Practical Applications:**
- {applications}
-
- Generate an assignment that helps students apply these concepts practically and reinforces the learning from this video content.
\ No newline at end of file
diff --git a/Girish_Basavaraj_Hiremath/session_2/prompt_samples/summarizer_youtube.yaml b/Girish_Basavaraj_Hiremath/session_2/prompt_samples/summarizer_youtube.yaml
deleted file mode 100644
index 5747507..0000000
--- a/Girish_Basavaraj_Hiremath/session_2/prompt_samples/summarizer_youtube.yaml
+++ /dev/null
@@ -1,78 +0,0 @@
-name: TechSummarizerAI
-version: 1.0
-description: >
- Expert AI assistant specializing in analyzing technical videos for software engineers,
- machine learning engineers, and other technical audiences.
-prompt: |
- # SYSTEM PROMPT
-
- You are **TechSummarizerAI**, an expert AI assistant specializing in analyzing technical videos for software engineers, machine learning engineers, and other technical audiences.
- You will be given:
- - **YouTube Video Title**
- - **YouTube Video Description**
- - **Full Transcript**
-
- Your goal is to produce a **comprehensive, technical, and structured summary** that highlights key engineering insights, tools, frameworks, architectures, processes, and design decisions from the video.
-
- ---
-
- ## ๐ฏ **Objectives**
- 1. **Engineerโs Perspective** โ Extract deep technical details, avoiding generic recaps.
- 2. **Implementation Relevance** โ Show *how* ideas in the video could be applied in real-world engineering.
- 3. **Precision** โ Avoid speculation. Only summarize from the provided title, description, and transcript.
- 4. **Clarity** โ Use concise, professional language.
- 5. **Structure** โ Organize the summary in clear technical sections.
-
- ---
-
- ## ๐ **Final Output Format**
- Produce the summary strictly following this Markdown structure:
-
- ### 1. **High-Level Overview**
- - One paragraph capturing the essence of the video from an engineering viewpoint.
-
- ### 2. **Technical Breakdown**
- #### 2.1 Tools, Frameworks, and Libraries
- - List all mentioned tools, frameworks, packages, or APIs.
- - Include their purpose in the workflow.
-
- #### 2.2 Architecture / System Design
- - Describe the system architecture or process flow discussed.
- - Use bullet points or a diagram-like indentation.
-
- #### 2.3 Step-by-Step Process
- - Present each step on its own line.
- - Never write multiple steps in a single line inside the `๐ {assistant_name}
" - f"Response Style: {response_style} | History Limit: {max_history} messages", - unsafe_allow_html=True, -) -st.info("Hello! I'm your demo assistant. How can I help you today?") - -# -------- Chat History Cards -------- -for msg in st.session_state.chat_history[-max_history:]: - icon = "๐ง" if msg["sender"] == "You" else "๐ค" - bg_col = "#222" if msg["sender"] == "You" else "#353a47" - text_col = "#fff" - sender_str = f"{icon} {msg['sender']}" - time_str = f"{msg['timestamp']}" if show_timestamps else "" - st.markdown( - f""" -
- {sender_str} {time_str}
{msg['message']} -
- """, unsafe_allow_html=True
- )
-
-# -------- Expanders and Dev Info --------
-with st.expander("About This Demo"):
- st.write("This is a sample Streamlit Chat Assistant inspired by your frontend screenshot.")
-
-with st.expander("Instructor Notes"):
- st.write("You can extend this code to include actual AI/model integration.")
-
-st.session_state.show_dev_info = st.checkbox("Show Development Info", value=st.session_state.show_dev_info)
-if st.session_state.show_dev_info:
- st.write(" ---- Development Info ---- ")
-
-# -------- Chat Input and Sending --------
-with st.form(key="chat_form", clear_on_submit=True):
- message = st.text_input("Message", placeholder=f"Message {assistant_name}...")
- submitted = st.form_submit_button("Send")
- if submitted and message.strip():
- timestamp = datetime.now().strftime("%H:%M:%S")
- # Add user message
- st.session_state.chat_history.append({
- "sender": "You",
- "message": message,
- "timestamp": timestamp
- })
- st.session_state.messages_sent += 1
- # Bot reply (demo logic)
- bot_reply = (
- f"Hey, great question about '{message}'! "
- f"I'm happy to help you with that. Here's what I'm thinking..."
- )
- st.session_state.chat_history.append({
- "sender": assistant_name,
- "message": bot_reply,
- "timestamp": timestamp
- })
-
diff --git a/Maruti_Divekar/Day_04/README.MD b/Maruti_Divekar/Day_04/README.MD
deleted file mode 100644
index 8b13789..0000000
--- a/Maruti_Divekar/Day_04/README.MD
+++ /dev/null
@@ -1 +0,0 @@
-
diff --git a/Maruti_Divekar/Day_05/README.md b/Maruti_Divekar/Day_05/README.md
deleted file mode 100644
index 8b13789..0000000
--- a/Maruti_Divekar/Day_05/README.md
+++ /dev/null
@@ -1 +0,0 @@
-
diff --git a/Maruti_Divekar/Day_07/README.md b/Maruti_Divekar/Day_07/README.md
deleted file mode 100644
index 8b13789..0000000
--- a/Maruti_Divekar/Day_07/README.md
+++ /dev/null
@@ -1 +0,0 @@
-
diff --git a/Maruti_Divekar/README.md b/Maruti_Divekar/README.md
deleted file mode 100644
index 92e084a..0000000
--- a/Maruti_Divekar/README.md
+++ /dev/null
@@ -1 +0,0 @@
-# Maruti_Divekar
diff --git a/Mayur_Chaudhari/README.md b/Mayur_Chaudhari/README.md
deleted file mode 100644
index 1a7947f..0000000
--- a/Mayur_Chaudhari/README.md
+++ /dev/null
@@ -1 +0,0 @@
-# Mayur_Chaudhari
diff --git a/Michal_Maciejewski/README.md b/Michal_Maciejewski/README.md
deleted file mode 100644
index 9c347f4..0000000
--- a/Michal_Maciejewski/README.md
+++ /dev/null
@@ -1 +0,0 @@
-# Michal_Maciejewski
diff --git a/Mishi_Vidya_Sinku/README.md b/Mishi_Vidya_Sinku/README.md
deleted file mode 100644
index 344f68d..0000000
--- a/Mishi_Vidya_Sinku/README.md
+++ /dev/null
@@ -1 +0,0 @@
-# Mishi_Vidya_Sinku
diff --git a/Monalisa_Samal/Day_3_Completed_3_assignments/README.md b/Monalisa_Samal/Day_3_Completed_3_assignments/README.md
deleted file mode 100644
index 8b13789..0000000
--- a/Monalisa_Samal/Day_3_Completed_3_assignments/README.md
+++ /dev/null
@@ -1 +0,0 @@
-
diff --git a/Monalisa_Samal/Day_6_Completed_4_assignments/README.md b/Monalisa_Samal/Day_6_Completed_4_assignments/README.md
deleted file mode 100644
index 2ebb74d..0000000
--- a/Monalisa_Samal/Day_6_Completed_4_assignments/README.md
+++ /dev/null
@@ -1 +0,0 @@
-Completed the 4 assigments of Day on RAG
diff --git a/Monalisa_Samal/Day_6_Completed_4_assignments/assignment_1_vector_db_basics.ipynb b/Monalisa_Samal/Day_6_Completed_4_assignments/assignment_1_vector_db_basics.ipynb
deleted file mode 100644
index ee1c097..0000000
--- a/Monalisa_Samal/Day_6_Completed_4_assignments/assignment_1_vector_db_basics.ipynb
+++ /dev/null
@@ -1 +0,0 @@
-{"cells":[{"cell_type":"markdown","metadata":{"id":"rX4VtY8Gc4Aa"},"source":["# Assignment 1: Vector Database Creation and Retrieval\n","## Day 6 Session 2 - RAG Fundamentals\n","\n","**OBJECTIVE:** Create a vector database from a folder of documents and implement basic retrieval functionality.\n","\n","**LEARNING GOALS:**\n","- Understand document loading with SimpleDirectoryReader\n","- Learn vector store setup with LanceDB\n","- Implement vector index creation\n","- Perform semantic search and retrieval\n","\n","**DATASET:** Use the data folder in `Day_6/session_2/data/` which contains multiple file types\n","\n","**INSTRUCTIONS:**\n","1. Complete each function by replacing the TODO comments with actual implementation\n","2. Run each cell after completing the function to test it\n","3. The answers can be found in the existing notebooks in the `llamaindex_rag/` folder\n"]},{"cell_type":"code","source":["from google.colab import drive\n","drive.mount('/content/drive')"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"mGtEGJgKdEty","executionInfo":{"status":"ok","timestamp":1762068951338,"user_tz":-330,"elapsed":31991,"user":{"displayName":"","userId":""}},"outputId":"3f9858b0-fd42-41ae-bddc-ed6cbd3baca5"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Mounted at /content/drive\n"]}]},{"cell_type":"code","source":["# If it's in a specific folder (e.g., \"Projects/MyProject/\")\n","!pip install -r '/content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt'"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":1000},"id":"L4iAVZ0Yf0g-","executionInfo":{"status":"ok","timestamp":1762069356067,"user_tz":-330,"elapsed":49537,"user":{"displayName":"","userId":""}},"outputId":"809ee24f-a3e1-4d97-c939-fab5baf825ff"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.12/dist-packages (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 1)) (4.13.5)\n","Requirement already satisfied: google-api-core in /usr/local/lib/python3.12/dist-packages (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 2)) (2.28.0)\n","Requirement already satisfied: google-api-python-client in /usr/local/lib/python3.12/dist-packages (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 3)) (2.185.0)\n","Requirement already satisfied: google-auth in /usr/local/lib/python3.12/dist-packages (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 4)) (2.38.0)\n","Requirement already satisfied: google-auth-httplib2 in /usr/local/lib/python3.12/dist-packages (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 5)) (0.2.0)\n","Requirement already satisfied: gradio in /usr/local/lib/python3.12/dist-packages (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (5.49.1)\n","Requirement already satisfied: gradio_client in /usr/local/lib/python3.12/dist-packages (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 7)) (1.13.3)\n","Requirement already satisfied: huggingface-hub in /usr/local/lib/python3.12/dist-packages (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 8)) (0.36.0)\n","Requirement already satisfied: ipykernel in /usr/local/lib/python3.12/dist-packages (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 9)) (6.17.1)\n","Requirement already satisfied: ipython in /usr/local/lib/python3.12/dist-packages (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 10)) (7.34.0)\n","Collecting lancedb (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 11))\n"," Downloading lancedb-0.25.2-cp39-abi3-manylinux_2_28_x86_64.whl.metadata (4.8 kB)\n","Collecting llama-index (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_index-0.14.7-py3-none-any.whl.metadata (13 kB)\n","Collecting llama-index-vector-stores-lancedb (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 13))\n"," Downloading llama_index_vector_stores_lancedb-0.4.1-py3-none-any.whl.metadata (460 bytes)\n","Collecting llama-index-embeddings-huggingface (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 14))\n"," Downloading llama_index_embeddings_huggingface-0.6.1-py3-none-any.whl.metadata (458 bytes)\n","Collecting llama-index-llms-huggingface-api (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 15))\n"," Downloading llama_index_llms_huggingface_api-0.6.1-py3-none-any.whl.metadata (1.1 kB)\n","Collecting llama-index-embeddings-openai (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 16))\n"," Downloading llama_index_embeddings_openai-0.5.1-py3-none-any.whl.metadata (400 bytes)\n","Collecting llama-index-llms-openrouter (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 17))\n"," Downloading llama_index_llms_openrouter-0.4.2-py3-none-any.whl.metadata (2.3 kB)\n","Requirement already satisfied: nltk in /usr/local/lib/python3.12/dist-packages (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 18)) (3.9.1)\n","Requirement already satisfied: numpy in /usr/local/lib/python3.12/dist-packages (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 19)) (2.0.2)\n","Requirement already satisfied: pandas in /usr/local/lib/python3.12/dist-packages (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 20)) (2.2.2)\n","Requirement already satisfied: openai in /usr/local/lib/python3.12/dist-packages (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 21)) (1.109.1)\n","Collecting openai-whisper (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 22))\n"," Downloading openai_whisper-20250625.tar.gz (803 kB)\n","\u001b[2K \u001b[90mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m \u001b[32m803.2/803.2 kB\u001b[0m \u001b[31m15.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25h Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n"," Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n"," Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n","Requirement already satisfied: pydantic in /usr/local/lib/python3.12/dist-packages (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 23)) (2.11.10)\n","Requirement already satisfied: sentence-transformers in /usr/local/lib/python3.12/dist-packages (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 24)) (5.1.2)\n","Collecting yt-dlp (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 25))\n"," Downloading yt_dlp-2025.10.22-py3-none-any.whl.metadata (176 kB)\n","\u001b[2K \u001b[90mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m \u001b[32m176.0/176.0 kB\u001b[0m \u001b[31m9.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hRequirement already satisfied: spacy in /usr/local/lib/python3.12/dist-packages (from -r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 26)) (3.8.7)\n","Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.12/dist-packages (from beautifulsoup4->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 1)) (2.8)\n","Requirement already satisfied: typing-extensions>=4.0.0 in /usr/local/lib/python3.12/dist-packages (from beautifulsoup4->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 1)) (4.15.0)\n","Requirement already satisfied: googleapis-common-protos<2.0.0,>=1.56.2 in /usr/local/lib/python3.12/dist-packages (from google-api-core->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 2)) (1.71.0)\n","Requirement already satisfied: protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<7.0.0,>=3.19.5 in /usr/local/lib/python3.12/dist-packages (from google-api-core->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 2)) (5.29.5)\n","Requirement already satisfied: proto-plus<2.0.0,>=1.22.3 in /usr/local/lib/python3.12/dist-packages (from google-api-core->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 2)) (1.26.1)\n","Requirement already satisfied: requests<3.0.0,>=2.18.0 in /usr/local/lib/python3.12/dist-packages (from google-api-core->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 2)) (2.32.4)\n","Requirement already satisfied: httplib2<1.0.0,>=0.19.0 in /usr/local/lib/python3.12/dist-packages (from google-api-python-client->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 3)) (0.31.0)\n","Requirement already satisfied: uritemplate<5,>=3.0.1 in /usr/local/lib/python3.12/dist-packages (from google-api-python-client->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 3)) (4.2.0)\n","Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.12/dist-packages (from google-auth->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 4)) (5.5.2)\n","Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.12/dist-packages (from google-auth->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 4)) (0.4.2)\n","Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.12/dist-packages (from google-auth->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 4)) (4.9.1)\n","Requirement already satisfied: aiofiles<25.0,>=22.0 in /usr/local/lib/python3.12/dist-packages (from gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (24.1.0)\n","Requirement already satisfied: anyio<5.0,>=3.0 in /usr/local/lib/python3.12/dist-packages (from gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (4.11.0)\n","Requirement already satisfied: brotli>=1.1.0 in /usr/local/lib/python3.12/dist-packages (from gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (1.1.0)\n","Requirement already satisfied: fastapi<1.0,>=0.115.2 in /usr/local/lib/python3.12/dist-packages (from gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (0.120.1)\n","Requirement already satisfied: ffmpy in /usr/local/lib/python3.12/dist-packages (from gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (0.6.4)\n","Requirement already satisfied: groovy~=0.1 in /usr/local/lib/python3.12/dist-packages (from gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (0.1.2)\n","Requirement already satisfied: httpx<1.0,>=0.24.1 in /usr/local/lib/python3.12/dist-packages (from gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (0.28.1)\n","Requirement already satisfied: jinja2<4.0 in /usr/local/lib/python3.12/dist-packages (from gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (3.1.6)\n","Requirement already satisfied: markupsafe<4.0,>=2.0 in /usr/local/lib/python3.12/dist-packages (from gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (3.0.3)\n","Requirement already satisfied: orjson~=3.0 in /usr/local/lib/python3.12/dist-packages (from gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (3.11.4)\n","Requirement already satisfied: packaging in /usr/local/lib/python3.12/dist-packages (from gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (25.0)\n","Requirement already satisfied: pillow<12.0,>=8.0 in /usr/local/lib/python3.12/dist-packages (from gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (11.3.0)\n","Requirement already satisfied: pydub in /usr/local/lib/python3.12/dist-packages (from gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (0.25.1)\n","Requirement already satisfied: python-multipart>=0.0.18 in /usr/local/lib/python3.12/dist-packages (from gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (0.0.20)\n","Requirement already satisfied: pyyaml<7.0,>=5.0 in /usr/local/lib/python3.12/dist-packages (from gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (6.0.3)\n","Requirement already satisfied: ruff>=0.9.3 in /usr/local/lib/python3.12/dist-packages (from gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (0.14.2)\n","Requirement already satisfied: safehttpx<0.2.0,>=0.1.6 in /usr/local/lib/python3.12/dist-packages (from gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (0.1.7)\n","Requirement already satisfied: semantic-version~=2.0 in /usr/local/lib/python3.12/dist-packages (from gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (2.10.0)\n","Requirement already satisfied: starlette<1.0,>=0.40.0 in /usr/local/lib/python3.12/dist-packages (from gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (0.49.1)\n","Requirement already satisfied: tomlkit<0.14.0,>=0.12.0 in /usr/local/lib/python3.12/dist-packages (from gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (0.13.3)\n","Requirement already satisfied: typer<1.0,>=0.12 in /usr/local/lib/python3.12/dist-packages (from gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (0.20.0)\n","Requirement already satisfied: uvicorn>=0.14.0 in /usr/local/lib/python3.12/dist-packages (from gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (0.38.0)\n","Requirement already satisfied: fsspec in /usr/local/lib/python3.12/dist-packages (from gradio_client->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 7)) (2025.3.0)\n","Requirement already satisfied: websockets<16.0,>=13.0 in /usr/local/lib/python3.12/dist-packages (from gradio_client->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 7)) (15.0.1)\n","Requirement already satisfied: filelock in /usr/local/lib/python3.12/dist-packages (from huggingface-hub->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 8)) (3.20.0)\n","Requirement already satisfied: tqdm>=4.42.1 in /usr/local/lib/python3.12/dist-packages (from huggingface-hub->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 8)) (4.67.1)\n","Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in /usr/local/lib/python3.12/dist-packages (from huggingface-hub->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 8)) (1.2.0)\n","Requirement already satisfied: debugpy>=1.0 in /usr/local/lib/python3.12/dist-packages (from ipykernel->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 9)) (1.8.15)\n","Requirement already satisfied: jupyter-client>=6.1.12 in /usr/local/lib/python3.12/dist-packages (from ipykernel->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 9)) (7.4.9)\n","Requirement already satisfied: matplotlib-inline>=0.1 in /usr/local/lib/python3.12/dist-packages (from ipykernel->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 9)) (0.2.1)\n","Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.12/dist-packages (from ipykernel->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 9)) (1.6.0)\n","Requirement already satisfied: psutil in /usr/local/lib/python3.12/dist-packages (from ipykernel->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 9)) (5.9.5)\n","Requirement already satisfied: pyzmq>=17 in /usr/local/lib/python3.12/dist-packages (from ipykernel->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 9)) (26.2.1)\n","Requirement already satisfied: tornado>=6.1 in /usr/local/lib/python3.12/dist-packages (from ipykernel->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 9)) (6.5.1)\n","Requirement already satisfied: traitlets>=5.1.0 in /usr/local/lib/python3.12/dist-packages (from ipykernel->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 9)) (5.7.1)\n","Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.12/dist-packages (from ipython->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 10)) (75.2.0)\n","Collecting jedi>=0.16 (from ipython->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 10))\n"," Downloading jedi-0.19.2-py2.py3-none-any.whl.metadata (22 kB)\n","Requirement already satisfied: decorator in /usr/local/lib/python3.12/dist-packages (from ipython->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 10)) (4.4.2)\n","Requirement already satisfied: pickleshare in /usr/local/lib/python3.12/dist-packages (from ipython->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 10)) (0.7.5)\n","Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /usr/local/lib/python3.12/dist-packages (from ipython->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 10)) (3.0.52)\n","Requirement already satisfied: pygments in /usr/local/lib/python3.12/dist-packages (from ipython->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 10)) (2.19.2)\n","Requirement already satisfied: backcall in /usr/local/lib/python3.12/dist-packages (from ipython->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 10)) (0.2.0)\n","Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.12/dist-packages (from ipython->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 10)) (4.9.0)\n","Collecting deprecation (from lancedb->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 11))\n"," Downloading deprecation-2.1.0-py2.py3-none-any.whl.metadata (4.6 kB)\n","Requirement already satisfied: pyarrow>=16 in /usr/local/lib/python3.12/dist-packages (from lancedb->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 11)) (18.1.0)\n","Collecting lance-namespace>=0.0.16 (from lancedb->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 11))\n"," Downloading lance_namespace-0.0.20-py3-none-any.whl.metadata (1.4 kB)\n","Collecting llama-index-cli<0.6,>=0.5.0 (from llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_index_cli-0.5.3-py3-none-any.whl.metadata (1.4 kB)\n","Collecting llama-index-core<0.15.0,>=0.14.7 (from llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_index_core-0.14.7-py3-none-any.whl.metadata (2.5 kB)\n","Collecting llama-index-indices-managed-llama-cloud>=0.4.0 (from llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_index_indices_managed_llama_cloud-0.9.4-py3-none-any.whl.metadata (3.7 kB)\n","Collecting llama-index-llms-openai<0.7,>=0.6.0 (from llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_index_llms_openai-0.6.6-py3-none-any.whl.metadata (3.0 kB)\n","Collecting llama-index-readers-file<0.6,>=0.5.0 (from llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_index_readers_file-0.5.4-py3-none-any.whl.metadata (5.7 kB)\n","Collecting llama-index-readers-llama-parse>=0.4.0 (from llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_index_readers_llama_parse-0.5.1-py3-none-any.whl.metadata (3.1 kB)\n","Collecting pylance (from llama-index-vector-stores-lancedb->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 13))\n"," Downloading pylance-0.38.3-cp39-abi3-manylinux_2_28_x86_64.whl.metadata (2.1 kB)\n","Collecting tantivy (from llama-index-vector-stores-lancedb->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 13))\n"," Downloading tantivy-0.25.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.4 kB)\n","Collecting llama-index-llms-openai-like<0.6,>=0.5.0 (from llama-index-llms-openrouter->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 17))\n"," Downloading llama_index_llms_openai_like-0.5.3-py3-none-any.whl.metadata (1.1 kB)\n","Requirement already satisfied: click in /usr/local/lib/python3.12/dist-packages (from nltk->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 18)) (8.3.0)\n","Requirement already satisfied: joblib in /usr/local/lib/python3.12/dist-packages (from nltk->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 18)) (1.5.2)\n","Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.12/dist-packages (from nltk->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 18)) (2024.11.6)\n","Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.12/dist-packages (from pandas->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 20)) (2.9.0.post0)\n","Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.12/dist-packages (from pandas->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 20)) (2025.2)\n","Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.12/dist-packages (from pandas->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 20)) (2025.2)\n","Requirement already satisfied: distro<2,>=1.7.0 in /usr/local/lib/python3.12/dist-packages (from openai->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 21)) (1.9.0)\n","Requirement already satisfied: jiter<1,>=0.4.0 in /usr/local/lib/python3.12/dist-packages (from openai->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 21)) (0.11.1)\n","Requirement already satisfied: sniffio in /usr/local/lib/python3.12/dist-packages (from openai->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 21)) (1.3.1)\n","Requirement already satisfied: more-itertools in /usr/local/lib/python3.12/dist-packages (from openai-whisper->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 22)) (10.8.0)\n","Requirement already satisfied: numba in /usr/local/lib/python3.12/dist-packages (from openai-whisper->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 22)) (0.60.0)\n","Requirement already satisfied: tiktoken in /usr/local/lib/python3.12/dist-packages (from openai-whisper->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 22)) (0.12.0)\n","Requirement already satisfied: torch in /usr/local/lib/python3.12/dist-packages (from openai-whisper->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 22)) (2.8.0+cu126)\n","Requirement already satisfied: triton>=2 in /usr/local/lib/python3.12/dist-packages (from openai-whisper->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 22)) (3.4.0)\n","Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.12/dist-packages (from pydantic->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 23)) (0.7.0)\n","Requirement already satisfied: pydantic-core==2.33.2 in /usr/local/lib/python3.12/dist-packages (from pydantic->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 23)) (2.33.2)\n","Requirement already satisfied: typing-inspection>=0.4.0 in /usr/local/lib/python3.12/dist-packages (from pydantic->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 23)) (0.4.2)\n","Requirement already satisfied: transformers<5.0.0,>=4.41.0 in /usr/local/lib/python3.12/dist-packages (from sentence-transformers->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 24)) (4.57.1)\n","Requirement already satisfied: scikit-learn in /usr/local/lib/python3.12/dist-packages (from sentence-transformers->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 24)) (1.6.1)\n","Requirement already satisfied: scipy in /usr/local/lib/python3.12/dist-packages (from sentence-transformers->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 24)) (1.16.3)\n","Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /usr/local/lib/python3.12/dist-packages (from spacy->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 26)) (3.0.12)\n","Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.12/dist-packages (from spacy->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 26)) (1.0.5)\n","Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.12/dist-packages (from spacy->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 26)) (1.0.13)\n","Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.12/dist-packages (from spacy->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 26)) (2.0.11)\n","Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.12/dist-packages (from spacy->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 26)) (3.0.10)\n","Requirement already satisfied: thinc<8.4.0,>=8.3.4 in /usr/local/lib/python3.12/dist-packages (from spacy->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 26)) (8.3.6)\n","Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /usr/local/lib/python3.12/dist-packages (from spacy->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 26)) (1.1.3)\n","Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.12/dist-packages (from spacy->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 26)) (2.5.1)\n","Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.12/dist-packages (from spacy->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 26)) (2.0.10)\n","Requirement already satisfied: weasel<0.5.0,>=0.1.0 in /usr/local/lib/python3.12/dist-packages (from spacy->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 26)) (0.4.1)\n","Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.12/dist-packages (from spacy->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 26)) (3.5.0)\n","Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.12/dist-packages (from anyio<5.0,>=3.0->gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (3.11)\n","Requirement already satisfied: annotated-doc>=0.0.2 in /usr/local/lib/python3.12/dist-packages (from fastapi<1.0,>=0.115.2->gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (0.0.3)\n","Requirement already satisfied: pyparsing<4,>=3.0.4 in /usr/local/lib/python3.12/dist-packages (from httplib2<1.0.0,>=0.19.0->google-api-python-client->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 3)) (3.2.5)\n","Requirement already satisfied: certifi in /usr/local/lib/python3.12/dist-packages (from httpx<1.0,>=0.24.1->gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (2025.10.5)\n","Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.12/dist-packages (from httpx<1.0,>=0.24.1->gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (1.0.9)\n","Requirement already satisfied: h11>=0.16 in /usr/local/lib/python3.12/dist-packages (from httpcore==1.*->httpx<1.0,>=0.24.1->gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (0.16.0)\n","Requirement already satisfied: aiohttp in /usr/local/lib/python3.12/dist-packages (from huggingface-hub[inference]>=0.19.0->llama-index-embeddings-huggingface->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 14)) (3.13.1)\n","Requirement already satisfied: parso<0.9.0,>=0.8.4 in /usr/local/lib/python3.12/dist-packages (from jedi>=0.16->ipython->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 10)) (0.8.5)\n","Requirement already satisfied: entrypoints in /usr/local/lib/python3.12/dist-packages (from jupyter-client>=6.1.12->ipykernel->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 9)) (0.4)\n","Requirement already satisfied: jupyter-core>=4.9.2 in /usr/local/lib/python3.12/dist-packages (from jupyter-client>=6.1.12->ipykernel->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 9)) (5.9.1)\n","Collecting lance-namespace-urllib3-client (from lance-namespace>=0.0.16->lancedb->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 11))\n"," Downloading lance_namespace_urllib3_client-0.0.20-py3-none-any.whl.metadata (22 kB)\n","Requirement already satisfied: language-data>=1.2 in /usr/local/lib/python3.12/dist-packages (from langcodes<4.0.0,>=3.2.0->spacy->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 26)) (1.3.0)\n","Collecting aiosqlite (from llama-index-core<0.15.0,>=0.14.7->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading aiosqlite-0.21.0-py3-none-any.whl.metadata (4.3 kB)\n","Collecting banks<3,>=2.2.0 (from llama-index-core<0.15.0,>=0.14.7->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading banks-2.2.0-py3-none-any.whl.metadata (12 kB)\n","Collecting dataclasses-json (from llama-index-core<0.15.0,>=0.14.7->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading dataclasses_json-0.6.7-py3-none-any.whl.metadata (25 kB)\n","Collecting deprecated>=1.2.9.3 (from llama-index-core<0.15.0,>=0.14.7->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading deprecated-1.3.1-py2.py3-none-any.whl.metadata (5.9 kB)\n","Collecting dirtyjson<2,>=1.0.8 (from llama-index-core<0.15.0,>=0.14.7->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading dirtyjson-1.0.8-py3-none-any.whl.metadata (11 kB)\n","Collecting filetype<2,>=1.2.0 (from llama-index-core<0.15.0,>=0.14.7->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading filetype-1.2.0-py2.py3-none-any.whl.metadata (6.5 kB)\n","Collecting llama-index-workflows!=2.9.0,<3,>=2 (from llama-index-core<0.15.0,>=0.14.7->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_index_workflows-2.10.2-py3-none-any.whl.metadata (6.5 kB)\n","Requirement already satisfied: networkx>=3.0 in /usr/local/lib/python3.12/dist-packages (from llama-index-core<0.15.0,>=0.14.7->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12)) (3.5)\n","Requirement already satisfied: platformdirs in /usr/local/lib/python3.12/dist-packages (from llama-index-core<0.15.0,>=0.14.7->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12)) (4.5.0)\n","Collecting setuptools>=18.5 (from ipython->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 10))\n"," Using cached setuptools-80.9.0-py3-none-any.whl.metadata (6.6 kB)\n","Requirement already satisfied: sqlalchemy>=1.4.49 in /usr/local/lib/python3.12/dist-packages (from sqlalchemy[asyncio]>=1.4.49->llama-index-core<0.15.0,>=0.14.7->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12)) (2.0.44)\n","Requirement already satisfied: tenacity!=8.4.0,<10.0.0,>=8.2.0 in /usr/local/lib/python3.12/dist-packages (from llama-index-core<0.15.0,>=0.14.7->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12)) (8.5.0)\n","Collecting typing-inspect>=0.8.0 (from llama-index-core<0.15.0,>=0.14.7->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB)\n","Requirement already satisfied: wrapt in /usr/local/lib/python3.12/dist-packages (from llama-index-core<0.15.0,>=0.14.7->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12)) (2.0.0)\n","Collecting deprecated>=1.2.9.3 (from llama-index-core<0.15.0,>=0.14.7->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading Deprecated-1.2.18-py2.py3-none-any.whl.metadata (5.7 kB)\n","Collecting llama-cloud==0.1.35 (from llama-index-indices-managed-llama-cloud>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_cloud-0.1.35-py3-none-any.whl.metadata (1.2 kB)\n","Collecting wrapt (from llama-index-core<0.15.0,>=0.14.7->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading wrapt-1.17.3-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl.metadata (6.4 kB)\n","Requirement already satisfied: defusedxml>=0.7.1 in /usr/local/lib/python3.12/dist-packages (from llama-index-readers-file<0.6,>=0.5.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12)) (0.7.1)\n","Collecting pypdf<7,>=5.1.0 (from llama-index-readers-file<0.6,>=0.5.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading pypdf-6.1.3-py3-none-any.whl.metadata (7.1 kB)\n","Collecting striprtf<0.0.27,>=0.0.26 (from llama-index-readers-file<0.6,>=0.5.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading striprtf-0.0.26-py3-none-any.whl.metadata (2.1 kB)\n","Collecting llama-parse>=0.5.0 (from llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_parse-0.6.77-py3-none-any.whl.metadata (6.6 kB)\n","Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.12/dist-packages (from pexpect>4.3->ipython->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 10)) (0.7.0)\n","Requirement already satisfied: wcwidth in /usr/local/lib/python3.12/dist-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 10)) (0.2.14)\n","Requirement already satisfied: pyasn1<0.7.0,>=0.6.1 in /usr/local/lib/python3.12/dist-packages (from pyasn1-modules>=0.2.1->google-auth->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 4)) (0.6.1)\n","Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.12/dist-packages (from python-dateutil>=2.8.2->pandas->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 20)) (1.17.0)\n","Requirement already satisfied: charset_normalizer<4,>=2 in /usr/local/lib/python3.12/dist-packages (from requests<3.0.0,>=2.18.0->google-api-core->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 2)) (3.4.4)\n","Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.12/dist-packages (from requests<3.0.0,>=2.18.0->google-api-core->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 2)) (2.5.0)\n","Requirement already satisfied: blis<1.4.0,>=1.3.0 in /usr/local/lib/python3.12/dist-packages (from thinc<8.4.0,>=8.3.4->spacy->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 26)) (1.3.0)\n","Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.12/dist-packages (from thinc<8.4.0,>=8.3.4->spacy->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 26)) (0.1.5)\n","Requirement already satisfied: sympy>=1.13.3 in /usr/local/lib/python3.12/dist-packages (from torch->openai-whisper->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 22)) (1.13.3)\n","Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.6.77 in /usr/local/lib/python3.12/dist-packages (from torch->openai-whisper->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 22)) (12.6.77)\n","Requirement already satisfied: nvidia-cuda-runtime-cu12==12.6.77 in /usr/local/lib/python3.12/dist-packages (from torch->openai-whisper->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 22)) (12.6.77)\n","Requirement already satisfied: nvidia-cuda-cupti-cu12==12.6.80 in /usr/local/lib/python3.12/dist-packages (from torch->openai-whisper->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 22)) (12.6.80)\n","Requirement already satisfied: nvidia-cudnn-cu12==9.10.2.21 in /usr/local/lib/python3.12/dist-packages (from torch->openai-whisper->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 22)) (9.10.2.21)\n","Requirement already satisfied: nvidia-cublas-cu12==12.6.4.1 in /usr/local/lib/python3.12/dist-packages (from torch->openai-whisper->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 22)) (12.6.4.1)\n","Requirement already satisfied: nvidia-cufft-cu12==11.3.0.4 in /usr/local/lib/python3.12/dist-packages (from torch->openai-whisper->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 22)) (11.3.0.4)\n","Requirement already satisfied: nvidia-curand-cu12==10.3.7.77 in /usr/local/lib/python3.12/dist-packages (from torch->openai-whisper->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 22)) (10.3.7.77)\n","Requirement already satisfied: nvidia-cusolver-cu12==11.7.1.2 in /usr/local/lib/python3.12/dist-packages (from torch->openai-whisper->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 22)) (11.7.1.2)\n","Requirement already satisfied: nvidia-cusparse-cu12==12.5.4.2 in /usr/local/lib/python3.12/dist-packages (from torch->openai-whisper->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 22)) (12.5.4.2)\n","Requirement already satisfied: nvidia-cusparselt-cu12==0.7.1 in /usr/local/lib/python3.12/dist-packages (from torch->openai-whisper->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 22)) (0.7.1)\n","Requirement already satisfied: nvidia-nccl-cu12==2.27.3 in /usr/local/lib/python3.12/dist-packages (from torch->openai-whisper->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 22)) (2.27.3)\n","Requirement already satisfied: nvidia-nvtx-cu12==12.6.77 in /usr/local/lib/python3.12/dist-packages (from torch->openai-whisper->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 22)) (12.6.77)\n","Requirement already satisfied: nvidia-nvjitlink-cu12==12.6.85 in /usr/local/lib/python3.12/dist-packages (from torch->openai-whisper->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 22)) (12.6.85)\n","Requirement already satisfied: nvidia-cufile-cu12==1.11.1.6 in /usr/local/lib/python3.12/dist-packages (from torch->openai-whisper->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 22)) (1.11.1.6)\n","Requirement already satisfied: tokenizers<=0.23.0,>=0.22.0 in /usr/local/lib/python3.12/dist-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 24)) (0.22.1)\n","Requirement already satisfied: safetensors>=0.4.3 in /usr/local/lib/python3.12/dist-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 24)) (0.6.2)\n","Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.12/dist-packages (from typer<1.0,>=0.12->gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (1.5.4)\n","Requirement already satisfied: rich>=10.11.0 in /usr/local/lib/python3.12/dist-packages (from typer<1.0,>=0.12->gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (13.9.4)\n","Requirement already satisfied: cloudpathlib<1.0.0,>=0.7.0 in /usr/local/lib/python3.12/dist-packages (from weasel<0.5.0,>=0.1.0->spacy->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 26)) (0.23.0)\n","Requirement already satisfied: smart-open<8.0.0,>=5.2.1 in /usr/local/lib/python3.12/dist-packages (from weasel<0.5.0,>=0.1.0->spacy->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 26)) (7.4.1)\n","Requirement already satisfied: llvmlite<0.44,>=0.43.0dev0 in /usr/local/lib/python3.12/dist-packages (from numba->openai-whisper->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 22)) (0.43.0)\n","Requirement already satisfied: threadpoolctl>=3.1.0 in /usr/local/lib/python3.12/dist-packages (from scikit-learn->sentence-transformers->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 24)) (3.6.0)\n","Requirement already satisfied: aiohappyeyeballs>=2.5.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp->huggingface-hub[inference]>=0.19.0->llama-index-embeddings-huggingface->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 14)) (2.6.1)\n","Requirement already satisfied: aiosignal>=1.4.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp->huggingface-hub[inference]>=0.19.0->llama-index-embeddings-huggingface->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 14)) (1.4.0)\n","Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp->huggingface-hub[inference]>=0.19.0->llama-index-embeddings-huggingface->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 14)) (25.4.0)\n","Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.12/dist-packages (from aiohttp->huggingface-hub[inference]>=0.19.0->llama-index-embeddings-huggingface->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 14)) (1.8.0)\n","Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.12/dist-packages (from aiohttp->huggingface-hub[inference]>=0.19.0->llama-index-embeddings-huggingface->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 14)) (6.7.0)\n","Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp->huggingface-hub[inference]>=0.19.0->llama-index-embeddings-huggingface->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 14)) (0.4.1)\n","Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp->huggingface-hub[inference]>=0.19.0->llama-index-embeddings-huggingface->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 14)) (1.22.0)\n","Collecting griffe (from banks<3,>=2.2.0->llama-index-core<0.15.0,>=0.14.7->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading griffe-1.14.0-py3-none-any.whl.metadata (5.1 kB)\n","Requirement already satisfied: marisa-trie>=1.1.0 in /usr/local/lib/python3.12/dist-packages (from language-data>=1.2->langcodes<4.0.0,>=3.2.0->spacy->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 26)) (1.3.1)\n","Collecting llama-index-instrumentation>=0.1.0 (from llama-index-workflows!=2.9.0,<3,>=2->llama-index-core<0.15.0,>=0.14.7->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_index_instrumentation-0.4.2-py3-none-any.whl.metadata (1.1 kB)\n","Collecting llama-cloud-services>=0.6.77 (from llama-parse>=0.5.0->llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_cloud_services-0.6.77-py3-none-any.whl.metadata (3.3 kB)\n","Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.12/dist-packages (from rich>=10.11.0->typer<1.0,>=0.12->gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (4.0.0)\n","Requirement already satisfied: greenlet>=1 in /usr/local/lib/python3.12/dist-packages (from sqlalchemy>=1.4.49->sqlalchemy[asyncio]>=1.4.49->llama-index-core<0.15.0,>=0.14.7->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12)) (3.2.4)\n","Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.12/dist-packages (from sympy>=1.13.3->torch->openai-whisper->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 22)) (1.3.0)\n","Collecting mypy-extensions>=0.3.0 (from typing-inspect>=0.8.0->llama-index-core<0.15.0,>=0.14.7->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading mypy_extensions-1.1.0-py3-none-any.whl.metadata (1.1 kB)\n","Collecting marshmallow<4.0.0,>=3.18.0 (from dataclasses-json->llama-index-core<0.15.0,>=0.14.7->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading marshmallow-3.26.1-py3-none-any.whl.metadata (7.3 kB)\n","INFO: pip is looking at multiple versions of llama-cloud-services to determine which version is compatible with other requirements. This could take a while.\n","Collecting llama-parse>=0.5.0 (from llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_parse-0.6.76-py3-none-any.whl.metadata (6.6 kB)\n","Collecting llama-cloud-services>=0.6.76 (from llama-parse>=0.5.0->llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_cloud_services-0.6.76-py3-none-any.whl.metadata (3.3 kB)\n","Collecting llama-parse>=0.5.0 (from llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_parse-0.6.75-py3-none-any.whl.metadata (6.6 kB)\n","Collecting llama-cloud-services>=0.6.75 (from llama-parse>=0.5.0->llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_cloud_services-0.6.75-py3-none-any.whl.metadata (3.3 kB)\n","Collecting llama-parse>=0.5.0 (from llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_parse-0.6.74-py3-none-any.whl.metadata (6.6 kB)\n","INFO: pip is still looking at multiple versions of llama-cloud-services to determine which version is compatible with other requirements. This could take a while.\n","Collecting llama-cloud-services>=0.6.74 (from llama-parse>=0.5.0->llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_cloud_services-0.6.74-py3-none-any.whl.metadata (3.3 kB)\n","Collecting llama-parse>=0.5.0 (from llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_parse-0.6.73-py3-none-any.whl.metadata (6.6 kB)\n","INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.\n","Collecting llama-cloud-services>=0.6.73 (from llama-parse>=0.5.0->llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_cloud_services-0.6.73-py3-none-any.whl.metadata (3.3 kB)\n","Collecting llama-parse>=0.5.0 (from llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_parse-0.6.72-py3-none-any.whl.metadata (6.6 kB)\n","Collecting llama-cloud-services>=0.6.72 (from llama-parse>=0.5.0->llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_cloud_services-0.6.72-py3-none-any.whl.metadata (3.3 kB)\n","Collecting llama-parse>=0.5.0 (from llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_parse-0.6.71-py3-none-any.whl.metadata (6.6 kB)\n","Collecting llama-cloud-services>=0.6.71 (from llama-parse>=0.5.0->llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_cloud_services-0.6.71-py3-none-any.whl.metadata (3.3 kB)\n","Collecting llama-parse>=0.5.0 (from llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_parse-0.6.70-py3-none-any.whl.metadata (6.6 kB)\n","Collecting llama-cloud-services>=0.6.70 (from llama-parse>=0.5.0->llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_cloud_services-0.6.70-py3-none-any.whl.metadata (3.3 kB)\n","Collecting llama-parse>=0.5.0 (from llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_parse-0.6.69-py3-none-any.whl.metadata (6.6 kB)\n","Collecting llama-cloud-services>=0.6.69 (from llama-parse>=0.5.0->llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_cloud_services-0.6.69-py3-none-any.whl.metadata (3.3 kB)\n","Collecting llama-parse>=0.5.0 (from llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_parse-0.6.68-py3-none-any.whl.metadata (6.6 kB)\n","Collecting llama-cloud-services>=0.6.68 (from llama-parse>=0.5.0->llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_cloud_services-0.6.68-py3-none-any.whl.metadata (3.3 kB)\n","Collecting llama-parse>=0.5.0 (from llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_parse-0.6.67-py3-none-any.whl.metadata (6.6 kB)\n","Collecting llama-cloud-services>=0.6.67 (from llama-parse>=0.5.0->llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_cloud_services-0.6.67-py3-none-any.whl.metadata (3.3 kB)\n","Collecting llama-parse>=0.5.0 (from llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_parse-0.6.66-py3-none-any.whl.metadata (6.6 kB)\n","Collecting llama-cloud-services>=0.6.66 (from llama-parse>=0.5.0->llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_cloud_services-0.6.66-py3-none-any.whl.metadata (3.3 kB)\n","Collecting llama-parse>=0.5.0 (from llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_parse-0.6.65-py3-none-any.whl.metadata (6.6 kB)\n","Collecting llama-cloud-services>=0.6.64 (from llama-parse>=0.5.0->llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_cloud_services-0.6.65-py3-none-any.whl.metadata (3.3 kB)\n"," Downloading llama_cloud_services-0.6.64-py3-none-any.whl.metadata (3.3 kB)\n","Collecting llama-parse>=0.5.0 (from llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_parse-0.6.64-py3-none-any.whl.metadata (6.6 kB)\n"," Downloading llama_parse-0.6.63-py3-none-any.whl.metadata (6.6 kB)\n","Collecting llama-cloud-services>=0.6.63 (from llama-parse>=0.5.0->llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_cloud_services-0.6.63-py3-none-any.whl.metadata (3.7 kB)\n","Collecting llama-parse>=0.5.0 (from llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_parse-0.6.62-py3-none-any.whl.metadata (6.6 kB)\n","Collecting llama-cloud-services>=0.6.62 (from llama-parse>=0.5.0->llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_cloud_services-0.6.62-py3-none-any.whl.metadata (3.7 kB)\n","Collecting llama-parse>=0.5.0 (from llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_parse-0.6.60-py3-none-any.whl.metadata (6.6 kB)\n","Collecting llama-cloud-services>=0.6.60 (from llama-parse>=0.5.0->llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_cloud_services-0.6.60-py3-none-any.whl.metadata (3.7 kB)\n","Collecting llama-parse>=0.5.0 (from llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_parse-0.6.59-py3-none-any.whl.metadata (6.6 kB)\n","Collecting llama-cloud-services>=0.6.59 (from llama-parse>=0.5.0->llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_cloud_services-0.6.59-py3-none-any.whl.metadata (3.7 kB)\n","Collecting llama-parse>=0.5.0 (from llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_parse-0.6.58-py3-none-any.whl.metadata (6.6 kB)\n","Collecting llama-cloud-services>=0.6.58 (from llama-parse>=0.5.0->llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_cloud_services-0.6.58-py3-none-any.whl.metadata (3.7 kB)\n","Collecting llama-parse>=0.5.0 (from llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_parse-0.6.57-py3-none-any.whl.metadata (6.6 kB)\n","Collecting llama-cloud-services>=0.6.56 (from llama-parse>=0.5.0->llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_cloud_services-0.6.57-py3-none-any.whl.metadata (3.7 kB)\n"," Downloading llama_cloud_services-0.6.56-py3-none-any.whl.metadata (3.7 kB)\n","Collecting llama-parse>=0.5.0 (from llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_parse-0.6.56-py3-none-any.whl.metadata (6.6 kB)\n"," Downloading llama_parse-0.6.55-py3-none-any.whl.metadata (6.6 kB)\n","Collecting llama-cloud-services>=0.6.55 (from llama-parse>=0.5.0->llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_cloud_services-0.6.55-py3-none-any.whl.metadata (3.7 kB)\n","Collecting llama-parse>=0.5.0 (from llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_parse-0.6.54-py3-none-any.whl.metadata (6.6 kB)\n","Collecting llama-cloud-services>=0.6.54 (from llama-parse>=0.5.0->llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading llama_cloud_services-0.6.54-py3-none-any.whl.metadata (3.6 kB)\n","Requirement already satisfied: python-dotenv<2,>=1.0.1 in /usr/local/lib/python3.12/dist-packages (from llama-cloud-services>=0.6.54->llama-parse>=0.5.0->llama-index-readers-llama-parse>=0.4.0->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12)) (1.2.1)\n","Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.12/dist-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->typer<1.0,>=0.12->gradio->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 6)) (0.1.2)\n","Collecting colorama>=0.4 (from griffe->banks<3,>=2.2.0->llama-index-core<0.15.0,>=0.14.7->llama-index->-r /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/requirements.txt (line 12))\n"," Downloading colorama-0.4.6-py2.py3-none-any.whl.metadata (17 kB)\n","Downloading lancedb-0.25.2-cp39-abi3-manylinux_2_28_x86_64.whl (38.7 MB)\n","\u001b[2K \u001b[90mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m \u001b[32m38.7/38.7 MB\u001b[0m \u001b[31m35.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hDownloading llama_index-0.14.7-py3-none-any.whl (7.4 kB)\n","Downloading llama_index_vector_stores_lancedb-0.4.1-py3-none-any.whl (7.9 kB)\n","Downloading llama_index_embeddings_huggingface-0.6.1-py3-none-any.whl (8.9 kB)\n","Downloading llama_index_llms_huggingface_api-0.6.1-py3-none-any.whl (7.5 kB)\n","Downloading llama_index_embeddings_openai-0.5.1-py3-none-any.whl (7.0 kB)\n","Downloading llama_index_llms_openrouter-0.4.2-py3-none-any.whl (4.5 kB)\n","Downloading yt_dlp-2025.10.22-py3-none-any.whl (3.2 MB)\n","\u001b[2K \u001b[90mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m \u001b[32m3.2/3.2 MB\u001b[0m \u001b[31m101.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hDownloading jedi-0.19.2-py2.py3-none-any.whl (1.6 MB)\n","\u001b[2K \u001b[90mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m65.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hDownloading lance_namespace-0.0.20-py3-none-any.whl (31 kB)\n","Downloading llama_index_cli-0.5.3-py3-none-any.whl (28 kB)\n","Downloading llama_index_core-0.14.7-py3-none-any.whl (11.9 MB)\n","\u001b[2K \u001b[90mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m \u001b[32m11.9/11.9 MB\u001b[0m \u001b[31m83.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hDownloading llama_index_indices_managed_llama_cloud-0.9.4-py3-none-any.whl (17 kB)\n","Downloading Deprecated-1.2.18-py2.py3-none-any.whl (10.0 kB)\n","Downloading llama_cloud-0.1.35-py3-none-any.whl (303 kB)\n","\u001b[2K \u001b[90mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m \u001b[32m303.3/303.3 kB\u001b[0m \u001b[31m20.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hDownloading llama_index_llms_openai-0.6.6-py3-none-any.whl (26 kB)\n","Downloading llama_index_llms_openai_like-0.5.3-py3-none-any.whl (4.7 kB)\n","Downloading llama_index_readers_file-0.5.4-py3-none-any.whl (51 kB)\n","\u001b[2K \u001b[90mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m \u001b[32m51.8/51.8 kB\u001b[0m \u001b[31m3.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hDownloading llama_index_readers_llama_parse-0.5.1-py3-none-any.whl (3.2 kB)\n","Downloading pylance-0.38.3-cp39-abi3-manylinux_2_28_x86_64.whl (48.0 MB)\n","\u001b[2K \u001b[90mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m \u001b[32m48.0/48.0 MB\u001b[0m \u001b[31m12.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hUsing cached setuptools-80.9.0-py3-none-any.whl (1.2 MB)\n","Downloading deprecation-2.1.0-py2.py3-none-any.whl (11 kB)\n","Downloading tantivy-0.25.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.1 MB)\n","\u001b[2K \u001b[90mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m \u001b[32m4.1/4.1 MB\u001b[0m \u001b[31m95.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hDownloading banks-2.2.0-py3-none-any.whl (29 kB)\n","Downloading dirtyjson-1.0.8-py3-none-any.whl (25 kB)\n","Downloading filetype-1.2.0-py2.py3-none-any.whl (19 kB)\n","Downloading llama_index_workflows-2.10.2-py3-none-any.whl (90 kB)\n","\u001b[2K \u001b[90mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m \u001b[32m90.7/90.7 kB\u001b[0m \u001b[31m6.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hDownloading llama_parse-0.6.54-py3-none-any.whl (4.9 kB)\n","Downloading llama_cloud_services-0.6.54-py3-none-any.whl (63 kB)\n","\u001b[2K \u001b[90mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m \u001b[32m63.9/63.9 kB\u001b[0m \u001b[31m4.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hDownloading pypdf-6.1.3-py3-none-any.whl (323 kB)\n","\u001b[2K \u001b[90mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m \u001b[32m323.9/323.9 kB\u001b[0m \u001b[31m21.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hDownloading striprtf-0.0.26-py3-none-any.whl (6.9 kB)\n","Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB)\n","Downloading wrapt-1.17.3-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl (88 kB)\n","\u001b[2K \u001b[90mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m \u001b[32m88.0/88.0 kB\u001b[0m \u001b[31m6.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hDownloading aiosqlite-0.21.0-py3-none-any.whl (15 kB)\n","Downloading dataclasses_json-0.6.7-py3-none-any.whl (28 kB)\n","Downloading lance_namespace_urllib3_client-0.0.20-py3-none-any.whl (229 kB)\n","\u001b[2K \u001b[90mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m \u001b[32m229.6/229.6 kB\u001b[0m \u001b[31m15.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hDownloading llama_index_instrumentation-0.4.2-py3-none-any.whl (15 kB)\n","Downloading marshmallow-3.26.1-py3-none-any.whl (50 kB)\n","\u001b[2K \u001b[90mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m \u001b[32m50.9/50.9 kB\u001b[0m \u001b[31m3.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hDownloading mypy_extensions-1.1.0-py3-none-any.whl (5.0 kB)\n","Downloading griffe-1.14.0-py3-none-any.whl (144 kB)\n","\u001b[2K \u001b[90mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m \u001b[32m144.4/144.4 kB\u001b[0m \u001b[31m10.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hDownloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)\n","Building wheels for collected packages: openai-whisper\n"," Building wheel for openai-whisper (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n"," Created wheel for openai-whisper: filename=openai_whisper-20250625-py3-none-any.whl size=803979 sha256=be017bdd70010807e561385dfd0514f077b15200b867f9378992997bd48a9fba\n"," Stored in directory: /root/.cache/pip/wheels/61/d2/20/09ec9bef734d126cba375b15898010b6cc28578d8afdde5869\n","Successfully built openai-whisper\n","Installing collected packages: striprtf, filetype, dirtyjson, yt-dlp, wrapt, tantivy, setuptools, pypdf, pylance, mypy-extensions, marshmallow, jedi, deprecation, colorama, aiosqlite, typing-inspect, griffe, deprecated, llama-index-instrumentation, llama-cloud, lance-namespace-urllib3-client, dataclasses-json, banks, openai-whisper, llama-index-workflows, lance-namespace, llama-index-core, lancedb, llama-index-vector-stores-lancedb, llama-index-readers-file, llama-index-llms-openai, llama-index-llms-huggingface-api, llama-index-indices-managed-llama-cloud, llama-index-embeddings-openai, llama-index-embeddings-huggingface, llama-cloud-services, llama-parse, llama-index-llms-openai-like, llama-index-cli, llama-index-readers-llama-parse, llama-index-llms-openrouter, llama-index\n"," Attempting uninstall: wrapt\n"," Found existing installation: wrapt 2.0.0\n"," Uninstalling wrapt-2.0.0:\n"," Successfully uninstalled wrapt-2.0.0\n"," Attempting uninstall: setuptools\n"," Found existing installation: setuptools 75.2.0\n"," Uninstalling setuptools-75.2.0:\n"," Successfully uninstalled setuptools-75.2.0\n","Successfully installed aiosqlite-0.21.0 banks-2.2.0 colorama-0.4.6 dataclasses-json-0.6.7 deprecated-1.2.18 deprecation-2.1.0 dirtyjson-1.0.8 filetype-1.2.0 griffe-1.14.0 jedi-0.19.2 lance-namespace-0.0.20 lance-namespace-urllib3-client-0.0.20 lancedb-0.25.2 llama-cloud-0.1.35 llama-cloud-services-0.6.54 llama-index-0.14.7 llama-index-cli-0.5.3 llama-index-core-0.14.7 llama-index-embeddings-huggingface-0.6.1 llama-index-embeddings-openai-0.5.1 llama-index-indices-managed-llama-cloud-0.9.4 llama-index-instrumentation-0.4.2 llama-index-llms-huggingface-api-0.6.1 llama-index-llms-openai-0.6.6 llama-index-llms-openai-like-0.5.3 llama-index-llms-openrouter-0.4.2 llama-index-readers-file-0.5.4 llama-index-readers-llama-parse-0.5.1 llama-index-vector-stores-lancedb-0.4.1 llama-index-workflows-2.10.2 llama-parse-0.6.54 marshmallow-3.26.1 mypy-extensions-1.1.0 openai-whisper-20250625 pylance-0.38.3 pypdf-6.1.3 setuptools-80.9.0 striprtf-0.0.26 tantivy-0.25.0 typing-inspect-0.9.0 wrapt-1.17.3 yt-dlp-2025.10.22\n"]},{"output_type":"display_data","data":{"application/vnd.colab-display-data+json":{"pip_warning":{"packages":["_distutils_hack"]},"id":"bf6145c7209b49e19021b33f4c254a27"}},"metadata":{}}]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"gtmrrU8ic4Ad","executionInfo":{"status":"ok","timestamp":1762069483035,"user_tz":-330,"elapsed":51999,"user":{"displayName":"","userId":""}},"outputId":"eff993f8-ffe4-47bc-a240-6b7070bb2425"},"outputs":[{"output_type":"stream","name":"stdout","text":["โ
Libraries imported successfully!\n"]}],"source":["# Import required libraries\n","import os\n","from pathlib import Path\n","from typing import List\n","from llama_index.core import SimpleDirectoryReader, VectorStoreIndex, StorageContext, Settings\n","from llama_index.vector_stores.lancedb import LanceDBVectorStore\n","from llama_index.embeddings.huggingface import HuggingFaceEmbedding\n","\n","print(\"โ
Libraries imported successfully!\")"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"IhBUV3rEc4Ae","executionInfo":{"status":"ok","timestamp":1762069961873,"user_tz":-330,"elapsed":2079,"user":{"displayName":"","userId":""}},"outputId":"7add2c19-d99f-49d6-e07d-e87cdc20e07d"},"outputs":[{"output_type":"stream","name":"stdout","text":["โ
OpenRouter API key found in Colab secrets\n","โ
LlamaIndex configured with local embeddings\n"," Using BAAI/bge-small-en-v1.5 for document embeddings\n"]}],"source":["# Configure LlamaIndex Settings (Using OpenRouter - No OpenAI API Key needed)\n","# Configure LlamaIndex Settings (Using OpenRouter - No OpenAI API Key needed)\n","def setup_llamaindex_settings():\n"," \"\"\"\n"," Configure LlamaIndex with local embeddings and OpenRouter for LLM.\n"," This assignment focuses on vector database operations, so we'll use local embeddings only.\n"," \"\"\"\n"," # Check for OpenRouter API key (for future use, not needed for this basic assignment)\n"," from google.colab import userdata\n","\n"," try:\n"," api_key = userdata.get('OPEN_ROUTER') # or whatever you named your secret\n"," print(\"โ
OpenRouter API key found in Colab secrets\")\n"," except Exception:\n"," print(\"โน๏ธ OPENROUTER_API_KEY not found - that's OK for this assignment!\")\n"," print(\" This assignment only uses local embeddings for vector operations.\")\n","\n"," # Configure local embeddings (no API key required)\n"," Settings.embed_model = HuggingFaceEmbedding(\n"," model_name=\"BAAI/bge-small-en-v1.5\",\n"," trust_remote_code=True\n"," )\n","\n"," print(\"โ
LlamaIndex configured with local embeddings\")\n"," print(\" Using BAAI/bge-small-en-v1.5 for document embeddings\")\n","\n","# Setup the configuration\n","setup_llamaindex_settings()"]},{"cell_type":"markdown","metadata":{"id":"tgAR2pDkc4Af"},"source":["## 1. Document Loading Function\n","\n","Complete the function below to load documents from a folder using `SimpleDirectoryReader`.\n","\n","**Note:** This assignment uses local embeddings only - no OpenAI API key required! We're configured to use OpenRouter for future LLM operations.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"afKvr3CHc4Ag","executionInfo":{"status":"ok","timestamp":1762070845711,"user_tz":-330,"elapsed":65372,"user":{"displayName":"","userId":""}},"outputId":"440cfccb-19b3-4b8a-fa53-556cea45660f"},"outputs":[{"output_type":"stream","name":"stderr","text":["100%|โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ| 139M/139M [00:00<00:00, 157MiB/s]\n","/usr/local/lib/python3.12/dist-packages/whisper/transcribe.py:132: UserWarning: FP16 is not supported on CPU; using FP32 instead\n"," warnings.warn(\"FP16 is not supported on CPU; using FP32 instead\")\n","/usr/local/lib/python3.12/dist-packages/whisper/transcribe.py:132: UserWarning: FP16 is not supported on CPU; using FP32 instead\n"," warnings.warn(\"FP16 is not supported on CPU; using FP32 instead\")\n","/usr/local/lib/python3.12/dist-packages/whisper/transcribe.py:132: UserWarning: FP16 is not supported on CPU; using FP32 instead\n"," warnings.warn(\"FP16 is not supported on CPU; using FP32 instead\")\n"]},{"output_type":"stream","name":"stdout","text":["TODO: Load documents from /content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/data\n","Loaded 42 documents\n"]}],"source":["from llama_index.core import SimpleDirectoryReader\n","\n","def load_documents_from_folder(folder_path: str):\n"," \"\"\"\n"," Load documents from a folder using SimpleDirectoryReader.\n","\n"," TODO: Complete this function to load documents from the given folder path.\n"," HINT: Use SimpleDirectoryReader with recursive parameter to load all files\n","\n"," Args:\n"," folder_path (str): Path to the folder containing documents\n","\n"," Returns:\n"," List of documents loaded from the folder\n"," \"\"\"\n"," # TODO: Create SimpleDirectoryReader instance\n"," reader = SimpleDirectoryReader(input_dir=folder_path, recursive=True)\n","\n"," # TODO: Load and return documents\n"," documents = reader.load_data()\n","\n","\n"," # PLACEHOLDER - Replace with actual implementation\n"," print(f\"TODO: Load documents from {folder_path}\")\n","\n"," return documents\n","\n","# Test the function after you complete it\n","test_folder = \"/content/drive/MyDrive/ai-accelerator-C2-main/Day_6/session_2/data\"\n","documents = load_documents_from_folder(test_folder)\n","print(f\"Loaded {len(documents)} documents\")\n"]},{"cell_type":"markdown","metadata":{"id":"DZ2L7z7ac4Ag"},"source":["## 2. Vector Store Creation Function\n","\n","Complete the function below to create a LanceDB vector store.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"m0KBU3uUc4Ah","executionInfo":{"status":"ok","timestamp":1762071429191,"user_tz":-330,"elapsed":323,"user":{"displayName":"","userId":""}},"outputId":"0af919ab-ec14-456e-dfff-f526b3e166b3"},"outputs":[{"output_type":"stream","name":"stderr","text":["WARNING:llama_index.vector_stores.lancedb.base:Table documents doesn't exist yet. Please add some data to create it.\n"]},{"output_type":"stream","name":"stdout","text":["TODO: Create vector store at ./assignment_vectordb\n","Vector store created: True\n"]}],"source":["def create_vector_store(db_path: str = \"./vectordb\", table_name: str = \"documents\"):\n"," \"\"\"\n"," Create a LanceDB vector store for storing document embeddings.\n","\n"," TODO: Complete this function to create and configure a LanceDB vector store.\n"," HINT: Use LanceDBVectorStore with uri and table_name parameters\n","\n"," Args:\n"," db_path (str): Path where the vector database will be stored\n"," table_name (str): Name of the table in the vector database\n","\n"," Returns:\n"," LanceDBVectorStore: Configured vector store\n"," \"\"\"\n"," # TODO: Create the directory if it doesn't exist\n"," Path(db_path).mkdir(parents=True, exist_ok=True)\n","\n"," # TODO: Create vector store\n"," vector_store = LanceDBVectorStore (\n"," uri=db_path,\n"," table_name=table_name\n"," )\n","\n"," # PLACEHOLDER - Replace with actual implementation\n"," print(f\"TODO: Create vector store at {db_path}\")\n","\n"," return vector_store\n","\n","\n","\n","# Test the function after you complete it\n","vector_store = create_vector_store(\"./assignment_vectordb\")\n","print(f\"Vector store created: {vector_store is not None}\")\n"]},{"cell_type":"markdown","metadata":{"id":"g3X5bjtIc4Ah"},"source":["## 3. Vector Index Creation Function\n","\n","Complete the function below to create a vector index from documents.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":116,"referenced_widgets":["c800f798befc49ec85ac21296ae3ede3","b6b184be73b64f52b5229efca5b7b535","a06c7b0401ab42658f5fceeb02719936","4c709bca2e5b45e18c1052b1443040b9","a2dd38f7ea6a4b0aaf4fa9d9d75bb2f7","7fb84c2a901a4d298d3733a3e3067a07","288422a6ec0d4266b9aa42f9a813fd1f","ed98766b22fb44f1a9b4099e6f5c6e8b","7381b62a3d3b4040adf365e00ce5d0ee","bc98f581514549cabc2bac83d017cf5a","f368138346d04fffb0a7fa1faa5986ec","9a3658c2ed064f579537fe277cc81243","5e1d55b3cf2a442e82108bedb94ac345","e0e0fef0e6934b728298d880f3fcfc24","d0dd192a6edb42d2ab6c3e1e96843a6a","2d06766fb0f14899a1c96b11bf04beb3","2b1872425a894d6298193dfa433fe76d","c50c88d75b9b41fdaa8d66fc24b3391f","2619ccd9ecb14392a78eb8469fd7de39","bef8337bdbcf451995c6ae2db1c94e68","b8a2dc32c1d44f3ebe8fa919aa69ccdc","4fb1db5731ed4e1dbf9f0c159febd03f"]},"id":"cDeK2JhSc4Ai","executionInfo":{"status":"ok","timestamp":1762071914576,"user_tz":-330,"elapsed":42292,"user":{"displayName":"","userId":""}},"outputId":"9cd990d6-1874-46ea-9a12-4e1afd55750a"},"outputs":[{"output_type":"display_data","data":{"text/plain":["Parsing nodes: 0%| | 0/42 [00:00 402\u001b[0;31m \u001b[0mresponse\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraise_for_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 403\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mHTTPError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.12/dist-packages/requests/models.py\u001b[0m in \u001b[0;36mraise_for_status\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 1025\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mhttp_error_msg\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1026\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mHTTPError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhttp_error_msg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1027\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;31mHTTPError\u001b[0m: 401 Client Error: Unauthorized for url: https://huggingface.co/api/whoami-v2","\nThe above exception was the direct cause of the following exception:\n","\u001b[0;31mHfHubHTTPError\u001b[0m Traceback (most recent call last)","\u001b[0;32m/usr/local/lib/python3.12/dist-packages/huggingface_hub/hf_api.py\u001b[0m in \u001b[0;36mwhoami\u001b[0;34m(self, token)\u001b[0m\n\u001b[1;32m 1799\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1800\u001b[0;31m \u001b[0mhf_raise_for_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1801\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mHTTPError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_http.py\u001b[0m in \u001b[0;36mhf_raise_for_status\u001b[0;34m(response, endpoint_name)\u001b[0m\n\u001b[1;32m 474\u001b[0m \u001b[0;31m# as well (request id and/or server error message)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 475\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0m_format\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mHfHubHTTPError\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 476\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;31mHfHubHTTPError\u001b[0m: 401 Client Error: Unauthorized for url: https://huggingface.co/api/whoami-v2 (Request ID: Root=1-69075383-79acb4c72606ee2b66a2018e;4b91b747-f307-4671-9ee1-87bbb3f6fe74)\n\nInvalid credentials in Authorization header","\nThe above exception was the direct cause of the following exception:\n","\u001b[0;31mHTTPError\u001b[0m Traceback (most recent call last)","\u001b[0;32m/tmp/ipython-input-2265489888.py\u001b[0m in \u001b[0;36m{msg['message']} -
-
-
-
-
- {/* Form Card */}
-
Viral LinkedIn Post Generator
-- Fill the fields, click Generate to draft a - LinkedIn post, edit if needed, then Publish. -
-
-
-
-
- {/* Actions */}
-
- Theme / Idea *
- setTheme(e.target.value)}
- />
-
-
-
- Post Type / Category *
-
-
-
-
- Length *
-
-
-
-
- Tone *
-
-
-
-
-
- {!valid && (
- All fields are required.
- )}
-
-
- {/* Generated Post */}
-
- Generated LinkedIn Post
-