Skip to content

Latest commit

 

History

History
173 lines (122 loc) · 4.96 KB

RELEVANCE_FILTERING_GUIDE.md

File metadata and controls

173 lines (122 loc) · 4.96 KB

🎯 Relevance Filtering - Enhanced Citation System

✅ Problem Solved

Your Agentic RAG system now implements intelligent relevance filtering that ensures only citations and sources that are actually relevant to the user's query are displayed.

🔍 How Relevance Filtering Works

1. PDF Content Relevance Scoring

def calculate_text_relevance(query: str, text: str, threshold: float = 0.2) -> float:
    """Calculate relevance score between query and text"""
    query_terms = query.lower().split()
    text_lower = text.lower()
    matches = sum(1 for term in query_terms if term in text_lower)
    return matches / len(query_terms) if query_terms else 0

Thresholds:

  • PDF chunks: 15% relevance required (0.15)
  • Google Drive docs: 10% relevance required (0.10)
  • Web search results: 20% relevance required (0.20)

2. Multi-Source Filtering

PDF Document Filtering:

  • Analyzes each retrieved chunk for query term matches
  • Only includes chunks with ≥15% relevance score
  • Shows filtered count: Relevant PDF chunks: 3/8

Google Drive Filtering:

  • Checks document name + content for relevance
  • Only includes documents with ≥10% relevance score
  • Shows filtered count: Relevant Google Drive docs: 2/5

Web Search Filtering:

  • Analyzes title + snippet for query terms
  • Only includes results with ≥20% relevance score
  • Returns "No relevant web search results" if none qualify

📊 Before vs After Comparison

Before Filtering:

Query: "What is the weather in Tokyo?"
Citations: [1] PDF Page 1, [2] PDF Page 2, [3] PDF Page 3, [4] Web Search
Problem: PDF citations are irrelevant to weather query

After Filtering:

Query: "What is the weather in Tokyo?"
Citations: [1] Web Search
Result: Only relevant weather information from web search

🎯 Example Scenarios

Scenario 1: External Query

  • Query: "What's the current Bitcoin price?"
  • PDF Relevance: 0.0 (no matching terms)
  • Result: Only web search citation shown
  • Citations: [1] Web Search Result

Scenario 2: Document Query

  • Query: "What methodology is described in section 3?"
  • PDF Relevance: 0.8 (high matching terms)
  • Result: Only relevant PDF pages shown
  • Citations: [1] PDF Page 3, [2] PDF Page 4

Scenario 3: Mixed Query

  • Query: "How does the document compare to industry standards?"
  • PDF Relevance: 0.5 (partial match)
  • Result: Relevant PDF + current web info
  • Citations: [1] PDF Page 2, [2] Web Search

📈 Relevance Metrics Provided

{
  "relevance_info": {
    "pdf_relevance_score": 0.75,
    "total_pdf_chunks_found": 8,
    "relevant_pdf_chunks": 3,
    "relevant_drive_docs": 1,
    "web_search_used": true
  }
}

🔄 Smart Citation Management

Citation Validation:

  1. Generate answer with all available sources
  2. Extract mentioned citation numbers from response
  3. Only include citations actually referenced in answer
  4. Filter citations by relevance scores
  5. Provide clean, focused citation list

Source Count Accuracy:

"sources_used": {
    "pdf_documents": len(relevant_docs),      # Only relevant chunks
    "google_drive_docs": len(drive_results),  # Only relevant docs
    "web_search": 1 if web_context else 0    # Only if relevant
}

✅ Benefits

For Users:

  • 🎯 Focused Results: Only see relevant information
  • 🚀 Faster Reading: No irrelevant citations to filter through
  • 📊 Trust: Higher confidence in source relevance
  • 🔍 Clarity: Clear understanding of information sources

For System:

  • ⚡ Performance: Reduced processing of irrelevant content
  • 🎪 Accuracy: Better response quality
  • 📈 Intelligence: Context-aware source selection
  • 🔄 Efficiency: Optimal resource utilization

🧪 Testing Results

Query Type Testing:

  • ✅ External queries → No irrelevant PDF citations
  • ✅ Document queries → Only relevant PDF pages
  • ✅ Mixed queries → Balanced relevant sources
  • ✅ Specific queries → Highly targeted results

Relevance Filtering:

  • ✅ PDF chunks: 15% threshold working
  • ✅ Google Drive: 10% threshold working
  • ✅ Web search: 20% threshold working
  • ✅ Accurate source counting implemented

🎉 Key Improvements

1. Intelligent Source Selection

  • Automatically determines which sources are relevant
  • No more irrelevant PDF chunks for external queries
  • No more irrelevant web results for document queries

2. Relevance Transparency

  • Shows relevance scores for each citation
  • Provides filtering statistics
  • Clear reasoning for source selection

3. Clean Citation Lists

  • Only includes citations mentioned in response
  • Filters by relevance thresholds
  • Accurate source counts and types

Your Agentic RAG system now provides precisely relevant information with clean, focused citations that directly support the user's query! 🚀