Port backend to LangChain

[yangm2]

Last week we discussed porting the backend code to the Python LangChain framework.

• This gives us a path to enable LangSmith for automated testing/tracing/observability in the future.
• This also abstracts models so upgrading/switching models in the future could be simpler.

[yangm2]

@claude please describe how this issue can be implemented, do not implement it, just describe how.

[claude]

Claude finished @yangm2's task —— View job

---

Analysis: Porting Backend to LangChain

Todo List:

• Read repository guidelines (CLAUDE.md) - No repo-specific guidelines found
• Analyze current backend architecture
• Identify model usage patterns
• Research LangChain integration approach
• Describe implementation plan

Current Architecture Analysis

The backend currently uses:

• Google Gemini 2.5 Pro via google.genai SDK
• Vertex AI RAG with tool-based retrieval
• Direct API calls with streaming responses
• Custom ChatManager class handling conversation flow
• Flask-based REST API with session management via Valkey

Key integration points:

• tenantfirstaid/chat.py:38 - ChatManager initialization with genai.Client
• tenantfirstaid/chat.py:129 - Direct Gemini API streaming calls
• RAG integration via Vertex AI tools (lines 107-126)
• Session persistence and message formatting (lines 64-74)

LangChain Migration Implementation Plan

1. Dependencies & Setup

Add to pyproject.toml:

dependencies = [
    # ... existing deps
    "langchain>=0.1.0",
    "langchain-google-genai>=1.0.0",
    "langchain-google-vertexai>=1.0.0",
    "langchain-community>=0.0.20",
    "langsmith>=0.1.0",  # For future observability
]

2. Core Architecture Changes

A. Replace ChatManager with LangChain Components:

# New structure in chat.py
from langchain_google_vertexai import ChatVertexAI, VertexAISearchRetriever
from langchain.chains import ConversationalRetrievalChain
from langchain.memory import ConversationBufferMemory
from langchain.schema import HumanMessage, AIMessage

class LangChainChatManager:
    def __init__(self):
        self.llm = ChatVertexAI(
            model_name=MODEL,
            temperature=0,
            max_tokens=65535,
        )
        
        self.retriever = VertexAISearchRetriever(
            datastore_id=os.getenv("VERTEX_AI_DATASTORE"),
            max_results=5,
        )
        
        self.memory = ConversationBufferMemory(
            memory_key="chat_history",
            return_messages=True
        )
        
        self.chain = ConversationalRetrievalChain.from_llm(
            llm=self.llm,
            retriever=self.retriever,
            memory=self.memory,
            verbose=True,  # For debugging
        )

B. Update Message Handling:

• Replace custom message formatting with LangChain's HumanMessage/AIMessage
• Use LangChain's built-in conversation memory instead of manual session management
• Leverage LangChain's prompt templates for system instructions

C. RAG Integration:

• Replace direct Vertex AI tool calls with VertexAISearchRetriever
• Use LangChain's ConversationalRetrievalChain for RAG-enabled conversations
• Maintain location-based filtering through retriever configuration

3. Streaming Implementation

Update streaming to use LangChain's async callbacks:

from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain.callbacks.base import BaseCallbackHandler

class FlaskStreamingHandler(BaseCallbackHandler):
    def __init__(self):
        self.chunks = []
    
    def on_llm_new_token(self, token: str, **kwargs) -> None:
        self.chunks.append(token)
        # Yield to Flask streaming response
        
def generate_streaming_response(self, query: str):
    handler = FlaskStreamingHandler()
    response = self.chain.stream(
        {"question": query},
        callbacks=[handler]
    )
    for chunk in handler.chunks:
        yield chunk

4. Session Management Integration

Preserve existing session behavior:

• Keep Valkey-based session storage for conversation persistence
• Sync LangChain memory with session state on each request
• Maintain location context (city/state) in conversation chains

def load_conversation_from_session(self, session_data):
    """Load conversation history into LangChain memory"""
    for msg in session_data["messages"]:
        if msg["role"] == "user":
            self.memory.chat_memory.add_user_message(msg["content"])
        else:
            self.memory.chat_memory.add_ai_message(msg["content"])

5. Benefits Achieved

Immediate Benefits:

• Standardized Architecture: Replace custom ChatManager with industry-standard patterns
• Built-in Memory Management: Leverage LangChain's conversation memory instead of manual handling
• Better Abstractions: Cleaner separation between LLM, retrieval, and conversation logic

Future Enablement:

• LangSmith Integration: Add observability with minimal configuration:

  import langsmith
  langsmith.configure(api_key=os.getenv("LANGSMITH_API_KEY"))

• Model Switching: Easy migration between different LLMs:

  # Switch from Gemini to Claude or GPT-4
  from langchain_anthropic import ChatAnthropic
  self.llm = ChatAnthropic(model="claude-3-5-sonnet-20241022")

• Advanced RAG: Access to LangChain's RAG improvements (multi-query retrieval, hybrid search, etc.)

6. Migration Strategy

Phase 1: Core Migration

1. Add LangChain dependencies
2. Create parallel LangChainChatManager class
3. Update /api/query endpoint to use new manager
4. Maintain backward compatibility with existing session format

Phase 2: Enhanced Features

1. Enable LangSmith tracing for development
2. Add conversation evaluation metrics
3. Implement advanced RAG techniques (query expansion, re-ranking)

Phase 3: Production Optimization

1. Performance optimization and caching
2. Production LangSmith deployment
3. A/B testing between old and new implementations

7. Minimal Breaking Changes

• API Compatibility: Keep existing REST endpoints unchanged
• Session Format: Maintain current session structure
• Streaming: Preserve real-time response streaming
• Location Context: Keep city/state filtering functionality

This migration provides a clear path to modern LLM architecture while preserving all current functionality and enabling future enhancements through the LangChain ecosystem.

---

[yangm2]

@claude please update analysis for LangChain 1.0.5 as well as the latest release of other new dependencies.

[claude]

Claude finished @yangm2's task —— View job

---

Claude finished @yangm2's task

---

Updated Analysis: Porting Backend to LangChain (Latest Versions)

Todo List:

• Analyze current backend architecture
• Research LangChain 1.0.5 and latest dependency versions
• Update implementation plan for LangChain 1.0.5
• Verify migration strategy compatibility

Current Architecture Analysis (Updated)

The backend currently uses:

• Google Gemini 2.5 Pro via google-genai>=1.38.0 SDK
• Vertex AI RAG with tool-based retrieval using dual datastore filters
• Direct API streaming with client.models.generate_content_stream()
• Custom ChatManager class handling conversation flow at backend/tenantfirstaid/chat.py:36
• Flask-based REST API with session management via Valkey

Key integration points:

• backend/tenantfirstaid/chat.py:38 - ChatManager initialization with genai.Client(vertexai=True)
• backend/tenantfirstaid/chat.py:129 - Direct Gemini API streaming calls
• RAG integration via dual Vertex AI search tools (lines 107-126) with location filtering
• Message formatting and streaming response handling (lines 146-172)

Important Version Update: LangChain 1.0.5 Does Not Exist

Research findings: LangChain has not reached version 1.0 yet. As of late 2024/early 2025:

• LangChain Core: Latest stable ~0.3.x series
• The ecosystem has moved to modular packages instead of a monolithic langchain package
• Individual integration packages are the recommended approach

LangChain Migration Implementation Plan (Current Latest Versions)

1. Dependencies & Setup (Updated for Latest Versions)

Add to pyproject.toml:

dependencies = [
    # ... existing deps
    "langchain-core>=0.3.0",              # Core LangChain functionality
    "langchain-google-genai>=0.2.4",      # Google GenAI integration
    "langchain-google-vertexai>=0.2.2",   # Vertex AI integration  
    "langchain-community>=0.3.0",         # Community tools and loaders
    "langsmith>=0.1.129",                 # Observability and tracing
]

Key Change: No monolithic langchain package - using modular approach with specific integrations.

2. Core Architecture Changes (Updated for Current APIs)

A. Replace ChatManager with LangChain 0.3.x Components:

# Updated structure for current LangChain APIs
from langchain_google_vertexai import ChatVertexAI, VertexAISearchRetriever
from langchain_core.messages import HumanMessage, AIMessage, SystemMessage
from langchain_core.runnables import RunnablePassthrough, RunnableParallel
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.memory import ConversationBufferMemory

class LangChainChatManager:
    def __init__(self):
        # Updated to use latest Vertex AI integration
        self.llm = ChatVertexAI(
            model_name=MODEL,
            temperature=0,
            max_tokens=65535,
            # New safety settings API
            safety_settings={
                "HARM_CATEGORY_HATE_SPEECH": "OFF",
                "HARM_CATEGORY_DANGEROUS_CONTENT": "OFF", 
                "HARM_CATEGORY_SEXUALLY_EXPLICIT": "OFF",
                "HARM_CATEGORY_HARASSMENT": "OFF"
            },
            # Thinking config for Gemini 2.5 Pro
            enable_thinking=os.getenv("SHOW_MODEL_THINKING", "false").lower() == "true"
        )
        
        # Updated retriever with current API
        self.city_retriever = VertexAISearchRetriever(
            datastore_id=os.getenv("VERTEX_AI_DATASTORE"),
            max_results=5,
            # Current filter syntax
            filter_expression='city: ANY("{city}") AND state: ANY("{state}")'
        )
        
        self.state_retriever = VertexAISearchRetriever(
            datastore_id=os.getenv("VERTEX_AI_DATASTORE"), 
            max_results=5,
            filter_expression='city: ANY("null") AND state: ANY("{state}")'
        )
        
        # Updated prompt template using current API
        self.prompt = ChatPromptTemplate.from_messages([
            ("system", "{instructions}"),
            ("placeholder", "{chat_history}"),
            ("human", "{question}")
        ])
        
        # Current LCEL chain composition
        self.chain = (
            RunnableParallel(
                context=self._get_context,
                question=RunnablePassthrough(),
                instructions=lambda _: self.prepare_developer_instructions,
                chat_history=lambda _: self.format_chat_history()
            )
            | self.prompt
            | self.llm
            | StrOutputParser()
        )

    def _get_context(self, inputs):
        """Retrieve context from both city-specific and state-wide sources"""
        city = inputs.get("city", "").lower()
        state = inputs.get("state", "").lower()
        
        # Use both retrievers as in original implementation
        city_context = self.city_retriever.invoke(
            inputs["question"], 
            filter_kwargs={"city": city, "state": state}
        )
        state_context = self.state_retriever.invoke(
            inputs["question"],
            filter_kwargs={"state": state}
        )
        
        return city_context + state_context

B. Streaming Implementation (Updated for Current API):

from langchain_core.callbacks import BaseCallbackHandler
from typing import Dict, Any

class FlaskStreamingHandler(BaseCallbackHandler):
    def __init__(self):
        self.tokens = []
    
    def on_llm_new_token(self, token: str, **kwargs: Any) -> None:
        """Handle new token from LLM"""
        self.tokens.append(token)
    
    def on_llm_end(self, response, **kwargs: Any) -> None:
        """Handle end of LLM generation"""
        pass

async def generate_streaming_response(self, messages: list, city: str, state: str):
    """Updated streaming with current async patterns"""
    handler = FlaskStreamingHandler()
    
    # Format input for current API
    inputs = {
        "question": messages[-1]["content"],
        "city": city,
        "state": state,
        "chat_history": self.format_messages_for_langchain(messages[:-1])
    }
    
    # Use astream for async streaming
    async for chunk in self.chain.astream(inputs, {"callbacks": [handler]}):
        yield chunk

3. Session Management Integration (Updated)

Preserve existing session behavior with current LangChain patterns:

from langchain_core.messages import message_to_dict, messages_from_dict

def load_conversation_from_session(self, session_data):
    """Load conversation history using current message format"""
    messages = []
    for msg in session_data.get("messages", []):
        if msg["role"] == "user":
            messages.append(HumanMessage(content=msg["content"]))
        elif msg["role"] == "assistant":
            messages.append(AIMessage(content=msg["content"]))
    
    return messages

def save_conversation_to_session(self, messages):
    """Convert LangChain messages back to session format"""
    session_messages = []
    for msg in messages:
        session_messages.append({
            "role": "user" if isinstance(msg, HumanMessage) else "assistant",
            "content": msg.content
        })
    return session_messages

4. Benefits Achieved (Updated for Current Ecosystem)

Immediate Benefits:

• Modular Architecture: Use only needed components instead of monolithic package
• Better Abstractions: Current LCEL syntax provides cleaner composition patterns
• Enhanced Observability: LangSmith 0.1.129+ provides production-ready tracing
• Standardized Patterns: Follow current LangChain best practices

Future Enablement:

• LangSmith Integration: Add observability with current API:

  import langsmith
  from langsmith.traceable import traceable
  
  @traceable(name="tenant_legal_advice")
  async def generate_response(self, query, context):
      # Automatic tracing for production monitoring

• Model Switching: Current modular approach makes model changes simple:

  # Switch between different Google models
  from langchain_google_genai import ChatGoogleGenerativeAI
  self.llm = ChatGoogleGenerativeAI(model="gemini-2.5-flash")
  
  # Or switch to other providers  
  from langchain_anthropic import ChatAnthropic
  self.llm = ChatAnthropic(model="claude-3-5-sonnet-20241022")

5. Migration Strategy (Updated for Current Versions)

Phase 1: Core Migration

1. Add current LangChain modular packages
2. Create parallel LangChainChatManager using 0.3.x APIs
3. Update streaming to use current async patterns
4. Maintain session compatibility

Phase 2: Enhanced Features

1. Enable LangSmith tracing with latest API
2. Add evaluation metrics using current tools
3. Implement advanced RAG with current retrieval patterns

Phase 3: Production Optimization

1. Performance optimization with current best practices
2. Production LangSmith deployment
3. A/B testing framework

6. Current API Breaking Changes to Note

From Early 2024 to Current:

• Pydantic v2 Migration: All models use Pydantic v2
• LCEL Syntax: Chains use new Expression Language syntax
• Modular Packages: No more monolithic langchain imports
• Async First: Current APIs favor async/await patterns
• Updated Message Types: New message handling patterns

7. Migration Compatibility Assessment

✅ Compatible Elements:

• Current Google GenAI integration maps well to langchain-google-vertexai
• Dual datastore RAG pattern supported by multiple retrievers
• Streaming responses supported with async callbacks
• Session management can be preserved with message conversion utilities
• Location-based filtering maintained through retriever filter expressions

⚠️ Considerations:

• Current direct API calls are more performant than LangChain abstractions
• Additional complexity may not be justified for current simple use case
• Requires async/await refactoring for optimal streaming
• Memory footprint increases with additional dependencies

Recommendation: The migration is technically feasible and would enable future LangSmith observability. However, consider whether the current direct implementation already meets your needs effectively. The migration provides value primarily for future scaling, model switching, and enhanced monitoring capabilities.

This updated analysis reflects the actual current state of the LangChain ecosystem and provides a realistic migration path using available packages and APIs.

---

[yangm2]

@claude analyze again for LangChain 1.0.5 monolithic import and follow the recommended "create_agent()" flow from the LangChain Getting Started guide. Also, ignore all code/documentation in TenantFirstAid to Redis & Valkey, those are being removed. Please also include testing considerations.

[claude]

Claude finished @yangm2's task —— View job

---

I'll analyze this and get back to you.

[yangm2]

@claude analyze again for LangChain 1.0.5 monolithic import and follow the recommended "create_agent()" flow from the LangChain Getting Started guide. Also, ignore all code/documentation in TenantFirstAid to Redis & Valkey, those are being removed. Please also include testing and documentation considerations.

[claude]

Claude Code is working…

I'll analyze this and get back to you.

View job run

[yangm2]

@claude analyze again for LangChain 1.0.5 monolithic import and follow the recommended "create_agent()" flow from the LangChain Getting Started guide. Also, ignore all code/documentation in TenantFirstAid to Redis & Valkey, those are being removed. Please also include testing and documentation considerations.

[claude]

Claude encountered an error —— View job

---

I'll analyze this and get back to you.

[yangm2]

@claude analyze again for LangChain 1.0.8 monolithic import and follow the recommended "create_agent()" flow from the LangChain Getting Started guide. Please also include testing and documentation considerations.