feat: Increase MAX_CONTENT_LENGTH to 50,000 chars and make it configurable

README.md

@@ -105,18 +105,27 @@ make mcp        # Start MCP server
 | Variable | Default | Description |
 |----------|---------|-------------|
 | `SERPER_API_KEY` | *(none)* | Serper API key for premium search (optional, falls back to DuckDuckGo if not set) |
+| `PERPLEXITY_API_KEY` | *(none)* | Perplexity API key for deep research tool (optional, get at https://www.perplexity.ai/settings/api) |
 | `WEBCAT_API_KEY` | *(none)* | Bearer token for authentication (optional, if set all requests must include `Authorization: Bearer <token>`) |
 | `PORT` | `8000` | Server port |
 | `LOG_LEVEL` | `INFO` | Logging level (DEBUG, INFO, WARNING, ERROR) |
 | `LOG_DIR` | `/tmp` | Log file directory |
+| `MAX_CONTENT_LENGTH` | `1000000` | Maximum characters to return per scraped article |
 
-### Get a Serper API Key
+### Get API Keys
 
+**Serper API (for web search):**
 1. Visit [serper.dev](https://serper.dev)
 2. Sign up for free tier (2,500 searches/month)
 3. Copy your API key
 4. Add to `.env` file: `SERPER_API_KEY=your_key`
 
+**Perplexity API (for deep research):**
+1. Visit [perplexity.ai/settings/api](https://www.perplexity.ai/settings/api)
+2. Sign up and get your API key
+3. Copy your API key
+4. Add to `.env` file: `PERPLEXITY_API_KEY=your_key`
+
 ### Enable Authentication (Optional)
 
 To require bearer token authentication for all MCP tool calls:

docker/.env.example

@@ -2,6 +2,11 @@
 # If not set, DuckDuckGo fallback will be used
 SERPER_API_KEY=
 
+# Perplexity API key for deep research (optional)
+# Required for the deep_research MCP tool
+# Get your API key at: https://www.perplexity.ai/settings/api
+PERPLEXITY_API_KEY=
+
 # WebCat API key for bearer token authentication (optional)
 # If set, all requests must include: Authorization: Bearer <token>
 # If not set, no authentication is required
@@ -12,6 +17,9 @@ PORT=8000
 LOG_LEVEL=INFO
 LOG_DIR=/tmp
 
+# Content limits
+MAX_CONTENT_LENGTH=1000000
+
 # Rate limiting
 RATE_LIMIT_WINDOW=60
 RATE_LIMIT_MAX_REQUESTS=10

docker/README.md

@@ -50,9 +50,11 @@ docker-compose up
 ### Environment Variables
 
 - `SERPER_API_KEY`: **Optional** - Serper API key for premium search (falls back to DuckDuckGo if not set)
+- `PERPLEXITY_API_KEY`: **Optional** - Perplexity API key for deep research tool (get key at https://www.perplexity.ai/settings/api)
 - `PORT`: Port to run the server on (default: 8000)
 - `LOG_LEVEL`: Logging level (default: INFO)
 - `LOG_DIR`: Directory for log files (default: /tmp)
+- `MAX_CONTENT_LENGTH`: Maximum characters to return per scraped article (default: 1000000)
 
 ### Simplified Setup
 
@@ -78,7 +80,14 @@ The server runs on **FastMCP** and exposes MCP protocol endpoints:
    - Falls back to DuckDuckGo automatically
    - Returns full webpage content in markdown format
 
-2. **`health_check`** - Check server health status
+2. **`deep_research`** - Comprehensive deep research (NEW!)
+   - Uses Perplexity AI's sonar-deep-research model
+   - Performs dozens of searches and reads hundreds of sources
+   - Synthesizes findings into comprehensive reports
+   - Takes 2-4 minutes (what would take humans many hours)
+   - Configurable research effort: low, medium, high
+
+3. **`health_check`** - Check server health status
 
 ## Testing the Server
 

docker/clients/perplexity_client.py

@@ -0,0 +1,93 @@
+# Copyright (c) 2024 Travis Frisinger
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""Perplexity API client - deep research search using Perplexity's sonar models."""
+
+import logging
+from typing import List, Literal
+
+from perplexity import Perplexity
+
+logger = logging.getLogger(__name__)
+
+
+def fetch_perplexity_deep_research(
+    query: str,
+    api_key: str,
+    max_results: int = 5,
+    research_effort: Literal["low", "medium", "high"] = "high",
+) -> tuple[str, List[str]]:
+    """
+    Fetch deep research results from Perplexity AI using official SDK.
+
+    Uses Perplexity's sonar-deep-research model which performs dozens of searches,
+    reads hundreds of sources, and reasons through material to deliver comprehensive
+    research reports. This is ideal for in-depth analysis and multi-source synthesis.
+
+    Args:
+        query: The research query/question
+        api_key: Perplexity API key
+        max_results: Maximum number of results to return (default: 5)
+        research_effort: Computational effort level - "low" (fast), "medium" (balanced),
+                        or "high" (deepest research). Only applies to sonar-deep-research.
+
+    Returns:
+        Tuple of (research_report: str, citations: List[str])
+        - research_report: Full synthesized research content in markdown
+        - citations: List of citation URLs used in the research
+    """
+    try:
+        logger.info(
+            f"Fetching Perplexity deep research (effort: {research_effort}): {query}"
+        )
+
+        # Initialize Perplexity client with 10-minute timeout for deep research
+        client = Perplexity(api_key=api_key, timeout=600.0)
+
+        # Create chat completion with deep research
+        # Note: Official SDK may not support all parameters, using minimal set
+        response = client.chat.completions.create(
+            model="sonar-deep-research",
+            messages=[
+                {
+                    "role": "system",
+                    "content": (
+                        "You are a comprehensive research assistant. Provide detailed, "
+                        "well-researched answers with clear structure and citations. "
+                        f"Focus on returning {max_results} most relevant sources."
+                    ),
+                },
+                {"role": "user", "content": query},
+            ],
+        )
+
+        # Extract research content
+        research_report = ""
+        citation_urls: List[str] = []
+
+        if response.choices and len(response.choices) > 0:
+            research_report = response.choices[0].message.content or ""
+
+            # Extract citations from response
+            if hasattr(response, "citations") and response.citations:
+                citation_urls = [
+                    citation if isinstance(citation, str) else citation.get("url", "")
+                    for citation in response.citations
+                    if citation
+                ][:max_results]
+
+            token_count = (
+                response.usage.total_tokens if hasattr(response, "usage") else "N/A"
+            )
+            logger.info(
+                f"Perplexity deep research completed: {len(research_report)} chars, "
+                f"{len(citation_urls)} citations, {token_count} tokens"
+            )
+
+        return research_report, citation_urls
+
+    except Exception as e:
+        logger.exception(f"Error fetching Perplexity deep research: {str(e)}")
+        return "", []

docker/constants.py

@@ -5,6 +5,8 @@
 
 """Constants for WebCat application."""
 
+import os
+
 # Application version
 VERSION = "2.3.1"
 
@@ -22,7 +24,10 @@
 ]
 
 # Content limits
-MAX_CONTENT_LENGTH = 8000
+try:
+    MAX_CONTENT_LENGTH = int(os.environ.get("MAX_CONTENT_LENGTH", "1000000"))
+except ValueError:
+    MAX_CONTENT_LENGTH = 1000000
 DEFAULT_SEARCH_RESULTS = 5
 
 # Timeout settings

docker/mcp_server.py

@@ -53,14 +53,19 @@ async def search_tool() -> dict:
 logger = setup_logging("webcat.log")
 
 # Import tools AFTER loading .env so they can access environment variables
+from tools.deep_research_tool import deep_research_tool  # noqa: E402
 from tools.health_check_tool import health_check_tool  # noqa: E402
 from tools.search_tool import search_tool  # noqa: E402
 
 # Log configuration status
 SERPER_API_KEY = os.environ.get("SERPER_API_KEY", "")
+PERPLEXITY_API_KEY = os.environ.get("PERPLEXITY_API_KEY", "")
 logging.info(
     f"SERPER API key: {'Set' if SERPER_API_KEY else 'Not set (using DuckDuckGo fallback)'}"
 )
+logging.info(
+    f"PERPLEXITY API key: {'Set' if PERPLEXITY_API_KEY else 'Not set (deep_research tool unavailable)'}"
+)
 
 
 # Create FastMCP instance
@@ -72,6 +77,16 @@ async def search_tool() -> dict:
     description="Search the web for information using Serper API or DuckDuckGo fallback",
 )(search_tool)
 
+mcp_server.tool(
+    name="deep_research",
+    description=(
+        "Perform comprehensive deep research on a topic using Perplexity AI. "
+        "This tool performs dozens of searches, reads hundreds of sources, and "
+        "synthesizes findings into a comprehensive report. Ideal for in-depth "
+        "analysis and multi-source research. Takes 2-4 minutes to complete."
+    ),
+)(deep_research_tool)
+
 mcp_server.tool(name="health_check", description="Check the health of the server")(
     health_check_tool
 )

docker/test_deep_research_mcp.py

@@ -0,0 +1,102 @@
+#!/usr/bin/env python3
+# Copyright (c) 2024 Travis Frisinger
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""Test deep_research tool via MCP server."""
+
+import json
+
+import requests
+
+MCP_SERVER_URL = "http://localhost:8000/mcp"
+
+# Create session with required headers
+session = requests.Session()
+session.headers.update({"Accept": "application/json, text/event-stream"})
+
+# Step 1: Initialize
+print("🔧 Initializing MCP session...")
+init_payload = {
+    "jsonrpc": "2.0",
+    "id": 1,
+    "method": "initialize",
+    "params": {
+        "protocolVersion": "2024-11-05",
+        "capabilities": {},
+        "clientInfo": {"name": "test-client", "version": "1.0.0"},
+    },
+}
+
+init_resp = session.post(MCP_SERVER_URL, json=init_payload)
+session_id = init_resp.headers.get("mcp-session-id")
+print(f"✅ Session ID: {session_id}\n")
+
+# Step 2: Send initialized notification
+print("🔔 Sending initialized notification...")
+initialized_payload = {"jsonrpc": "2.0", "method": "notifications/initialized"}
+session.post(
+    MCP_SERVER_URL, json=initialized_payload, headers={"mcp-session-id": session_id}
+)
+
+# Step 3: Call deep_research tool
+print("\n🔍 Calling deep_research tool...")
+print("Query: What is Python programming language?")
+print("Effort: low")
+print("\nThis will take ~1-2 minutes...\n")
+
+tool_payload = {
+    "jsonrpc": "2.0",
+    "id": 2,
+    "method": "tools/call",
+    "params": {
+        "name": "deep_research",
+        "arguments": {
+            "query": "What is Python programming language?",
+            "research_effort": "low",
+            "max_results": 5,
+        },
+    },
+}
+
+tool_resp = session.post(
+    MCP_SERVER_URL,
+    json=tool_payload,
+    headers={"mcp-session-id": session_id},
+    timeout=300,
+)
+
+print(f"Status: {tool_resp.status_code}\n")
+
+# Parse SSE response
+if tool_resp.status_code == 200:
+    lines = tool_resp.text.strip().split("\n")
+    for line in lines:
+        if line.startswith("data: "):
+            data_json = line[6:]  # Remove "data: " prefix
+            result = json.loads(data_json)
+
+            if "result" in result and "content" in result["result"]:
+                content = result["result"]["content"]
+                if isinstance(content, list) and len(content) > 0:
+                    text_content = content[0].get("text", "")
+                    # Parse the JSON string inside text
+                    research_result = json.loads(text_content)
+
+                    print("=" * 80)
+                    print("DEEP RESEARCH RESULT:")
+                    print("=" * 80)
+                    print(f"Query: {research_result.get('query')}")
+                    print(f"Source: {research_result.get('search_source')}")
+                    print(f"Results: {len(research_result.get('results', []))}")
+
+                    if research_result.get("results"):
+                        print("\n" + "=" * 80)
+                        print("RESEARCH REPORT:")
+                        print("=" * 80)
+                        print(research_result["results"][0].get("content", ""))
+
+            break
+else:
+    print(f"❌ Error: {tool_resp.text}")