fixed unit tests

Author: travis-bauer

docs/architecture/tool.md

@@ -4,9 +4,15 @@
 
 TalkPipe integrates with the Model Context Protocol (MCP) to enable LLMs to call custom tools during conversations. Tools are defined using the `TOOL` syntax in ChatterLang scripts, which compiles TalkPipe pipelines into callable functions that LLMs can invoke.
 
+## Unified Server Model
+
+Both `TOOL` and `MCP_SERVER` use the same server name concept. The `mcp_server` parameter in `TOOL` references a server that can be either:
+- **Local server**: Automatically created when first `TOOL` uses it
+- **External server**: Explicitly defined via `MCP_SERVER`
+
 ## TOOL Syntax
 
-The `TOOL` keyword in ChatterLang allows you to register TalkPipe pipelines as tools that can be used by LLMs. Tools are registered during script compilation and stored in FastMCP server instances.
+The `TOOL` keyword registers TalkPipe pipelines as tools that can be used by LLMs.
 
 ### Basic Syntax
 
@@ -21,25 +27,99 @@ TOOL tool_name = "pipeline_string" [parameters];
 - **`description`** (optional): A description of what the tool does, shown to the LLM
 - **`input_param`** (optional): Input parameter specification in format `"name:type:description"` (e.g., `"item:int:Number to double"`)
 - **`param_name`**, **`param_type`**, **`param_desc`** (optional): Individual parameter specifications as an alternative to `input_param`
-- **`mcp_server`** (optional): The name of the MCP server to register the tool with. Defaults to `"mcp"` if not specified
+- **`mcp_server`** (optional): The name of the MCP server to register the tool with. References a server name (local or external). Defaults to `"mcp"` if not specified
+
+## MCP_SERVER Syntax
 
-### Example
+The `MCP_SERVER` keyword connects to external MCP servers. Once connected, tools from that server are available for use with `llmPrompt`.
+
+### Basic Syntax
 
 ```chatterlang
-TOOL double = "| lambda[expression='item*2']" [input_param="item:int:Number to double", description="Doubles a number", mcp_server="math_tools"];
-TOOL add_ten = "| lambda[expression='item+10']" [input_param="item:int:Number to add 10 to", description="Adds 10 to a number", mcp_server="math_tools"];
+MCP_SERVER server_name = "url_or_config" [parameters];
 ```
 
-## Multiple MCP Servers
+### Parameters
+
+- **`url`** (required): The URL or connection string for the external MCP server
+- **`transport`** (optional): Transport type - `"http"`, `"sse"`, or `"stdio"`. Defaults to `"http"`
+- **`auth`** (optional): Authentication type - `"bearer"` or `"oauth"`
+- **`token`** (optional): Bearer token for authentication
+- **`headers`** (optional): Custom HTTP headers as a dictionary
+- **`command`** (optional): For stdio transport - command to execute
+- **`args`** (optional): For stdio transport - command arguments
+- **`cwd`** (optional): For stdio transport - working directory
+- **`env`** (optional): For stdio transport - environment variables
+
+The server is stored in `runtime.const_store[server_name]` and can be referenced in `llmPrompt` using `tools=server_name`.
+
+## MCP Server Management
 
-You can define multiple MCP servers in a single script by specifying different `mcp_server` values. Each server maintains its own set of tools.
+Both local and external MCP servers use the same server name system. The `mcp_server` parameter in `TOOL` references the server name defined by `MCP_SERVER` or created automatically.
+
+### Local Servers (Created Automatically)
+
+When you define a `TOOL` with a `mcp_server` parameter, a local FastMCP server is automatically created if it doesn't exist:
 
 ```chatterlang
 TOOL double = "| lambda[expression='item*2']" [input_param="item:int:Number to double", mcp_server="math_tools"];
-TOOL uppercase = "| lambda[expression='item.upper()']" [input_param="item:str:Text to uppercase", mcp_server="text_tools"];
+TOOL add_ten = "| lambda[expression='item+10']" [input_param="item:int:Number to add 10 to", mcp_server="math_tools"];
 ```
 
-Each MCP server instance is stored in `runtime.const_store[server_name]` and can be referenced directly in the script.
+The server `math_tools` is created automatically when the first tool is registered.
+
+### External Servers (MCP_SERVER Syntax)
+
+To connect to an external MCP server, use the `MCP_SERVER` keyword:
+
+```chatterlang
+# Connect to an external MCP server over HTTP
+MCP_SERVER external_tools = "https://api.example.com/mcp" [transport="http", auth="bearer", token="your-token"];
+
+# Tools from the external server are automatically available
+INPUT FROM echo[data="Use the external tools"]
+| llmPrompt[model="llama3.1", source="ollama", tools=external_tools]
+| print
+```
+
+### MCP_SERVER Parameters
+
+- **`url`** (required): The URL or connection string for the external server
+- **`transport`** (optional): Transport type - `"http"`, `"sse"`, or `"stdio"`. Defaults to `"http"`
+- **`auth`** (optional): Authentication type - `"bearer"` or `"oauth"`
+- **`token`** (optional): Bearer token for authentication
+- **`headers`** (optional): Custom HTTP headers as a dictionary
+- **`command`** (optional): For stdio transport - command to execute
+- **`args`** (optional): For stdio transport - command arguments
+- **`cwd`** (optional): For stdio transport - working directory
+- **`env`** (optional): For stdio transport - environment variables
+
+### Mixed Usage
+
+You can combine local tools and external servers in the same script:
+
+```chatterlang
+# Connect to external server
+MCP_SERVER external_api = "https://api.example.com/mcp" [transport="http", auth="bearer", token="token"];
+
+# Define local tools on a local server
+TOOL double = "| lambda[expression='item*2']" [input_param="item:int:Number to double", mcp_server="math_tools"];
+
+# Use either server with llmPrompt
+INPUT FROM echo[data="Use math tools"]
+| llmPrompt[model="llama3.1", source="ollama", tools=math_tools]
+| print
+
+INPUT FROM echo[data="Use external API"]
+| llmPrompt[model="llama3.1", source="ollama", tools=external_api]
+| print
+```
+
+**Note**: You cannot register local `TOOL` definitions on an external `MCP_SERVER`. External servers provide their own tools that are already available. If you try to register a `TOOL` with `mcp_server` pointing to an external server, a warning will be logged and the tool registration will be skipped.
+
+### Server Storage
+
+All MCP servers (both local and external) are stored in `runtime.const_store[server_name]` and can be referenced directly in the script using the server name identifier.
 
 ## Using Tools with LLMPrompt
 
@@ -60,11 +140,12 @@ The `tools` parameter accepts the MCP server identifier, which is automatically
 
 ### Compilation Process
 
-1. **Parsing**: The `TOOL` definitions are parsed during script compilation, similar to `CONST` definitions
-2. **Grouping**: Tools are grouped by their `mcp_server` parameter
-3. **Server Creation**: FastMCP instances are created for each unique server name
-4. **Tool Registration**: Each tool is registered with its corresponding MCP server using `register_talkpipe_tool()`
-5. **Storage**: MCP server instances are stored in `runtime.const_store[server_name]` for later reference
+1. **Parsing**: The `MCP_SERVER` and `TOOL` definitions are parsed during script compilation, similar to `CONST` definitions
+2. **External Server Connection**: `MCP_SERVER` definitions create FastMCP Client connections to external servers
+3. **Grouping**: Tools are grouped by their `mcp_server` parameter
+4. **Local Server Creation**: For tools referencing servers that don't exist, local FastMCP server instances are created
+5. **Tool Registration**: Each tool is registered with its corresponding MCP server using `register_talkpipe_tool()` (only for local servers)
+6. **Storage**: All MCP server instances (local and external) are stored in `runtime.const_store[server_name]` for later reference
 
 ### Tool Execution
 

src/talkpipe/chatterlang/compiler.py

@@ -19,6 +19,88 @@
 logger = logging.getLogger(__name__)
 
 
+def _register_mcp_servers(mcp_servers: Dict[str, Dict[str, Any]], runtime: RuntimeComponent) -> None:
+    """Register external MCP server connections defined in the script.
+    
+    Args:
+        mcp_servers: Dictionary mapping server names to server definitions
+        runtime: RuntimeComponent containing the MCP server instances
+    """
+    try:
+        from fastmcp import Client
+    except ImportError:
+        logger.warning("FastMCP is not installed. MCP servers will not be registered. Install with: pip install fastmcp")
+        return
+    
+    for server_name, server_def in mcp_servers.items():
+        url = server_def.get('url')
+        transport = server_def.get('transport', 'http')
+        
+        try:
+            if transport == 'http' or transport == 'sse':
+                # HTTP/SSE transport - Client can be initialized with URL directly
+                # For auth, we may need to use a config dict or set headers
+                if server_def.get('auth') == 'bearer' and server_def.get('token'):
+                    # Use config dict for authenticated connections
+                    config = {
+                        "mcpServers": {
+                            server_name: {
+                                "transport": transport,
+                                "url": url,
+                                "auth": "bearer",
+                                "token": server_def.get('token')
+                            }
+                        }
+                    }
+                    client = Client(config)
+                elif server_def.get('headers'):
+                    # Custom headers - would need to be passed via config
+                    config = {
+                        "mcpServers": {
+                            server_name: {
+                                "transport": transport,
+                                "url": url,
+                                "headers": server_def.get('headers')
+                            }
+                        }
+                    }
+                    client = Client(config)
+                else:
+                    # Simple URL connection
+                    client = Client(url)
+            elif transport == 'stdio':
+                # Stdio transport - requires command
+                command = server_def.get('command')
+                if not command:
+                    logger.error(f"MCP_SERVER '{server_name}': 'command' is required for stdio transport")
+                    continue
+                
+                config = {
+                    "mcpServers": {
+                        server_name: {
+                            "transport": "stdio",
+                            "command": command,
+                        }
+                    }
+                }
+                if server_def.get('args'):
+                    config["mcpServers"][server_name]["args"] = server_def.get('args')
+                if server_def.get('cwd'):
+                    config["mcpServers"][server_name]["cwd"] = server_def.get('cwd')
+                if server_def.get('env'):
+                    config["mcpServers"][server_name]["env"] = server_def.get('env')
+                
+                client = Client(config)
+            else:
+                logger.error(f"MCP_SERVER '{server_name}': Unknown transport '{transport}'")
+                continue
+            
+            runtime.const_store[server_name] = client
+            logger.debug(f"Created FastMCP Client for external server '{server_name}' (transport: {transport})")
+        except Exception as e:
+            logger.error(f"Failed to create MCP_SERVER '{server_name}': {e}")
+
+
 def _register_tools(tools: Dict[str, Dict[str, Any]], runtime: RuntimeComponent) -> None:
     """Register tools defined in the script with FastMCP.
     
@@ -49,13 +131,21 @@ def _register_tools(tools: Dict[str, Dict[str, Any]], runtime: RuntimeComponent)
 
     # Create or get MCP instances for each server and register tools
     for server_name, tool_list in tools_by_server.items():
-        # Get or create FastMCP instance for this server
+        # Get existing MCP instance (could be FastMCP server or Client)
         mcp = runtime.const_store.get(server_name)
         if mcp is None:
-            # Use server_name as the FastMCP instance name
+            # No existing server found - create a new local FastMCP server instance
             mcp = FastMCP(server_name)
             runtime.const_store[server_name] = mcp
-            logger.debug(f"Created new FastMCP instance '{server_name}' for tool registration")
+            logger.debug(f"Created new local FastMCP server instance '{server_name}' for tool registration")
+        else:
+            # Server already exists (could be from MCP_SERVER definition or previous TOOL)
+            # Check if it's a Client (external) or FastMCP (local)
+            if hasattr(mcp, 'call_tool'):
+                # It's a FastMCP Client (external server) - cannot register local tools on it
+                logger.warning(f"Cannot register local tools on external MCP server '{server_name}'. Tools from external servers are already available.")
+                continue
+            # Otherwise it's a FastMCP server instance - we can register tools on it
 
         # Register each tool for this server
         for tool_name, tool_def in tool_list:
@@ -123,6 +213,11 @@ def compile(script: ParsedScript, runtime: RuntimeComponent = None) -> Callable:
     runtime.add_constants(script.constants, override=False)
     logger.debug(f"Initialized runtime with {len(runtime.const_store)} constants")
 
+    # Process MCP server definitions first (external servers)
+    if script.mcp_servers:
+        _register_mcp_servers(script.mcp_servers, runtime)
+        logger.debug(f"Registered {len(script.mcp_servers)} MCP server connections")
+    
     # Process tool definitions and register them
     if script.tools:
         _register_tools(script.tools, runtime)

src/talkpipe/chatterlang/parsers.py

@@ -105,6 +105,7 @@ class ParsedScript:
     pipelines: List[Union[ParsedPipeline, ParsedLoop]]
     constants: Dict[str, Any] = field(default_factory=dict)
     tools: Dict[str, Dict[str, Any]] = field(default_factory=dict)
+    mcp_servers: Dict[str, Dict[str, Any]] = field(default_factory=dict)
 
     def __iter__(self):
         return iter(self.pipelines)
@@ -194,6 +195,32 @@ def constant_definition():
     return (const_name.name, const_value)
 """A parser for defining constants with 'SET' keyword."""
 
+# Parser for MCP server definitions (external servers)
+@generate
+def mcp_server_definition():
+    """Parser for MCP_SERVER definitions: MCP_SERVER name = "url_or_config" [params]
+    
+    Example:
+        MCP_SERVER external_tools = "https://api.example.com/mcp" [transport="http", auth="bearer", token="token"];
+    """
+    yield lexeme('MCP_SERVER')
+    server_name = yield identifier
+    yield lexeme('=')
+    url_or_config = yield quoted_string  # URL or config string
+    params = yield bracket_parser  # Optional parameters in brackets
+    return (server_name.name, {
+        'url': url_or_config,
+        'transport': params.get('transport', 'http'),  # Default to http
+        'auth': params.get('auth', None),
+        'token': params.get('token', None),
+        'headers': params.get('headers', None),
+        'command': params.get('command', None),  # For stdio transport
+        'args': params.get('args', None),  # For stdio transport
+        'cwd': params.get('cwd', None),  # For stdio transport
+        'env': params.get('env', None),  # For stdio transport
+    })
+"""A parser for defining external MCP server connections with 'MCP_SERVER' keyword."""
+
 # Parser for tool definitions
 @generate
 def tool_definition():
@@ -371,21 +398,29 @@ def loop():
 pipeline_separator = lexeme(';')
 """A parser for the separator between pipelines in a script."""
 
-# Combined parser for constants and tools (both use semicolon separators)
-definition = constant_definition.map(lambda x: ('const', x)) | tool_definition.map(lambda x: ('tool', x))
+# Combined parser for constants, MCP servers, and tools (all use semicolon separators)
+definition = (
+    constant_definition.map(lambda x: ('const', x)) |
+    mcp_server_definition.map(lambda x: ('mcp_server', x)) |
+    tool_definition.map(lambda x: ('tool', x))
+)
 
 @generate
 def script_parser():
-    # Parse constants and tools - they can be interleaved, separated by semicolons
+    # Parse constants, MCP servers, and tools - they can be interleaved, separated by semicolons
     definitions = yield definition.sep_by(pipeline_separator, min=0)
 
-    # Separate constants and tools
+    # Separate constants, MCP servers, and tools
     constants = {}
+    mcp_servers = {}
     tools = {}
     for def_type, def_value in definitions:
         if def_type == 'const':
             k, v = def_value
             constants[k] = v
+        elif def_type == 'mcp_server':
+            k, v = def_value
+            mcp_servers[k] = v
         else:  # tool
             k, v = def_value
             tools[k] = v
@@ -395,7 +430,7 @@ def script_parser():
 
     pipelines = [p for p in pipelines if not isinstance(p, ParsedPipeline) or (p.input_node or len(p.transforms)>0)]
 
-    # Create and return a ParsedScript with constants and tools
-    return ParsedScript(pipelines, constants, tools)
+    # Create and return a ParsedScript with constants, MCP servers, and tools
+    return ParsedScript(pipelines, constants, tools, mcp_servers)
 """A parser for a script in the pipeline language.  Scripts contain a series of pipelines and loops."""