Merge pull request #159 from vamplabAI/157-tools-docs

Tunes of documentation

Author: EvilFreelancer

.cursor/rules/architecture.mdc

@@ -26,9 +26,9 @@ The project uses **modular architecture with clear separation of concerns**. The
 
 ### Layer 3: Factory and Services
 - `agent_factory.py` - AgentFactory for creating agents
-- `services/mcp_service.py` - MCP2ToolConverter
+- `services/mcp_service.py` - MCP2ToolConverter for MCP tool integration
 - `services/tavily_search.py` - Tavily search service
-- `next_step_tool.py` - NextStepToolsBuilder, NextStepToolStub
+- `next_step_tool.py` - NextStepToolsBuilder, NextStepToolStub, DiscriminantToolMixin
 
 ### Layer 4: Agent Implementations
 - `agents/sgr_agent.py` - SGRAgent (Structured Output)
@@ -54,43 +54,59 @@ The project uses **modular architecture with clear separation of concerns**. The
 
 ## Agent Execution Cycle
 
-All agents follow a two-phase cycle:
+All agents follow a three-phase cycle implemented in `BaseAgent._execution_step()`:
 
 ```python
-while agent.state not in FINISH_STATES:
+while agent._context.state not in FINISH_STATES:
     reasoning = await agent._reasoning_phase()
     action_tool = await agent._select_action_phase(reasoning)
     await agent._action_phase(action_tool)
 ```
 
-### Phase 1: Reasoning Phase
-- Agent analyzes current context
-- Decides on next action
-- Implementation varies by agent type (SO, FC, or Hybrid)
+The main `execute()` method runs this cycle until agent reaches a finish state.
 
-### Phase 2: Select Action Phase
-- Selects appropriate tool based on reasoning
-- Returns tool instance ready for execution
+### Phase 1: Reasoning Phase (`_reasoning_phase()`)
+- Agent analyzes current context via `_prepare_context()`
+- Decides on next action using LLM
+- Implementation varies by agent type:
+  - **SGRAgent**: Uses structured output with `NextStepToolStub` (reasoning + tool selection)
+  - **ToolCallingAgent**: Returns `None` (no explicit reasoning)
+  - **SGRToolCallingAgent**: Uses Function Calling to get `ReasoningTool` result, then executes it
+- Returns `ReasoningTool` or `NextStepToolStub` (or `None` for ToolCallingAgent)
 
-### Phase 3: Action Phase
-- Executes selected tool
-- Updates conversation history
-- Updates agent context
+### Phase 2: Select Action Phase (`_select_action_phase()`)
+- Selects appropriate tool based on reasoning
+- Implementation varies by agent type:
+  - **SGRAgent**: Extracts tool from `reasoning.function` field
+  - **ToolCallingAgent**: Uses Function Calling with `tool_choice="required"`
+  - **SGRToolCallingAgent**: Uses Function Calling with `tool_choice="required"` (handles text response edge case)
+- Returns `BaseTool` instance ready for execution
+
+### Phase 3: Action Phase (`_action_phase()`)
+- Executes selected tool: `result = await tool(context, config)`
+- Updates conversation history with tool call and result
+- Updates agent context (state, iteration, sources, etc.)
+- Logs tool execution
+- Handles special cases (e.g., `ClarificationTool` pauses execution)
 
 ## Module Rules
 
 ### Agents (`sgr_agent_core/agents/`)
 - Must inherit from `BaseAgent`
 - Must implement `_reasoning_phase()`, `_select_action_phase()`, `_action_phase()`
-- Automatically registered in `AgentRegistry` via `AgentRegistryMixin`
+- Can override `_execution_step()` to customize execution cycle
 - Can override `_prepare_context()` and `_prepare_tools()` for customization
+- Automatically registered in `AgentRegistry` via `AgentRegistryMixin`
+- Must set `name` class variable (used for registration)
 
 ### Tools (`sgr_agent_core/tools/`)
 - Must inherit from `BaseTool` or `MCPBaseTool`
-- Must be Pydantic models
-- Must implement `__call__()` method
+- Must be Pydantic models (inherit from `BaseModel`)
+- Must implement `__call__(context: AgentContext, config: AgentConfig, **kwargs) -> str`
+- Must set `tool_name` and `description` class variables (or use defaults)
 - Automatically registered in `ToolRegistry` via `ToolRegistryMixin`
 - Return string or JSON string from `__call__()`
+- Can be used in `NextStepToolsBuilder` for structured output
 
 ### Services (`sgr_agent_core/services/`)
 - Stateless utility classes
@@ -99,8 +115,11 @@ while agent.state not in FINISH_STATES:
 
 ### Configuration (`agent_config.py`, `agent_definition.py`)
 - Hierarchical configuration: GlobalConfig → AgentDefinition → AgentConfig
-- Supports YAML loading
+- Supports YAML loading via `GlobalConfig.from_yaml()` and `definitions_from_yaml()`
+- Supports environment variables via `pydantic-settings` (prefix `SGR__`)
 - Automatic inheritance and override of settings
+- `AgentDefinition` inherits from `AgentConfig` (has all config fields + name, base_class, tools)
+- Config classes use `extra="allow"` to support custom fields
 
 ## Design Principles
 

.cursor/rules/core-modules.mdc

@@ -10,95 +10,165 @@ alwaysApply: true
 
 ### BaseAgent (`sgr_agent_core/base_agent.py`)
 - Parent class for all agents
-- Implements two-phase execution cycle: Reasoning → Action
+- Implements three-phase execution cycle: Reasoning → Select Action → Action
 - Manages agent context, conversation history, and streaming
 - Must be subclassed to implement `_reasoning_phase()`, `_select_action_phase()`, `_action_phase()`
 - Automatically registered in `AgentRegistry` via `AgentRegistryMixin`
+- Key attributes:
+  - `id`: Unique agent identifier (format: `{def_name or name}_{uuid}`)
+  - `name`: Agent class name
+  - `task_messages`: Initial task messages in OpenAI format
+  - `config`: AgentConfig instance with all settings
+  - `openai_client`: AsyncOpenAI client for LLM API
+  - `toolkit`: List of tool classes available to agent
+  - `_context`: AgentContext instance with execution state
+  - `conversation`: List of messages in OpenAI format for LLM context
+  - `streaming_generator`: OpenAIStreamingGenerator for streaming responses
+  - `logger`: Logger instance for agent logging
+  - `log`: List of execution logs
+  - `creation_time`: Datetime when agent was created
+- Key methods:
+  - `execute()`: Main execution loop (called externally)
+  - `_execution_step()`: Single step of execution cycle (can be overridden)
+  - `_prepare_context()`: Prepare conversation context (can be overridden)
+  - `_prepare_tools()`: Prepare available tools (can be overridden)
+  - `provide_clarification()`: Receive clarification from external source
+  - `_log_reasoning()`: Log reasoning phase results
+  - `_log_tool_execution()`: Log tool execution results
+  - `_save_agent_log()`: Save execution log to file
 
 ### BaseTool (`sgr_agent_core/base_tool.py`)
-- Parent class for all tools
-- Must be a Pydantic model
-- Must implement `__call__(context, config)` method
-- Returns string or JSON string
+- Parent class for all tools (Pydantic `BaseModel`)
+- Class variables: `tool_name` (ClassVar[str]), `description` (ClassVar[str])
+- Must implement `__call__(context: AgentContext, config: AgentConfig, **kwargs) -> str`
+- Returns string or JSON string from `__call__()`
 - Automatically registered in `ToolRegistry` via `ToolRegistryMixin`
+- `tool_name` defaults to class name (lowercase) if not set
+- `description` defaults to class docstring if not set
 
 ### MCPBaseTool (`sgr_agent_core/base_tool.py`)
-- Base class for MCP-integrated tools
-- Handles MCP client calls
-- Converts MCP responses to tool format
+- Base class for MCP-integrated tools (inherits from `BaseTool`)
+- Class variable: `_client` (ClassVar[Client | None]) - MCP client instance
+- `__call__()`: Calls MCP tool via `fastmcp.Client.call_tool()`
+- Converts MCP responses to JSON string
+- Respects `mcp_context_limit` from `ExecutionConfig`
+- Handles errors gracefully (returns error message as string)
 
 ## Configuration Modules
 
 ### GlobalConfig (`sgr_agent_core/agent_config.py`)
 - Singleton pattern for global configuration
-- Loads from YAML files (`config.yaml`, `agents.yaml`)
-- Provides default values for all agent settings
+- All calls to `GlobalConfig()` return the same instance
+- Loads from YAML files via `from_yaml()` method
+- Loads from environment variables via `pydantic-settings` (prefix `SGR__`)
+- Contains: `llm`, `search`, `execution`, `prompts`, `mcp`, `agents`, `tools`
+- `agents`: Dictionary of `AgentDefinition` instances by name
+- `tools`: Dictionary of tool definitions by name
+- `definitions_from_yaml()`: Loads agent definitions from YAML (merges with existing)
 
 ### AgentDefinition (`sgr_agent_core/agent_definition.py`)
 - Definition template for creating agents
-- Contains: name, base_class, tools, llm, prompts, execution, search, mcp configs
-- Supports YAML loading
-- Validates required fields
+- Inherits from `AgentConfig` (has all config fields)
+- Additional fields: `name`, `base_class`, `tools`
+- `base_class`: Can be class, ImportString (e.g., `"sgr_agent_core.agents.SGRAgent"`), or registry name
+- `tools`: List of tool names (strings) or tool classes
+- Supports YAML loading via `GlobalConfig.definitions_from_yaml()`
+- Validates import strings point to existing files
+- Automatically merges with `GlobalConfig` defaults
 
 ### AgentConfig (`sgr_agent_core/agent_definition.py`)
 - Runtime configuration for agent instance
-- Combines: LLMConfig, SearchConfig, ExecutionConfig, PromptsConfig, MCPConfig
-- Supports hierarchical inheritance from GlobalConfig
+- Combines: `LLMConfig`, `SearchConfig`, `ExecutionConfig`, `PromptsConfig`, `MCPConfig`
+- Supports hierarchical inheritance from `GlobalConfig`
+- Uses `extra="allow"` to support custom fields for agent-specific parameters
 
 ## Factory and Services
 
 ### AgentFactory (`sgr_agent_core/agent_factory.py`)
-- Creates agent instances from AgentDefinition
-- Resolves agent classes from AgentRegistry
-- Resolves tools from ToolRegistry
-- Builds MCP tools via MCP2ToolConverter
-- Creates OpenAI client with proxy support
+- Creates agent instances from `AgentDefinition`
+- Resolves agent classes from `AgentRegistry` (by name or ImportString)
+- Resolves tools from `ToolRegistry` or `config.tools` section
+- Tool resolution order:
+  1. Tools defined in `config.tools` section
+  2. Tools in `ToolRegistry` by name (snake_case or PascalCase)
+  3. Auto-conversion snake_case → PascalCase for backward compatibility
+- Builds MCP tools via `MCP2ToolConverter`
+- Creates OpenAI client with proxy support via `httpx.AsyncClient`
+- `get_definitions_list()`: Returns all agent definitions from `GlobalConfig`
 
 ### AgentRegistry (`sgr_agent_core/services/registry.py`)
-- Centralized registry for agent classes
-- Automatic registration via `AgentRegistryMixin`
+- Registry for agent classes (subclass of `Registry[BaseAgent]`)
+- Automatic registration via `AgentRegistryMixin` in `BaseAgent.__init_subclass__()`
+- Registers by class name (lowercase) and `name` attribute
 - Supports lookup by name (case-insensitive)
 
 ### ToolRegistry (`sgr_agent_core/services/registry.py`)
-- Centralized registry for tool classes
-- Automatic registration via `ToolRegistryMixin`
+- Registry for tool classes (subclass of `Registry[BaseTool]`)
+- Automatic registration via `ToolRegistryMixin` in `BaseTool.__init_subclass__()`
+- Registers by class name (lowercase) and `tool_name` attribute
 - Supports lookup by name (case-insensitive)
 
 ### PromptLoader (`sgr_agent_core/services/prompt_loader.py`)
-- Loads and formats prompts from files or strings
-- Generates system prompts with tool descriptions
-- Formats initial user requests and clarification responses
+- Static class for loading and formatting prompts
+- `get_system_prompt()`: Formats system prompt with available tools list
+- `get_initial_user_request()`: Formats initial user request with current date
+- `get_clarification_template()`: Formats clarification response template
+- Uses templates from `PromptsConfig` (files or strings)
+- Supports placeholders: `{available_tools}`, `{current_date}`
 
 ### MCP2ToolConverter (`sgr_agent_core/services/mcp_service.py`)
-- Converts MCP server tools to BaseTool instances
-- Handles MCP client initialization
-- Builds tools from MCP configuration
+- Converts MCP server tools to `BaseTool` instances
+- `build_tools_from_mcp()`: Async method to build tools from MCP config
+- Uses `fastmcp.Client` to connect to MCP servers
+- Uses `jambo.SchemaConverter` to convert JSON schemas to Pydantic models
+- Creates dynamic tool classes inheriting from `MCPBaseTool`
+- Tool names converted to CamelCase (e.g., `web_search` → `MCPWebSearch`)
 
 ## Agent Implementations
 
 ### SGRAgent (`sgr_agent_core/agents/sgr_agent.py`)
-- Uses Structured Output approach
-- Creates dynamic JSON schema for tools
-- LLM returns reasoning + tool schema in one call
-- Extracts tool directly from reasoning result
+- Uses Structured Output approach with `response_format`
+- Uses `NextStepToolsBuilder` to create dynamic union tool type
+- `_prepare_tools()` returns `Type[NextStepToolStub]` for structured output
+- `_reasoning_phase()` uses `response_format` to get `NextStepToolStub` with selected tool
+- `_select_action_phase()` extracts tool from `reasoning.function` field
+- LLM returns reasoning + tool selection in one structured call
+- Best for models with strong structured output support
 
 ### ToolCallingAgent (`sgr_agent_core/agents/tool_calling_agent.py`)
-- Uses native Function Calling
-- No explicit reasoning phase
-- Uses `tool_choice="required"` for tool selection
-- Best for advanced LLM models
+- Uses native Function Calling approach
+- `_reasoning_phase()` returns `None` (no explicit reasoning)
+- `_select_action_phase()` uses `tool_choice="required"` to force tool selection
+- LLM directly selects and calls tool via function calling
+- No structured reasoning output
+- Best for advanced LLM models with strong function calling support
 
 ### SGRToolCallingAgent (`sgr_agent_core/agents/sgr_tool_calling_agent.py`)
-- Hybrid approach: SGR + Function Calling
-- Uses ReasoningTool for explicit reasoning
-- Uses Function Calling for tool selection
-- Best balance for most tasks
+- Hybrid approach: SGR reasoning + Function Calling for tools
+- `_reasoning_phase()` uses Function Calling to get `ReasoningTool` result
+- Executes `ReasoningTool` to get structured reasoning output
+- `_select_action_phase()` uses Function Calling with `tool_choice="required"` for tool selection
+- Handles edge case: if LLM returns text instead of tool call, creates `FinalAnswerTool`
+- Best balance for most tasks - combines structured reasoning with flexible tool selection
 
 ## Tools
 
+### NextStepToolsBuilder (`sgr_agent_core/next_step_tool.py`)
+- Builder for creating dynamic union tool types
+- `build_NextStepTools()`: Creates `NextStepToolStub` subclass with union of available tools
+- Uses discriminated union pattern with `tool_name_discriminator` field
+- Enables structured output with dynamic tool selection
+- Used by `SGRAgent` for tool selection
+
+### NextStepToolStub (`sgr_agent_core/next_step_tool.py`)
+- Stub class for `NextStepTools` created by `NextStepToolsBuilder`
+- Inherits from `ReasoningTool` and contains `function` field with union of tools
+- Used in structured output to select tool for next step
+
 ### ReasoningTool (`sgr_agent_core/tools/reasoning_tool.py`)
 - Provides structured reasoning output
 - Contains: reasoning_steps, current_situation, plan_status, enough_data, remaining_steps, task_completed
+- Base class for `NextStepToolStub`
 
 ### ClarificationTool (`sgr_agent_core/tools/clarification_tool.py`)
 - Requests clarification from user
@@ -137,21 +207,41 @@ alwaysApply: true
 
 ## Server and API
 
+### Server Settings (`sgr_agent_core/server/settings.py`)
+- Server configuration (host, port, etc.)
+- Used by FastAPI application
+
+### Server Models (`sgr_agent_core/server/models.py`)
+- Pydantic models for API requests/responses
+- `ChatCompletionRequest`: OpenAI-compatible request model
+- `MessagesList`: Root model for message lists with base64 truncation
+- `AgentStateResponse`: Agent state response model
+- `ClarificationRequest`: Clarification request model
+- `HealthResponse`: Health check response
+
 ### FastAPI Application (`sgr_agent_core/server/app.py`)
 - Main FastAPI application
-- Configures CORS, middleware
-- Registers endpoints
+- Configures CORS, middleware, logging
+- Registers API router from `endpoints.py`
+- Uses `server/settings.py` for configuration
 
 ### API Endpoints (`sgr_agent_core/server/endpoints.py`)
-- `/v1/chat/completions` - OpenAI-compatible chat endpoint
-- `/v1/agents/{agent_id}/state` - Get agent state
-- `/v1/agents/{agent_id}/clarification` - Provide clarification
-- `/v1/agents` - List available agents
+- `/v1/chat/completions` - OpenAI-compatible chat endpoint (streaming only)
+  - Creates agent from `AgentDefinition` by model name
+  - Supports clarification requests via agent ID in model field
+  - Returns streaming response via `OpenAIStreamingGenerator`
+- `/v1/models` - List available agent definitions (OpenAI-compatible)
+- `/agents/{agent_id}/state` - Get agent state (GET)
+- `/agents/{agent_id}/provide_clarification` - Provide clarification (POST)
+- `/agents` - List all active agents (GET)
+- `/health` - Health check endpoint
 
 ### Streaming (`sgr_agent_core/stream.py`)
-- OpenAIStreamingGenerator for streaming responses
-- Formats events in OpenAI-compatible format
+- `OpenAIStreamingGenerator` for streaming responses
+- Formats events in OpenAI-compatible Server-Sent Events (SSE) format
 - Handles tool calls and content chunks
+- Provides async iterator via `stream()` method
+- Methods: `add_chunk()`, `add_tool_call()`, `add_chunk_from_str()`, `finish()`
 
 ## General Rules for All Modules