diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index eae39885..911d8754 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -73,9 +73,9 @@ jobs: id: generate_rg_name run: | echo "Generating a unique resource group name..." - TIMESTAMP=$(date +%Y%m%d%H%M%S) - COMMON_PART="ci-biab" - UNIQUE_RG_NAME="${COMMON_PART}${TIMESTAMP}" + ACCL_NAME="ci-biab" # Account name as specified + SHORT_UUID=$(uuidgen | cut -d'-' -f1) + UNIQUE_RG_NAME="arg-${ACCL_NAME}-${SHORT_UUID}" echo "RESOURCE_GROUP_NAME=${UNIQUE_RG_NAME}" >> $GITHUB_ENV echo "Generated Resource_GROUP_PREFIX: ${UNIQUE_RG_NAME}" diff --git a/docs/CustomizeSolution.md b/docs/CustomizeSolution.md new file mode 100644 index 00000000..d07e02d8 --- /dev/null +++ b/docs/CustomizeSolution.md @@ -0,0 +1,618 @@ +# Accelerating your own Multi-Agent - Custom Automation Engine MVP + +As the name suggests, this project is designed to accelerate development of Multi-Agent solutions in your environment. The example solution presented shows how such a solution would be implemented and provides example agent definitions along with stubs for possible tools those agents could use to accomplish tasks. You will want to implement real functions in your own environment, to be used by agents customized around your own use cases. Users can choose the LLM that is optimized for responsible use. The default LLM is GPT-4o which inherits the existing responsible AI mechanisms and filters from the LLM provider. We encourage developers to review [OpenAI’s Usage policies](https://openai.com/policies/usage-policies/) and [Azure OpenAI’s Code of Conduct](https://learn.microsoft.com/en-us/legal/cognitive-services/openai/code-of-conduct) when using GPT-4o. This document is designed to provide the in-depth technical information to allow you to add these customizations. Once the agents and tools have been developed, you will likely want to implement your own real world front end solution to replace the example in this accelerator. + +## Technical Overview + +This application is an AI-driven orchestration system that manages a group of AI agents to accomplish tasks based on user input. It uses a FastAPI backend to handle HTTP requests, processes them through various specialized agents, and stores stateful information using Azure Cosmos DB. The system is designed to: + +- Receive input tasks from users. +- Generate a detailed plan to accomplish the task using a Planner agent. +- Execute the plan by delegating steps to specialized agents (e.g., HR, Procurement, Marketing). +- Incorporate human feedback into the workflow. +- Maintain state across sessions with persistent storage. + +This code has not been tested as an end-to-end, reliable production application- it is a foundation to help accelerate building out multi-agent systems. You are encouraged to add your own data and functions to the agents, and then you must apply your own performance and safety evaluation testing frameworks to this system before deploying it. + +Below, we'll dive into the details of each component, focusing on the endpoints, data types, and the flow of information through the system. +# Table of Contents + +- [Table of Contents](#table-of-contents) + - [Accelerating your own Multi-Agent - Custom Automation Engine MVP](#accelerating-your-own-multi-agent---custom-automation-engine-mvp) + - [Technical Overview](#technical-overview) + - [Adding a New Agent to the Multi-Agent System](#adding-a-new-agent-to-the-multi-agent-system) + - [API Reference](#api-reference) + - [Models and Datatypes](#models-and-datatypes) + - [Application Flow](#application-flow) + - [Agents Overview](#agents-overview) + - [Persistent Storage with Cosmos DB](#persistent-storage-with-cosmos-db) + - [Utilities](#utilities) + - [Summary](#summary) + + +## Adding a New Agent to the Multi-Agent System + +This guide details the steps required to add a new agent to the Multi-Agent Custom Automation Engine. The process includes registering the agent, defining its capabilities through tools, and ensuring the PlannerAgent includes the new agent when generating activity plans. + +### **Step 1: Define the New Agent's Tools** +Every agent is equipped with a set of tools (functions) that it can call to perform specific tasks. These tools need to be defined first. + +1. **Create New Tools**: In a new or existing file, define the tools your agent will use. + + Example (for a `BakerAgent`): + ```python + from autogen_core.components.tools import FunctionTool, Tool + from typing import List + + async def bake_cookies(cookie_type: str, quantity: int) -> str: + return f"Baked {quantity} {cookie_type} cookies." + + async def prepare_dough(dough_type: str) -> str: + return f"Prepared {dough_type} dough." + + def get_baker_tools() -> List[Tool]: + return [ + FunctionTool(bake_cookies, description="Bake cookies of a specific type.", name="bake_cookies"), + FunctionTool(prepare_dough, description="Prepare dough of a specific type.", name="prepare_dough"), + ] + ``` + + +2. **Implement the Agent Class** +Create a new agent class that inherits from `BaseAgent`. + +Example (for `BakerAgent`): +```python +from agents.base_agent import BaseAgent + +class BakerAgent(BaseAgent): + def __init__(self, model_client, session_id, user_id, memory, tools, agent_id): + super().__init__( + "BakerAgent", + model_client, + session_id, + user_id, + memory, + tools, + agent_id, + system_message="You are an AI Agent specialized in baking tasks.", + ) +``` +### **Step 2: Register the new Agent in the messages** +Update `messages.py` to include the new agent. + + ```python + class BAgentType(str, Enum): + baker_agent = "BakerAgent" +``` + +### **Step 3: Register the Agent in the Initialization Process** +Update the `initialize_runtime_and_context` function in `utils.py` to include the new agent. + +1. **Import new agent**: + ```python + from agents.baker_agent import BakerAgent, get_baker_tools + ``` + +2. **Add the bakers tools**: + ```python + baker_tools = get_baker_tools() + ``` + +3. **Generate Agent IDs**: + ```python + baker_agent_id = AgentId("baker_agent", session_id) + baker_tool_agent_id = AgentId("baker_tool_agent", session_id) + ``` + +4. **Register to ToolAgent**: + ```python + await ToolAgent.register( + runtime, + "baker_tool_agent", + lambda: ToolAgent("Baker tool execution agent", baker_tools), + ) + ``` + +5. **Register the Agent and ToolAgent**: + ```python + await BakerAgent.register( + runtime, + baker_agent_id.type, + lambda: BakerAgent( + aoai_model_client, + session_id, + user_id, + cosmos_memory, + get_baker_tools(), + baker_tool_agent_id, + ), + ) + ``` +6. **Add to agent_ids**: + ```python + agent_ids = { + BAgentType.baker_agent: baker_agent_id, + ``` +7. **Add to retrieve_all_agent_tools**: + ```python + def retrieve_all_agent_tools() -> List[Dict[str, Any]]: + baker_tools: List[Tool] = get_baker_tools() + ``` +8. **Append baker_tools to functions**: + ```python + for tool in baker_tools: + functions.append( + { + "agent": "BakerAgent", + "function": tool.name, + "description": tool.description, + "arguments": str(tool.schema["parameters"]["properties"]), + } + ) + ``` +### **Step 4: Update home page** +Update `src/frontend/wwwroot/home/home.html` adding new html block + +1. **Add a new UI element was added to allow users to request baking tasks from the BakerAgent** +```html +
+
+
+
+ Bake Cookies +

Please bake 12 chocolate chip cookies for tomorrow's event.

+
+
+
+``` +### **Step 5: Update tasks** +Update `src/frontend/wwwroot/task/task.js` + +1. **Add `BakerAgent` as a recognized agent type in the frontend JavaScript file** +```js + case "BakerAgent": + agentIcon = "manager"; + break; +``` +### **Step 6: Validate the Integration** +Deploy the updated system and ensure the new agent is properly included in the planning process. For example, if the user requests to bake cookies, the `PlannerAgent` should: + +- Identify the `BakerAgent` as the responsible agent. +- Call `bake_cookies` or `prepare_dough` from the agent's toolset. + +### **Step 7: Update Documentation** +Ensure that the system documentation reflects the addition of the new agent and its capabilities. Update the `README.md` and any related technical documentation to include information about the `BakerAgent`. + +### **Step 8: Testing** +Thoroughly test the agent in both automated and manual scenarios. Verify that: + +- The agent responds correctly to tasks. +- The PlannerAgent includes the new agent in relevant plans. +- The agent's tools are executed as expected. + +Following these steps will successfully integrate a new agent into the Multi-Agent Custom Automation Engine. + +### API Reference +To view the API reference, go to the API endpoint in a browser and add "/docs". This will bring up a full Swagger environment and reference documentation for the REST API included with this accelerator. For example, ```https://macae-backend.eastus2.azurecontainerapps.io/docs```. +If you prefer ReDoc, this is available by appending "/redoc". + +![docs interface](./images/customize_solution/redoc_ui.png) + +### Models and Datatypes +#### Models +##### **`BaseDataModel`** +The `BaseDataModel` is a foundational class for creating structured data models using Pydantic. It provides the following attributes: + +- **`id`**: A unique identifier for the data, generated using `uuid`. +- **`ts`**: An optional timestamp indicating when the model instance was created or modified. + +#### **`AgentMessage`** +The `AgentMessage` model represents communication between agents and includes the following fields: + +- **`id`**: A unique identifier for the message, generated using `uuid`. +- **`data_type`**: A literal value of `"agent_message"` to identify the message type. +- **`session_id`**: The session associated with this message. +- **`user_id`**: The ID of the user associated with this message. +- **`plan_id`**: The ID of the related plan. +- **`content`**: The content of the message. +- **`source`**: The origin or sender of the message (e.g., an agent). +- **`ts`**: An optional timestamp for when the message was created. +- **`step_id`**: An optional ID of the step associated with this message. + +#### **`Session`** +The `Session` model represents a user session and extends the `BaseDataModel`. It has the following attributes: + +- **`data_type`**: A literal value of `"session"` to identify the type of data. +- **`current_status`**: The current status of the session (e.g., `active`, `completed`). +- **`message_to_user`**: An optional field to store any messages sent to the user. +- **`ts`**: An optional timestamp for the session's creation or last update. + + +#### **`Plan`** +The `Plan` model represents a high-level structure for organizing actions or tasks. It extends the `BaseDataModel` and includes the following attributes: + +- **`data_type`**: A literal value of `"plan"` to identify the data type. +- **`session_id`**: The ID of the session associated with this plan. +- **`initial_goal`**: A description of the initial goal derived from the user’s input. +- **`overall_status`**: The overall status of the plan (e.g., `in_progress`, `completed`, `failed`). + +#### **`Step`** +The `Step` model represents a discrete action or task within a plan. It extends the `BaseDataModel` and includes the following attributes: + +- **`data_type`**: A literal value of `"step"` to identify the data type. +- **`plan_id`**: The ID of the plan the step belongs to. +- **`action`**: The specific action or task to be performed. +- **`agent`**: The name of the agent responsible for executing the step. +- **`status`**: The status of the step (e.g., `planned`, `approved`, `completed`). +- **`agent_reply`**: An optional response from the agent after executing the step. +- **`human_feedback`**: Optional feedback provided by a user about the step. +- **`updated_action`**: Optional modified action based on human feedback. +- **`session_id`**: The session ID associated with the step. +- **`user_id`**: The ID of the user providing feedback or interacting with the step. + +#### **`PlanWithSteps`** +The `PlanWithSteps` model extends the `Plan` model and includes additional information about the steps in the plan. It has the following attributes: + +- **`steps`**: A list of `Step` objects associated with the plan. +- **`total_steps`**: The total number of steps in the plan. +- **`completed_steps`**: The number of steps that have been completed. +- **`pending_steps`**: The number of steps that are pending approval or completion. + +**Additional Features**: +The `PlanWithSteps` model provides methods to update step counts: +- `update_step_counts()`: Calculates and updates the `total_steps`, `completed_steps`, and `pending_steps` fields based on the associated steps. + +#### **`InputTask`** +The `InputTask` model represents the user’s initial input for creating a plan. It includes the following attributes: + +- **`session_id`**: An optional string for the session ID. If not provided, a new UUID will be generated. +- **`description`**: A string describing the task or goal the user wants to accomplish. +- **`user_id`**: The ID of the user providing the input. + +#### **`ApprovalRequest`** +The `ApprovalRequest` model represents a request to approve a step or multiple steps. It includes the following attributes: + +- **`step_id`**: An optional string representing the specific step to approve. If not provided, the request applies to all steps. +- **`plan_id`**: The ID of the plan containing the step(s) to approve. +- **`session_id`**: The ID of the session associated with the approval request. +- **`approved`**: A boolean indicating whether the step(s) are approved. +- **`human_feedback`**: An optional string containing comments or feedback from the user. +- **`updated_action`**: An optional string representing a modified action based on feedback. +- **`user_id`**: The ID of the user making the approval request. + + +#### **`HumanFeedback`** +The `HumanFeedback` model captures user feedback on a specific step or plan. It includes the following attributes: + +- **`step_id`**: The ID of the step the feedback is related to. +- **`plan_id`**: The ID of the plan containing the step. +- **`session_id`**: The session ID associated with the feedback. +- **`approved`**: A boolean indicating if the step is approved. +- **`human_feedback`**: Optional comments or feedback provided by the user. +- **`updated_action`**: Optional modified action based on the feedback. +- **`user_id`**: The ID of the user providing the feedback. + +#### **`HumanClarification`** +The `HumanClarification` model represents clarifications provided by the user about a plan. It includes the following attributes: + +- **`plan_id`**: The ID of the plan requiring clarification. +- **`session_id`**: The session ID associated with the plan. +- **`human_clarification`**: The clarification details provided by the user. +- **`user_id`**: The ID of the user providing the clarification. + +#### **`ActionRequest`** +The `ActionRequest` model captures a request to perform an action within the system. It includes the following attributes: + +- **`session_id`**: The session ID associated with the action request. +- **`plan_id`**: The ID of the plan associated with the action. +- **`step_id`**: Optional ID of the step associated with the action. +- **`action`**: A string describing the action to be performed. +- **`user_id`**: The ID of the user requesting the action. + +#### **`ActionResponse`** +The `ActionResponse` model represents the response to an action request. It includes the following attributes: + +- **`status`**: A string indicating the status of the action (e.g., `success`, `failure`). +- **`message`**: An optional string providing additional details or context about the action's result. +- **`data`**: Optional data payload containing any relevant information from the action. +- **`user_id`**: The ID of the user associated with the action response. + +#### **`PlanStateUpdate`** +The `PlanStateUpdate` model represents an update to the state of a plan. It includes the following attributes: + +- **`plan_id`**: The ID of the plan being updated. +- **`session_id`**: The session ID associated with the plan. +- **`new_state`**: A string representing the new state of the plan (e.g., `in_progress`, `completed`, `failed`). +- **`user_id`**: The ID of the user making the state update. +- **`timestamp`**: An optional timestamp indicating when the update was made. + +--- + +#### **`GroupChatMessage`** +The `GroupChatMessage` model represents a message sent in a group chat context. It includes the following attributes: + +- **`message_id`**: A unique ID for the message. +- **`session_id`**: The session ID associated with the group chat. +- **`user_id`**: The ID of the user sending the message. +- **`content`**: The text content of the message. +- **`timestamp`**: A timestamp indicating when the message was sent. + +--- + +#### **`RequestToSpeak`** +The `RequestToSpeak` model represents a user's request to speak or take action in a group chat or collaboration session. It includes the following attributes: + +- **`request_id`**: A unique ID for the request. +- **`session_id`**: The session ID associated with the request. +- **`user_id`**: The ID of the user making the request. +- **`reason`**: A string describing the reason or purpose of the request. +- **`timestamp`**: A timestamp indicating when the request was made. + + +### Data Types + +#### **`DataType`** +The `DataType` enumeration defines the types of data used in the system. Possible values include: +- **`plan`**: Represents a plan data type. +- **`session`**: Represents a session data type. +- **`step`**: Represents a step data type. +- **`agent_message`**: Represents an agent message data type. + +--- + +#### **`BAgentType`** +The `BAgentType` enumeration defines the types of agents in the system. Possible values include: +- **`human`**: Represents a human agent. +- **`ai_assistant`**: Represents an AI assistant agent. +- **`external_service`**: Represents an external service agent. + +#### **`StepStatus`** +The `StepStatus` enumeration defines the possible statuses for a step. Possible values include: +- **`planned`**: Indicates the step is planned but not yet approved or completed. +- **`approved`**: Indicates the step has been approved. +- **`completed`**: Indicates the step has been completed. +- **`failed`**: Indicates the step has failed. + + +#### **`PlanStatus`** +The `PlanStatus` enumeration defines the possible statuses for a plan. Possible values include: +- **`in_progress`**: Indicates the plan is currently in progress. +- **`completed`**: Indicates the plan has been successfully completed. +- **`failed`**: Indicates the plan has failed. + + +#### **`HumanFeedbackStatus`** +The `HumanFeedbackStatus` enumeration defines the possible statuses for human feedback. Possible values include: +- **`pending`**: Indicates the feedback is awaiting review or action. +- **`addressed`**: Indicates the feedback has been addressed. +- **`rejected`**: Indicates the feedback has been rejected. + + +### Application Flow + +#### **Initialization** + +The initialization process sets up the necessary agents and context for a session. This involves: + +- **Generating Unique AgentIds**: Each agent is assigned a unique `AgentId` based on the `session_id`, ensuring that multiple sessions can operate independently. +- **Instantiating Agents**: Various agents, such as `PlannerAgent`, `HrAgent`, and `GroupChatManager`, are initialized and registered with unique `AgentIds`. +- **Setting Up Azure OpenAI Client**: The Azure OpenAI Chat Completion Client is initialized to handle LLM interactions with support for function calling, JSON output, and vision handling. +- **Creating Cosmos DB Context**: A `CosmosBufferedChatCompletionContext` is established for stateful interaction storage. + +**Code Reference: `utils.py`** + +**Steps:** +1. **Session ID Generation**: If `session_id` is not provided, a new UUID is generated. +2. **Agent Registration**: Each agent is assigned a unique `AgentId` and registered with the runtime. +3. **Azure OpenAI Initialization**: The LLM client is configured for advanced interactions. +4. **Cosmos DB Context Creation**: A buffered context is created for storing stateful interactions. +5. **Runtime Start**: The runtime is started, enabling communication and agent operation. + + + +### Input Task Handling + +When the `/input_task` endpoint receives an `InputTask`, it performs the following steps: + +1. Ensures a `session_id` is available. +2. Calls `initialize` to set up agents and context for the session. +3. Creates a `GroupChatManager` agent ID using the `session_id`. +4. Sends the `InputTask` message to the `GroupChatManager`. +5. Returns the `session_id` and `plan_id`. + +**Code Reference: `app.py`** + + @app.post("/input_task") + async def input_task(input_task: InputTask): + # Initialize session and agents + # Send InputTask to GroupChatManager + # Return status, session_id, and plan_id + +### Planning + +The `GroupChatManager` handles the `InputTask` by: + +1. Passing the `InputTask` to the `PlannerAgent`. +2. The `PlannerAgent` generates a `Plan` with detailed `Steps`. +3. The `PlannerAgent` uses LLM capabilities to create a structured plan based on the task description. +4. The plan and steps are stored in the Cosmos DB context. +5. The `GroupChatManager` starts processing the first step. + +**Code Reference: `group_chat_manager.py` and `planner.py`** + + # GroupChatManager.handle_input_task + plan: Plan = await self.send_message(message, self.planner_agent_id) + await self.memory.add_plan(plan) + # Start processing steps + await self.process_next_step(message.session_id) + + # PlannerAgent.handle_input_task + plan, steps = await self.create_structured_message(...) + await self.memory.add_plan(plan) + for step in steps: + await self.memory.add_step(step) + +### Step Execution and Approval + +For each step in the plan: + +1. The `GroupChatManager` retrieves the next planned step. +2. It sends an `ApprovalRequest` to the `HumanAgent` to get human approval. +3. The `HumanAgent` waits for human feedback (provided via the `/human_feedback` endpoint). +4. The step status is updated to `awaiting_feedback`. + +**Code Reference: `group_chat_manager.py`** + + async def process_next_step(self, session_id: str): + # Get plan and steps + # Find next planned step + # Update step status to 'awaiting_feedback' + # Send ApprovalRequest to HumanAgent + +### Human Feedback + +The human can provide feedback on a step via the `/human_feedback` endpoint: + +1. The `HumanFeedback` message is received by the FastAPI app. +2. The message is sent to the `HumanAgent`. +3. The `HumanAgent` updates the step with the feedback. +4. The `HumanAgent` sends the feedback to the `GroupChatManager`. +5. The `GroupChatManager` either proceeds to execute the step or handles rejections. + +**Code Reference: `app.py` and `human.py`** + + # app.py + @app.post("/human_feedback") + async def human_feedback(human_feedback: HumanFeedback): + # Send HumanFeedback to HumanAgent + + # human.py + @message_handler + async def handle_human_feedback(self, message: HumanFeedback, ctx: MessageContext): + # Update step with feedback + # Send feedback back to GroupChatManager + +### Action Execution by Specialized Agents + +If a step is approved: + +1. The `GroupChatManager` sends an `ActionRequest` to the appropriate specialized agent (e.g., `HrAgent`, `ProcurementAgent`). +2. The specialized agent executes the action using tools and LLMs. +3. The agent sends an `ActionResponse` back to the `GroupChatManager`. +4. The `GroupChatManager` updates the step status and proceeds to the next step. + +**Code Reference: `group_chat_manager.py` and `base_agent.py`** + + # GroupChatManager.execute_step + action_request = ActionRequest(...) + await self.send_message(action_request, agent_id) + + # BaseAgent.handle_action_request + # Execute action using tools and LLM + # Update step status + # Send ActionResponse back to GroupChatManager + +## Agents Overview + +### GroupChatManager + +**Role:** Orchestrates the entire workflow. +**Responsibilities:** + +- Receives `InputTask` from the user. +- Interacts with `PlannerAgent` to generate a plan. +- Manages the execution and approval process of each step. +- Handles human feedback and directs approved steps to the appropriate agents. + +**Code Reference: `group_chat_manager.py`** + +### PlannerAgent + +**Role:** Generates a detailed plan based on the input task. +**Responsibilities:** + +- Parses the task description. +- Creates a structured plan with specific actions and agents assigned to each step. +- Stores the plan in the context. +- Handles re-planning if steps fail. + +**Code Reference: `planner.py`** + +### HumanAgent + +**Role:** Interfaces with the human user for approvals and feedback. +**Responsibilities:** + +- Receives `ApprovalRequest` messages. +- Waits for human feedback (provided via the API). +- Updates steps in the context based on feedback. +- Communicates feedback back to the `GroupChatManager`. + +**Code Reference: `human.py`** + +### Specialized Agents + +**Types:** `HrAgent`, `LegalAgent`, `MarketingAgent`, etc. +**Role:** Execute specific actions related to their domain. +**Responsibilities:** + +- Receive `ActionRequest` messages. +- Perform actions using tools and LLM capabilities. +- Provide results and update steps in the context. +- Communicate `ActionResponse` back to the `GroupChatManager`. + +**Common Implementation:** +All specialized agents inherit from `BaseAgent`, which handles common functionality. +**Code Reference:** `base_agent.py`, `hr.py`, etc. + +![agent flow](./images/customize_solution/logic_flow.svg) + +## Persistent Storage with Cosmos DB + +The application uses Azure Cosmos DB to store and retrieve session data, plans, steps, and messages. This ensures that the state is maintained across different components and can handle multiple sessions concurrently. + +**Key Points:** + +- **Session Management:** Stores session information and current status. +- **Plan Storage:** Plans are saved and can be retrieved or updated. +- **Step Tracking:** Each step's status, actions, and feedback are stored. +- **Message History:** Chat messages between agents are stored for context. + +**Cosmos DB Client Initialization:** + +- Uses `ClientSecretCredential` for authentication. +- Asynchronous operations are used throughout to prevent blocking. + +**Code Reference: `cosmos_memory.py`** + +## Utilities + +### `initialize` Function + +**Location:** `utils.py` +**Purpose:** Initializes agents and context for a session, ensuring that each session has its own unique agents and runtime. +**Key Actions:** + +- Generates unique AgentIds with the `session_id`. +- Creates instances of agents and registers them with the runtime. +- Initializes `CosmosBufferedChatCompletionContext` for session-specific storage. +- Starts the runtime. + +**Example Usage:** + + runtime, cosmos_memory = await initialize(input_task.session_id) + +## Summary + +This application orchestrates a group of AI agents to accomplish user-defined tasks by: + +- Accepting tasks via HTTP endpoints. +- Generating detailed plans using LLMs. +- Delegating actions to specialized agents. +- Incorporating human feedback. +- Maintaining state using Azure Cosmos DB. + +Understanding the flow of data through the endpoints, agents, and persistent storage is key to grasping the logic of the application. Each component plays a specific role in ensuring tasks are planned, executed, and adjusted based on feedback, providing a robust and interactive system. + +For instructions to setup a local development environment for the solution, please see [local deployment guide](./LocalDeployment.md). diff --git a/docs/LocalDeployment.md b/docs/LocalDeployment.md new file mode 100644 index 00000000..a34ba583 --- /dev/null +++ b/docs/LocalDeployment.md @@ -0,0 +1,164 @@ +# Guide to local development + +## Requirements: + +- Python 3.10 or higher + PIP +- Azure CLI, and an Azure Subscription +- Visual Studio Code IDE + +# Local setup + +> **Note for macOS Developers**: If you are using macOS on Apple Silicon (ARM64) the DevContainer will **not** work. This is due to a limitation with the Azure Functions Core Tools (see [here](https://github.com/Azure/azure-functions-core-tools/issues/3112)). We recommend using the [Non DevContainer Setup](./NON_DEVCONTAINER_SETUP.md) instructions to run the accelerator locally. + +The easiest way to run this accelerator is in a VS Code Dev Containers, which will open the project in your local VS Code using the [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers): + +1. Start Docker Desktop (install it if not already installed) +1. Open the project: + [![Open in Dev Containers](https://img.shields.io/static/v1?style=for-the-badge&label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/microsoft/Multi-Agent-Custom-Automation-Engine-Solution-Accelerator) + +1. In the VS Code window that opens, once the project files show up (this may take several minutes), open a terminal window + +## Detailed Development Container setup instructions + +The solution contains a [development container](https://code.visualstudio.com/docs/remote/containers) with all the required tooling to develop and deploy the accelerator. To deploy the Chat With Your Data accelerator using the provided development container you will also need: + +* [Visual Studio Code](https://code.visualstudio.com) +* [Remote containers extension for Visual Studio Code](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) + +If you are running this on Windows, we recommend you clone this repository in [WSL](https://code.visualstudio.com/docs/remote/wsl) + +```cmd +git clone https://github.com/microsoft/Multi-Agent-Custom-Automation-Engine-Solution-Accelerator +``` + +Open the cloned repository in Visual Studio Code and connect to the development container. + +```cmd +code . +``` + +!!! tip + Visual Studio Code should recognize the available development container and ask you to open the folder using it. For additional details on connecting to remote containers, please see the [Open an existing folder in a container](https://code.visualstudio.com/docs/remote/containers#_quick-start-open-an-existing-folder-in-a-container) quickstart. + +When you start the development container for the first time, the container will be built. This usually takes a few minutes. **Please use the development container for all further steps.** + +The files for the dev container are located in `/.devcontainer/` folder. + +## Local deployment and debugging: + +1. **Clone the repository.** + +2. **Log into the Azure CLI:** + + - Check your login status using: + ```bash + az account show + ``` + - If not logged in, use: + ```bash + az login + ``` + - To specify a tenant, use: + ```bash + az login --tenant + ``` + +3. **Create a Resource Group:** + + - You can create it either through the Azure Portal or the Azure CLI: + ```bash + az group create --name --location EastUS2 + ``` + +4. **Deploy the Bicep template:** + + - You can use the Bicep extension for VSCode (Right-click the `.bicep` file, then select "Show deployment plane") or use the Azure CLI: + ```bash + az deployment group create -g -f deploy/macae-dev.bicep --query 'properties.outputs' + ``` + - **Note**: You will be prompted for a `principalId`, which is the ObjectID of your user in Entra ID. To find it, use the Azure Portal or run: + ```bash + az ad signed-in-user show --query id -o tsv + ``` + You will also be prompted for locations for Cosmos and OpenAI services. This is to allow separate regions where there may be service quota restrictions. + + - **Additional Notes**: + + **Role Assignments in Bicep Deployment:** + + The **macae-dev.bicep** deployment includes the assignment of the appropriate roles to AOAI and Cosmos services. If you want to modify an existing implementation—for example, to use resources deployed as part of the simple deployment for local debugging—you will need to add your own credentials to access the Cosmos and AOAI services. You can add these permissions using the following commands: + ```bash + az cosmosdb sql role assignment create --resource-group --account-name --role-definition-name "Cosmos DB Built-in Data Contributor" --principal-id --scope /subscriptions//resourceGroups//providers/Microsoft.DocumentDB/databaseAccounts/ + ``` + + ```bash + az role assignment create --assignee --role "Cognitive Services OpenAI User" --scope /subscriptions//resourceGroups//providers/Microsoft.CognitiveServices/accounts/ + ``` + **Using a Different Database in Cosmos:** + + You can set the solution up to use a different database in Cosmos. For example, you can name it something like autogen-dev. To do this: + 1. Change the environment variable **COSMOSDB_DATABASE** to the new database name. + 2. You will need to create the database in the Cosmos DB account. You can do this from the Data Explorer pane in the portal, click on the drop down labeled “_+ New Container_” and provide all the necessary details. + +6. **Create a `.env` file:** + + - Navigate to the `src` folder and create a `.env` file based on the provided `.env.sample` file. + +7. **Fill in the `.env` file:** + + - Use the output from the deployment or check the Azure Portal under "Deployments" in the resource group. + +8. **(Optional) Set up a virtual environment:** + + - If you are using `venv`, create and activate your virtual environment for both the frontend and backend folders. + +9. **Install requirements - frontend:** + + - In each of the frontend and backend folders - + Open a terminal in the `src` folder and run: + ```bash + pip install -r requirements.txt + ``` + +10. **Run the application:** + - From the src/backend directory: + ```bash + python app.py + ``` + - In a new terminal from the src/frontend directory + ```bash + python frontend_server.py + ``` + +10. Open a browser and navigate to `http://localhost:3000` +11. To see swagger API documentation, you can navigate to `http://localhost:8000/docs` + +## Debugging the solution locally + +You can debug the API backend running locally with VSCode using the following launch.json entry: + +``` + { + "name": "Python Debugger: Backend", + "type": "debugpy", + "request": "launch", + "cwd": "${workspaceFolder}/src/backend", + "module": "uvicorn", + "args": ["app:app", "--reload"], + "jinja": true + } +``` +To debug the python server in the frontend directory (frontend_server.py) and related, add the following launch.json entry: + +``` + { + "name": "Python Debugger: Frontend", + "type": "debugpy", + "request": "launch", + "cwd": "${workspaceFolder}/src/frontend", + "module": "uvicorn", + "args": ["frontend_server:app", "--port", "3000", "--reload"], + "jinja": true + } +``` + diff --git a/docs/azure_app_service_auth_setup.md b/docs/azure_app_service_auth_setup.md new file mode 100644 index 00000000..62c11834 --- /dev/null +++ b/docs/azure_app_service_auth_setup.md @@ -0,0 +1,58 @@ +# Set Up Authentication in Azure App Service + +## Step 1: Add Authentication in Azure App Service configuration + +1. Click on `Authentication` from left menu. + +![Authentication](./images/azure-app-service-auth-setup/AppAuthentication.png) + +2. Click on `+ Add Provider` to see a list of identity providers. + +![Authentication Identity](./images/azure-app-service-auth-setup/AppAuthenticationIdentity.png) + +3. Click on `+ Add Provider` to see a list of identity providers. + +![Add Provider](./images/azure-app-service-auth-setup/AppAuthIdentityProvider.png) + +4. Select the first option `Microsoft Entra Id` from the drop-down list. If `Create new app registration` is disabled, go to [Step 1a](#step-1a-creating-a-new-app-registration). + +![Add Provider](./images/azure-app-service-auth-setup/AppAuthIdentityProviderAdd.png) + +5. Accept the default values and click on `Add` button to go back to the previous page with the idenity provider added. + +![Add Provider](./images/azure-app-service-auth-setup/AppAuthIdentityProviderAdded.png) + +### Step 1a: Creating a new App Registration + +1. Click on `Home` and select `Microsoft Entra ID`. + +![Microsoft Entra ID](./images/azure-app-service-auth-setup/MicrosoftEntraID.png) + +2. Click on `App registrations`. + +![App registrations](./images/azure-app-service-auth-setup/Appregistrations.png) + +3. Click on `+ New registration`. + +![New Registrations](./images/azure-app-service-auth-setup/NewRegistration.png) + +4. Provide the `Name`, select supported account types as `Accounts in this organizational directory only(Contoso only - Single tenant)`, select platform as `Web`, enter/select the `URL` and register. + +![Add Details](./images/azure-app-service-auth-setup/AddDetails.png) + +5. After application is created sucessfully, then click on `Add a Redirect URL`. + +![Redirect URL](./images/azure-app-service-auth-setup/AddRedirectURL.png) + +6. Click on `+ Add a platform`. + +![+ Add platform](./images/azure-app-service-auth-setup/AddPlatform.png) + +7. Click on `Web`. + +![Web](./images/azure-app-service-auth-setup/Web.png) + +8. Enter the `web app URL` (Provide the app service name in place of XXXX) and Save. Then go back to [Step 1](#step-1-add-authentication-in-azure-app-service-configuration) and follow from _Point 4_ choose `Pick an existing app registration in this directory` from the Add an Identity Provider page and provide the newly registered App Name. +E.g. https://appservicename.azurewebsites.net/.auth/login/aad/callback + +![Add Details](./images/azure-app-service-auth-setup/WebAppURL.png) diff --git a/docs/images/azure-app-service-auth-setup/AddDetails.png b/docs/images/azure-app-service-auth-setup/AddDetails.png new file mode 100644 index 00000000..f36b596f Binary files /dev/null and b/docs/images/azure-app-service-auth-setup/AddDetails.png differ diff --git a/docs/images/azure-app-service-auth-setup/AddPlatform.png b/docs/images/azure-app-service-auth-setup/AddPlatform.png new file mode 100644 index 00000000..6c74919b Binary files /dev/null and b/docs/images/azure-app-service-auth-setup/AddPlatform.png differ diff --git a/docs/images/azure-app-service-auth-setup/AddRedirectURL.png b/docs/images/azure-app-service-auth-setup/AddRedirectURL.png new file mode 100644 index 00000000..d5cbcfac Binary files /dev/null and b/docs/images/azure-app-service-auth-setup/AddRedirectURL.png differ diff --git a/docs/images/azure-app-service-auth-setup/AppAuthIdentityProvider.png b/docs/images/azure-app-service-auth-setup/AppAuthIdentityProvider.png new file mode 100644 index 00000000..4cf476ad Binary files /dev/null and b/docs/images/azure-app-service-auth-setup/AppAuthIdentityProvider.png differ diff --git a/docs/images/azure-app-service-auth-setup/AppAuthIdentityProviderAdd.png b/docs/images/azure-app-service-auth-setup/AppAuthIdentityProviderAdd.png new file mode 100644 index 00000000..a57f0769 Binary files /dev/null and b/docs/images/azure-app-service-auth-setup/AppAuthIdentityProviderAdd.png differ diff --git a/docs/images/azure-app-service-auth-setup/AppAuthIdentityProviderAdded.png b/docs/images/azure-app-service-auth-setup/AppAuthIdentityProviderAdded.png new file mode 100644 index 00000000..d839d27e Binary files /dev/null and b/docs/images/azure-app-service-auth-setup/AppAuthIdentityProviderAdded.png differ diff --git a/docs/images/azure-app-service-auth-setup/AppAuthentication.png b/docs/images/azure-app-service-auth-setup/AppAuthentication.png new file mode 100644 index 00000000..286e1020 Binary files /dev/null and b/docs/images/azure-app-service-auth-setup/AppAuthentication.png differ diff --git a/docs/images/azure-app-service-auth-setup/AppAuthenticationIdentity.png b/docs/images/azure-app-service-auth-setup/AppAuthenticationIdentity.png new file mode 100644 index 00000000..d839d27e Binary files /dev/null and b/docs/images/azure-app-service-auth-setup/AppAuthenticationIdentity.png differ diff --git a/docs/images/azure-app-service-auth-setup/Appregistrations.png b/docs/images/azure-app-service-auth-setup/Appregistrations.png new file mode 100644 index 00000000..af2d3ae4 Binary files /dev/null and b/docs/images/azure-app-service-auth-setup/Appregistrations.png differ diff --git a/docs/images/azure-app-service-auth-setup/MicrosoftEntraID.png b/docs/images/azure-app-service-auth-setup/MicrosoftEntraID.png new file mode 100644 index 00000000..1f24b89d Binary files /dev/null and b/docs/images/azure-app-service-auth-setup/MicrosoftEntraID.png differ diff --git a/docs/images/azure-app-service-auth-setup/NewRegistration.png b/docs/images/azure-app-service-auth-setup/NewRegistration.png new file mode 100644 index 00000000..288f9af5 Binary files /dev/null and b/docs/images/azure-app-service-auth-setup/NewRegistration.png differ diff --git a/docs/images/azure-app-service-auth-setup/Web.png b/docs/images/azure-app-service-auth-setup/Web.png new file mode 100644 index 00000000..35f84645 Binary files /dev/null and b/docs/images/azure-app-service-auth-setup/Web.png differ diff --git a/docs/images/azure-app-service-auth-setup/WebAppURL.png b/docs/images/azure-app-service-auth-setup/WebAppURL.png new file mode 100644 index 00000000..40c6740e Binary files /dev/null and b/docs/images/azure-app-service-auth-setup/WebAppURL.png differ diff --git a/docs/images/customize_solution/logic_flow.svg b/docs/images/customize_solution/logic_flow.svg new file mode 100644 index 00000000..9914ae8a --- /dev/null +++ b/docs/images/customize_solution/logic_flow.svg @@ -0,0 +1,4 @@ + + + +
/input_task
/input_task
User
User
initialize all the agents
initialize all the ag...
send it to group chat manager
send it to group chat...
Orchestrator
Orchestrator
Planner agent
Planner agent
Interact with LLM model and generate a plan
Interact with LLM mo...
Store plan into cosmos DB
Store plan...
Plan Created
Plan Created
Approve the plan(stages)
Approve the plan(stag...
Human
Human
/approve_step_or_steps
/approve_step_or_steps
initialize all the agents
initialize all the agen...
Group chat manager
Group chat manager
Get the steps/plan from cosmos DB
Get the steps/p...
Human agent for feedback
Human agent for fee...
combine the feedback
combine the feedback
Yes
Yes
Approved/Rejected
Approved/Rejected
execute the step
execute the step
Approved/Accepted
Approved/Accepted
Rejected by human
Rejected by human
Rejected
Rejected
update record into cosmos DB
update reco...
Call the appropriate agent
Call the appropriate a...
Base Agent
Base Agent
HR Agent
HR Agent
Marketing Agent
Marketing Agent
Procurement Agent
Procurement Agent
Product Agent
Product Agent
Tech Support Agent
Tech Support Agent
Generic Agent
Generic Agent
response
response
User
User
response
response
Text is not SVG - cannot display
\ No newline at end of file diff --git a/docs/images/customize_solution/redoc_ui.png b/docs/images/customize_solution/redoc_ui.png new file mode 100644 index 00000000..cd7e445b Binary files /dev/null and b/docs/images/customize_solution/redoc_ui.png differ diff --git a/docs/images/readme/customerTruth.png b/docs/images/readme/customerTruth.png new file mode 100644 index 00000000..20cc799b Binary files /dev/null and b/docs/images/readme/customerTruth.png differ diff --git a/docs/images/readme/macae-application.png b/docs/images/readme/macae-application.png new file mode 100644 index 00000000..923c3ad2 Binary files /dev/null and b/docs/images/readme/macae-application.png differ diff --git a/docs/images/readme/macae-architecture.png b/docs/images/readme/macae-architecture.png new file mode 100644 index 00000000..95826744 Binary files /dev/null and b/docs/images/readme/macae-architecture.png differ diff --git a/docs/images/readme/oneClickDeploy.png b/docs/images/readme/oneClickDeploy.png new file mode 100644 index 00000000..846e0e1f Binary files /dev/null and b/docs/images/readme/oneClickDeploy.png differ diff --git a/docs/images/readme/userStory.png b/docs/images/readme/userStory.png new file mode 100644 index 00000000..ba485f54 Binary files /dev/null and b/docs/images/readme/userStory.png differ diff --git a/src/backend/.env.old b/src/backend/.env.old new file mode 100644 index 00000000..81af9306 --- /dev/null +++ b/src/backend/.env.old @@ -0,0 +1,19 @@ +COSMOSDB_ENDPOINT=https://biberty-cosmos-vpvzruptcepiu.documents.azure.com:443/ +COSMOSDB_DATABASE=autogen +COSMOSDB_CONTAINER=memory + +AZURE_OPENAI_ENDPOINT=https://biberty-aiservices.openai.azure.com/ +AZURE_OPENAI_MODEL_NAME=gpt-4o +AZURE_OPENAI_DEPLOYMENT_NAME=gpt-4o +AZURE_OPENAI_API_VERSION=2025-01-01-preview + +APPLICATIONINSIGHTS_INSTRUMENTATION_KEY=b57d47e6-1007-48bb-8c9b-71ecea7e3a12 +AZURE_AI_SUBSCRIPTION_ID=ff9b5430-90ea-44c0-8a00-e488c1bf56f4 +AZURE_AI_RESOURCE_GROUP=rg-biberty +AZURE_AI_PROJECT_NAME=biberty-aiproject +AZURE_AI_AGENT_PROJECT_CONNECTION_STRING="swedencentral.api.azureml.ms;ff9b5430-90ea-44c0-8a00-e488c1bf56f4;rg-biberty;biberty-aiproject" +AZURE_AI_MODEL_DEPLOYMENT_NAME=gpt-4o +APPLICATIONINSIGHTS_CONNECTION_STRING="InstrumentationKey=b57d47e6-1007-48bb-8c9b-71ecea7e3a12;IngestionEndpoint=https://swedencentral-0.in.applicationinsights.azure.com/;LiveEndpoint=https://swedencentral.livediagnostics.monitor.azure.com/;ApplicationId=7ca9c5a9-047e-4ba4-b2aa-6038b5f2d5f3" + +BACKEND_API_URL='http://localhost:8000' +FRONTEND_SITE_NAME='http://127.0.0.1:3000' \ No newline at end of file