Merge pull request #6405 from MicrosoftDocs/main

Auto Publish – main to live - 2025-08-05 22:05 UTC

Author: learn-build-service-prod[bot]

articles/ai-foundry/agents/how-to/tools/browser-automation-samples.md

@@ -0,0 +1,129 @@
+---
+title: How to use the Browser Automation tool with the Azure AI Foundry Agent Service
+titleSuffix: Azure AI Foundry
+description: Learn how to automate browser use and interact with websites using AI Agents.
+services: azure-ai-agent-service
+manager: nitinme
+ms.service: azure-ai-agent-service
+ms.topic: how-to
+ms.date: 07/29/2025
+author: aahill
+ms.author: aahi
+ms.custom: azure-ai-agents
+---
+
+# How to use the Browser Automation tool (preview)
+
+Use this article to find step-by-step instructions and code samples for using the Browser Automation tool in the Azure AI Foundry Agent Service.
+
+## Prerequisites
+
+* The requirements in the [Browser Automation overview](./deep-research.md).
+* Your Azure AI Foundry Project endpoint.
+    
+    [!INCLUDE [endpoint-string-portal](../../includes/endpoint-string-portal.md)]
+
+    Save this endpoint to an environment variable named `PROJECT_ENDPOINT`.
+
+* Your playwright connection ID. You can find it in the Azure AI Foundry portal by selecting **Management center** from the left navigation menu. Then select **Connected resources**.
+    
+    <!--
+    :::image type="content" source="../../media/tools/deep-research/bing-resource-name.png" alt-text="A screenshot showing the Playwright connection. " lightbox="../../media/tools/deep-research/bing-resource-name.png":::
+    -->
+    Save this name to an environment variable named `PLAYWRIGHT_CONNECTION_NAME`.
+
+* [!INCLUDE [model-name-portal](../../includes/model-name-portal.md)]
+
+    Save this name to an environment variable named `MODEL_DEPLOYMENT_NAME`.
+
+## Example code
+
+```python
+import os
+from azure.identity import DefaultAzureCredential
+from azure.ai.agents import AgentsClient
+from azure.ai.agents.models import MessageRole
+from azure.ai.projects import AIProjectClient
+
+project_endpoint = os.environ["PROJECT_ENDPOINT"]  # Ensure the PROJECT_ENDPOINT environment variable is set
+
+project_client = AIProjectClient(
+    endpoint=project_endpoint,
+    credential=DefaultAzureCredential()
+)
+
+playwright_connection = project_client.connections.get(
+    name=os.environ["PLAYWRIGHT_CONNECTION_NAME"]
+)
+print(playwright_connection.id)
+
+with project_client:
+    agent = project_client.agents.create_agent(
+        model=os.environ["MODEL_DEPLOYMENT_NAME"], 
+        name="my-agent", 
+        instructions="use the tool to respond", 
+        tools=[{
+            "type": "browser_automation",
+            "browser_automation": {
+            "connection": {
+                "id": playwright_connection.id,
+            }
+        }
+        }],
+    )
+
+    print(f"Created agent, ID: {agent.id}")
+
+    thread = project_client.agents.threads.create()
+    print(f"Created thread and run, ID: {thread.id}")
+
+    # Create message to thread
+    message = project_client.agents.messages.create(
+        thread_id=thread.id, 
+        role="user", 
+        content="something you want the tool to perform")
+    print(f"Created message: {message['id']}")
+
+    # Create and process an Agent run in thread with tools
+    run = project_client.agents.runs.create_and_process(
+        thread_id=thread.id, 
+        agent_id=agent.id,
+        )
+    print(f"Run created, ID: {run.id}")
+    print(f"Run finished with status: {run.status}")
+
+    if run.status == "failed":
+        print(f"Run failed: {run.last_error}")
+
+    run_steps = project_client.agents.run_steps.list(thread_id=thread.id, run_id=run.id)
+    for step in run_steps:
+        print(step)
+        print(f"Step {step['id']} status: {step['status']}")
+
+        # Check if there are tool calls in the step details
+        step_details = step.get("step_details", {})
+        tool_calls = step_details.get("tool_calls", [])
+
+        if tool_calls:
+            print("  Tool calls:")
+            for call in tool_calls:
+                print(f"    Tool Call ID: {call.get('id')}")
+                print(f"    Type: {call.get('type')}")
+
+                function_details = call.get("function", {})
+                if function_details:
+                    print(f"    Function name: {function_details.get('name')}")
+        print()  # add an extra newline between steps
+
+    # Delete the Agent when done
+    project_client.agents.delete_agent(agent.id)
+    print("Deleted agent")
+
+    # Fetch and log all messages
+    response_message = project_client.agents.messages.get_last_message_by_role(thread_id=thread.id, role=MessageRole.AGENT)
+    if response_message:
+        for text_message in response_message.text_messages:
+            print(f"Agent response: {text_message.text.value}")
+        for annotation in response_message.url_citation_annotations:
+            print(f"URL Citation: [{annotation.url_citation.title}]({annotation.url_citation.url})")
+```

articles/ai-foundry/agents/how-to/tools/browser-automation.md

@@ -0,0 +1,76 @@
+---
+title: 'How to use Browser Automation in Azure AI Foundry Agent Service'
+titleSuffix: Azure AI Foundry
+description: Learn how to automate browser tasks using AI agents.
+services: cognitive-services
+manager: nitinme
+ms.service: azure-ai-agent-service
+ms.topic: how-to
+ms.date: 07/28/2025
+author: aahill
+ms.author: aahi
+ms.custom: azure-ai-agents
+---
+
+# Browser Automation (preview)
+
+> [!WARNING]
+> The Browser Automation tool comes with significant security risks. Both errors in judgment by the AI and the presence of malicious or confusing instructions on web pages which the AI encounters may cause it to execute commands you or others do not intend, which could compromise the security of your or other users' browsers, computers, and any accounts to which the browser or AI has access, including personal, financial, or enterprise systems. By using the Browser Automation tool, you are acknowledging that you bear responsibility and liability for any use of it and of any resulting agents you create with it, including with respect to any other users to whom you make Browser Automation tool functionality available, including through resulting agents. We strongly recommend using the Browser Automation tool on low-privilege virtual machines with no access to sensitive data or critical resources.
+
+
+The Browser Automation tool enables users to perform real-world browser tasks through natural language prompts. Powered by [Microsoft Playwright Workspaces](/azure/playwright-testing/overview-what-is-microsoft-playwright-testing), it facilitates multi-turn conversations to automate browser-based workflows such as searching, navigating, filling forms, and booking.
+
+## How it works
+
+The interaction begins when the user sends a user query to an agent connected to the Browser Automation tool. For example, *"Show me all available yoga classes this week from the following url \<url\>."* Upon receiving the request, Azure AI Foundry Agent Service creates an isolated browser session using your own provisioned Playwright workspace. Each session is sandboxed for privacy and security. The browser session mimics a real user browsing experience, enabling interaction with complex web UIs (for example, class schedules, filters, or booking pages). The browser performs Playwright-driven actions, such as navigating to relevant pages, and applying filters or parameters based on user preferences (such as time, location, instructor). Combining the model with Playwright allows the model to see the browser screen by parsing the HTML or XML pages into DOM documents, make decisions, and perform actions like clicking, typing, and navigating websites. You should exercise caution when using this tool.
+
+An example flow would be:
+
+1. A user sends a request to the model that includes a call to the Browser Automation tool with the URL you want to go to.
+
+1. The Browser Automation tool receives a response from the model. If the response has action items, those items contain suggested actions to make progress toward the specified goal. For example an action might be a screenshot so the model can assess the current state with an updated screenshot or click with X/Y coordinates indicating where the mouse should be moved.
+
+1. The Browser Automation tool executes the action in a sandboxed environment.
+
+1. After executing the action, The Browser Automation tool captures the updated state of the environment as a screenshot.
+
+1. The tool sends a new request with the updated state, and repeats this loop until the model stops requesting actions or the user decides to stop.
+
+    The Browser Automation tool supports multi-turn conversations, allowing the user to refine their request and complete a booking.
+
+## Example scenarios:
+
+- Booking & Reservations: Automate form-filling and schedule confirmation across booking portals.
+
+- Product Discovery: Navigate ecommerce or review sites, search by criteria, and extract summaries.
+
+## Setup
+
+1. Create a [Playwright Workspace](https://aka.ms/pww/docs/manage-workspaces) resource.
+
+    1. [Generate an access token](https://aka.ms/pww/docs/manage-access-tokens) for the Playwright Workspace resource. 
+    
+    1. Access the workspace region endpoint in the **Workspace Details** page.
+    1. Give the project identity a "Contributor" role on the Playwright Workspace resource, or [configure a custom role](https://aka.ms/pww/docs/manage-workspace-access). 
+    
+1. Create a serverless connection in the Azure AI Foundry project with the Playwright workspace region endpoint and the Playwright workspace Access Token.
+
+    1. Go to the [Azure AI Foundry portal](https://ai.azure.com/) and select your project. Go to the **Management center** and select **connected resources**.
+
+    1. Create a new **Serverless Model** connection, and enter the following information.
+
+        * **Target URI**: The Playwright workspace region endpoint, for example `wss://{region}.api.playwright.microsoft.com/playwrightworkspaces/{workspaceId}/browsers`.
+
+        For more information on getting this value, see the [PlayWright documentation](https://aka.ms/pww/docs/configure-service-endpoint)
+
+    1. * **Key**: [Get the Playwright access token](https://aka.ms/pww/docs/generate-access-token)
+
+    For more information on creating a connection, see [Create a connection](../../../how-to/connections-add.md?pivots=fdp-project).
+
+1. Create a Browser Automation tool with your connection ID.
+
+## Transparency note
+
+Review the [transparency note](/azure/ai-foundry/responsible-ai/agents/transparency-note#enabling-autonomous-actions-with-or-without-human-input-through-action-tools) when using this tool. The Browser Automation tool is a tool that can perform real-world browser tasks through natural language prompts, enabling automated browsing activities without human intervention.
+
+Review the [responsible AI considerations](/azure/ai-foundry/responsible-ai/agents/transparency-note#considerations-when-choosing-a-use-case) when using this tool.

articles/ai-foundry/agents/how-to/tools/model-context-protocol-samples.md

@@ -6,7 +6,7 @@ services: cognitive-services
 manager: nitinme
 ms.service: azure-ai-agent-service
 ms.topic: how-to
-ms.date: 07/14/2025
+ms.date: 08/05/2025
 author: aahill
 ms.author: aahi
 zone_pivot_groups: selection-mcp-code
@@ -17,6 +17,133 @@ ms.custom: azure-ai-agents-code
 
 Use this article to find code samples for connecting Azure AI Foundry Agent Service with Model Context Protocol (MCP) servers.
 
+:::zone pivot="csharp"
+
+## Create a project client
+
+Create a client object that contains the endpoint for connecting to your AI project and other resources.
+
+> [!NOTE]
+> You can find an asynchronous example on [GitHub](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/ai/Azure.AI.Agents.Persistent/samples/Sample26_PersistentAgents_MCP.md)
+
+```csharp
+var projectEndpoint = System.Environment.GetEnvironmentVariable("PROJECT_ENDPOINT");
+var modelDeploymentName = System.Environment.GetEnvironmentVariable("MODEL_DEPLOYMENT_NAME");
+var mcpServerUrl = System.Environment.GetEnvironmentVariable("MCP_SERVER_URL");
+var mcpServerLabel = System.Environment.GetEnvironmentVariable("MCP_SERVER_LABEL");
+
+PersistentAgentsClient agentClient = new(projectEndpoint, new DefaultAzureCredential());
+
+```
+
+## Create the MCP tool definition
+
+Create the MCP tool definition and configure allowed tools.
+
+```csharp
+// Create MCP tool definition
+MCPToolDefinition mcpTool = new(mcpServerLabel, mcpServerUrl);
+
+// Configure allowed tools (optional)
+string searchApiCode = "search_azure_rest_api_code";
+mcpTool.AllowedTools.Add(searchApiCode);
+```
+
+Use the `MCPToolDefinition` during the agent initialization.
+
+```csharp
+PersistentAgent agent = agentClient.Administration.CreateAgent(
+   model: modelDeploymentName,
+   name: "my-mcp-agent",
+   instructions: "You are a helpful agent that can use MCP tools to assist users. Use the available MCP tools to answer questions and perform tasks.",
+   tools: [mcpTool]);
+```
+
+## Create a thread and add a message
+
+Create the thread, add the message containing a question for agent and start the run with MCP tool resources.
+
+```csharp
+PersistentAgentThread thread = agentClient.Threads.CreateThread();
+
+// Create message to thread
+PersistentThreadMessage message = agentClient.Messages.CreateMessage(
+    thread.Id,
+    MessageRole.User,
+    "Please summarize the Azure REST API specifications Readme");
+
+MCPToolResource mcpToolResource = new(mcpServerLabel);
+mcpToolResource.UpdateHeader("SuperSecret", "123456");
+ToolResources toolResources = mcpToolResource.ToToolResources();
+
+// Run the agent with MCP tool resources
+ThreadRun run = agentClient.Runs.CreateRun(thread, agent, toolResources);
+
+// Handle run execution and tool approvals
+while (run.Status == RunStatus.Queued || run.Status == RunStatus.InProgress || run.Status == RunStatus.RequiresAction)
+{
+    Thread.Sleep(TimeSpan.FromMilliseconds(1000));
+    run = agentClient.Runs.GetRun(thread.Id, run.Id);
+
+    if (run.Status == RunStatus.RequiresAction && run.RequiredAction is SubmitToolApprovalAction toolApprovalAction)
+    {
+        var toolApprovals = new List<ToolApproval>();
+        foreach (var toolCall in toolApprovalAction.SubmitToolApproval.ToolCalls)
+        {
+            if (toolCall is RequiredMcpToolCall mcpToolCall)
+            {
+                Console.WriteLine($"Approving MCP tool call: {mcpToolCall.Name}, Arguments: {mcpToolCall.Arguments}");
+                toolApprovals.Add(new ToolApproval(mcpToolCall.Id, approve: true)
+                {
+                    Headers = { ["SuperSecret"] = "123456" }
+                });
+            }
+        }
+
+        if (toolApprovals.Count > 0)
+        {
+            run = agentClient.Runs.SubmitToolOutputsToRun(thread.Id, run.Id, toolApprovals: toolApprovals);
+        }
+    }
+}
+```
+
+## Print the messages
+
+```csharp
+Pageable<PersistentThreadMessage> messages = agentClient.Messages.GetMessages(
+    threadId: thread.Id,
+    order: ListSortOrder.Ascending
+);
+
+foreach (PersistentThreadMessage threadMessage in messages)
+{
+    Console.Write($"{threadMessage.CreatedAt:yyyy-MM-dd HH:mm:ss} - {threadMessage.Role,10}: ");
+    foreach (MessageContent contentItem in threadMessage.ContentItems)
+    {
+        if (contentItem is MessageTextContent textItem)
+        {
+            Console.Write(textItem.Text);
+        }
+        else if (contentItem is MessageImageFileContent imageFileItem)
+        {
+            Console.Write($"<image from ID: {imageFileItem.FileId}>");
+        }
+        Console.WriteLine();
+    }
+}
+```
+
+## Optional: Delete the agent
+
+When you are done with your agent, you can delete it with:
+
+```csharp
+agentClient.Threads.DeleteThread(threadId: thread.Id);
+agentClient.Administration.DeleteAgent(agentId: agent.Id);
+```
+:::zone-end
+
 :::zone pivot="python"
 
 ## Initialize the client

articles/ai-foundry/agents/how-to/tools/overview.md

@@ -6,7 +6,7 @@ services: cognitive-services
 manager: nitinme
 ms.service: azure-ai-agent-service
 ms.topic: how-to
-ms.date: 06/17/2025
+ms.date: 07/29/2025
 author: aahill
 ms.author: aahi
 ms.custom: azure-ai-agents
@@ -36,6 +36,7 @@ To streamline workflows with your AI agent with capabilities to take actions. Th
 - **MCP tool**: Bring an existing Model Context Protocol (MCP) endpoint that you want to add to your AI agent.
 - **Function calling**: Write your own custom, stateless functions to define the expected behaviors.
 - **Azure Functions**: Write and manage your own custom, stateful functions. 
+- **Browser Automation**: Perform real-world browser tasks through natural language prompts.
 
 ## How does a tool work in the Foundry Agent Service?
 
@@ -72,6 +73,7 @@ The Foundry Agent Service provides the following built-in tools. You can use the
 |---------|---------|
 |[Azure AI Search](azure-ai-search.md)     | Use an existing Azure AI Search index to ground agents with data in the index, and chat with your data.        |
 |[Azure Functions](azure-functions.md)     | Leverage your Azure Functions to create intelligent, event-driven applications.        |
+|[Browser Automation](browser-automation.md)     | Perform real-world browser tasks through natural language prompts.         |
 |[Code Interpreter](code-interpreter.md)     | Enable agents to write and run Python code in a sandboxed execution environment.         |
 |[Deep Research (preview)](./deep-research.md) | Use OpenAI's advanced agentic research capability for analysis and reasoning. | 
 |[File Search](file-search.md)     | Augment agents with knowledge from outside its model, such as proprietary product information or documents provided by your users.          |

articles/ai-foundry/agents/toc.yml

@@ -47,6 +47,13 @@ items:
               displayName: azure functions, functions
             - name: How to use Azure Functions
               href: how-to/tools/azure-functions-samples.md
+        - name: Browser automation (preview)
+          items:
+          - name: Overview
+            href: how-to/tools/browser-automation.md
+            displayName: browser automation, web automation
+          - name: How to use browser automation
+            href: how-to/tools/browser-automation-samples.md
         - name: Code interpreter
           items:
             - name: Overview