Merge pull request #4289 from MicrosoftDocs/main

4/23/2025 PM Publish

Author: Taojunshen

articles/ai-services/agents/how-to/metrics.md

@@ -0,0 +1,81 @@
+---
+title: Monitor Azure AI Agent Service
+description: Start here to learn how to use Azure Monitor to capture and analyze metrics for your Azure AI Agent Service.
+ms.date: 03/20/2025
+ms.custom: horz-monitor, subject-monitoring
+ms.topic: conceptual
+author: aahill
+ms.author: aahi
+ms.service: azure-ai-agent-service
+---
+
+# Monitor Azure AI Agent Service
+
+[!INCLUDE [horz-monitor-intro](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-intro.md)]
+
+Monitoring is available for agents in a [standard agent setup](../quickstart.md?pivots=programming-language-csharp#choose-basic-or-standard-agent-setup).
+
+## Dashboards
+
+Azure AI Agent Service provides out-of-box dashboards. There are two key dashboards to monitor your resource: 
+
+- The metrics dashboard in the AI Foundry resource view 
+- The dashboard in the overview pane within the Azure portal 
+
+To access the monitoring dashboards, sign in to the [Azure portal](https://portal.azure.com) and then select **Monitoring** in the left navigation menu, then click **Metrics**.
+
+
+:::image type="content" source="../media/monitoring/dashboard.png" alt-text="Screenshot that shows out-of-box dashboards for a resource in the Azure portal." lightbox="../media/monitoring/dashboard.png" border="false":::
+
+## Azure monitor platform metrics
+
+Azure Monitor provides platform metrics for most services. These metrics are:
+
+* Individually defined for each namespace.
+* Stored in the Azure Monitor time-series metrics database.
+* Lightweight and capable of supporting near real-time alerting.
+* Used to track the performance of a resource over time.
+* Collection: Azure Monitor collects platform metrics automatically. No configuration is required.
+
+For a list of all metrics it's possible to gather for all resources in Azure Monitor, see [Supported metrics in Azure Monitor](/azure/azure-monitor/platform/metrics-supported).
+
+Azure AI Agent Service has commonality with a subset of Azure AI services. For a list of available metrics for Azure AI Agent Service, see the [monitoring data reference](../reference/monitor-service.md#metrics).
+
+## Analyze monitoring data
+
+There are many tools for analyzing monitoring data.
+
+### Azure Monitor tools
+
+Azure Monitor supports the [metrics explorer](/azure/azure-monitor/essentials/metrics-getting-started), a tool in the Azure portal that allows you to view and analyze metrics for Azure resources. For more information, see Analyze metrics with Azure Monitor metrics explorer.
+
+## Azure Monitor export tools
+
+You can get data out of Azure Monitor into other tools by using the [REST API for metrics](/rest/api/monitor/operation-groups) to extract metric data from the Azure Monitor metrics database. The API supports filter expressions to refine the data retrieved. For more information, see [Azure Monitor REST API reference](/rest/api/monitor/filter-syntax).
+
+To get started with the REST API for Azure Monitor, see [Azure monitoring REST API walkthrough](/azure/azure-monitor/essentials/rest-api-walkthrough).
+
+## Alerts
+
+Azure Monitor alerts proactively notify you when specific conditions are found in your monitoring data. Alerts allow you to identify and address issues in your system before your customers notice them. For more information, see Azure Monitor alerts.
+
+There are many sources of common alerts for Azure resources. [The Azure Monitor Baseline Alerts (AMBA)](https://aka.ms/amba) site provides a semi-automated method of implementing important platform metric alerts, dashboards, and guidelines. The site applies to a continually expanding subset of Azure services, including all services that are part of the Azure Landing Zone (ALZ).
+
+The common alert schema standardizes the consumption of Azure Monitor alert notifications. For more information, see [Common alert schema](/azure/azure-monitor/alerts/alerts-common-schema).
+
+[Metric alerts](/azure/azure-monitor/alerts/alerts-types#metric-alerts) evaluate resource metrics at regular intervals. Metric alerts can also apply multiple conditions and dynamic thresholds.
+
+Every organization's alerting needs vary and can change over time. Generally, all alerts should be actionable and have a specific intended response if the alert occurs. If an alert doesn't require an immediate response, the condition can be captured in a report rather than an alert. Some use cases might require alerting anytime certain error conditions exist. In other cases, you might need alerts for errors that exceed a certain threshold for a designated time period.
+
+Depending on what type of application you're developing with your use of Azure AI Agent Service, [Azure Monitor Application Insights](/azure/azure-monitor/overview) might offer more monitoring benefits at the application layer.
+
+### Azure AI Agent service alert rules
+
+You can set alerts for any metric listed in the [monitoring data reference](../reference/monitor-service.md).
+
+[!INCLUDE [horz-monitor-advisor-recommendations](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-advisor-recommendations.md)]
+
+## Related content
+
+- See [Monitoring data reference](../reference/monitor-service.md) for a reference of the metrics and other important values created for Azure AI Agent Service.
+- See [Monitoring Azure resources with Azure Monitor](/azure/azure-monitor/essentials/monitor-azure-resource) for general details on monitoring Azure resources.

articles/ai-services/agents/how-to/use-your-own-resources.md

@@ -41,7 +41,7 @@ Replace the parameter value for `aiServiceAccountResourceId` with the full arm r
 
     The value returned is the `aiServiceAccountResourceId` you need to use in the template.
 
-2. In the basic agent template file, replace the following placeholders:
+3. In the basic agent template file, replace the following placeholders:
 
     ```
     aiServiceAccountResourceId:/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.CognitiveServices/accounts/{serviceName}
@@ -74,9 +74,10 @@ Use an existing AI Services / Azure OpenAI, Azure Storage account, Azure Cosmos
     ```az login``` 
 2. Then run the command:
 
-    ```az search service show --resource-group  <your-resource-group> --name <your-storage-account>  --query "id" --output tsv```
-    
+    ```az storage account show --resource-group  <your-resource-group> --name <your-storage-account>  --query "id" --output tsv```
+   
      The output is the `aiStorageAccountResourceID` you need to use in the template.
+   
 3. In the standard agent template file, replace the following placeholders:
     
     ```

articles/ai-services/agents/media/monitoring/dashboard.png

@@ -0,0 +1,45 @@
+---
+title: Monitoring data reference for Azure AI Agent Service
+description: This article contains important reference material you need when you monitor Azure AI Agent Service by using Azure Monitor.
+ms.date: 03/24/2025
+ms.custom: horz-monitor, subject-monitoring
+ms.topic: reference
+author: aahill
+ms.author: aahi
+ms.service: azure-ai-agent-service
+---
+
+# Azure AI Agent Service monitoring data reference
+
+[!INCLUDE [horz-monitor-ref-intro](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-intro.md)]
+
+See [Monitor Azure AI Agent Service](../how-to/metrics.md) for details on the data you can collect on your agents.
+
+## Metrics
+
+Here are the most important metrics we think you should monitor for Azure AI Agent Service. Later in this article is a longer list of all available metrics which contains more details on metrics in this shorter list. _See the below list for most up to date information. We're working on refreshing the tables in the following sections._
+
+- [Runs](#category-agents)
+- [Indexed files](#category-agents)
+<!-- - Indexed files -->
+
+## Supported metrics
+
+This section lists all the automatically collected platform metrics for this service. These metrics are also part of the global list of [all platform metrics supported in Azure Monitor](/azure/azure-monitor/reference/supported-metrics/metrics-index#supported-metrics-per-resource-type).
+
+[!INCLUDE [horz-monitor-ref-metrics-tableheader](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-metrics-tableheader.md)]
+[!INCLUDE [Microsoft.MachineLearningServices/workspaces](~/reusable-content/ce-skilling/azure/includes/azure-monitor/reference/metrics/microsoft-machinelearningservices-workspaces-metrics-include.md)]
+
+## Category: Agents
+
+
+|Metric  |Name in REST API  |Unit  | Aggregation | Dimension | Time grains | DS Export |
+|---------|---------|---------|---------|---------|---------|---------|
+|Runs <br>  The number of runs in a given timeframe.     | `Runs`        | Count        | Total (sum), Average, Minimum, Maximum, Count        | `ResourceId`, `ProjectId`, `AgentId`, `StreamType`, `Region`, `StatusCode (successful, clienterrors, server errors)`, `RunStatus (started, completed, failed, cancelled, expired)` | PT1M | Yes |
+|Indexed files <br> Number of files indexed for file search    |  `IndexedFiles`       | Count        |  Count, Average, Minimum, Maximum       | `ResourceId`, `ProjectId`, `VectorStoreId`, `StreamType`, `Region`, `Status`, `ErrorCode` | PT1M | Yes |
+
+
+## Related content
+
+- See [Monitor Azure AI Agent Service](../how-to/metrics.md) for a description of monitoring Azure AI Agent Service.
+- See [Monitor Azure resources with Azure Monitor](/azure/azure-monitor/essentials/monitor-azure-resource) for details on monitoring Azure resources.

articles/ai-services/agents/media/monitoring/diagnostic-settings.png

@@ -58,6 +58,8 @@ items:
           href: ../openai/how-to/content-filters.md?context=/azure/ai-services/agents/context/context
         - name: Use virtual networks
           href: how-to/virtual-networks.md
+    - name: Service monitoring
+      href: how-to/metrics.md
     - name: Use the Visual Studio Code extension
       href: ../../ai-foundry/how-to/develop/vs-code-agents.md?context=/azure/ai-services/agents/context/context
 - name: Responsible AI
@@ -84,6 +86,8 @@ items:
           href: https://github.com/openai/openai-dotnet/blob/main/README.md
         - name: Python
           href: https://github.com/openai/openai-python/blob/main/README.md
+    - name: Data monitoring reference
+      href: reference/monitor-service.md
 - name: Resources
   items: 
     - name: Support and help options

articles/ai-services/agents/media/monitoring/log-analytics-metrics-query.png

@@ -7,7 +7,7 @@ author: aahill
 ms.author: aahi
 ms.service: azure-ai-agent-service
 ms.topic: overview
-ms.date: 01/30/2025
+ms.date: 04/23/2025
 ms.custom: azure-ai-agents
 ---
 
@@ -16,9 +16,19 @@ ms.custom: azure-ai-agents
 This article provides a summary of the latest releases and major documentation updates for Azure AI Agent Service.
 
 ## April 2025
+
+### Azure monitor integration
+
+You can now see metrics related to Agents in Azure monitor
+* The number of files indexed for file search.
+* The number of runs in a given timeframe.
+
+See the [Azure monitor](./how-to/metrics.md) and [metrics reference](./reference/monitor-service.md) articles for more information.
+
 ### BYO thread storage
 The Standard Agent Setup now supports **Bring Your Own (BYO) thread storage using an Azure Cosmos DB for NoSQL account**. This feature ensures all thread messages and conversation history are stored in your own resources. See the [Quickstart](./quickstart.md) for more information on how to deploy a Standard agent project.
 
+
 ## March 2025
 
 ### Microsoft Fabric tool

articles/ai-services/agents/reference/monitor-service.md

@@ -144,11 +144,11 @@ If you have your own DNS server hosted in Azure or on-premises, you need to crea
 
 We have two types of outbound; read only and read/write. Malicious actor's can't expoilt read only outbound, but read/write outbound can be. Azure Storage and Azure Frontdoor (the frontdoor.frontend service tag) are read/write outbound in our case. 
 
-You can mitigate this data exfiltration risk using our data exfiltration prevention solution. We use a service endpoint policy with an Azure Machine Learning alias to allow outbound to only Azure Machine Learning managed storage accounts. You don't need to open outbound to Storage on your firewall. 
+You can mitigate this data exfiltration risk using our data exfiltration prevention solution. We use a service endpoint policy with an Azure Machine Learning alias to allow outbound to only Azure Machine Learning managed storage accounts.
 
 :::image type="content" source="./media/how-to-network-isolation-planning/data-exfiltration-protection-diagram.png" alt-text="Diagram of network with exfiltration protection configuration.":::
 
-In this diagram, the compute instance and cluster need to access Azure Machine Learning managed storage accounts to get set-up scripts. Instead of opening the outbound to storage, you can use service endpoint policy with Azure Machine Learning alias to allow the storage access only to Azure Machine Learning storage accounts.
+In this diagram, the compute instance and cluster need to access Azure Machine Learning managed storage accounts to get set-up scripts. When opening the outbound to storage, you can use service endpoint policy with Azure Machine Learning alias to allow the storage access only to Azure Machine Learning storage accounts.
 
 ### Managed online endpoints