Skip to content

Commit a79443a

Browse files
v-dirichardsv-dirichards
authored
Merge pull request #260464 from rcdun/aoi_monitoring
Add Operator Insights monitoring documentation
2 parents 714ad71 + d6e2bbd commit a79443a

File tree

5 files changed

+237
-1
lines changed

5 files changed

+237
-1
lines changed

‎articles/azure-monitor/essentials/resource-logs-schema.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,7 @@ The schema for resource logs varies depending on the resource and log category.
8585
| Azure Machine Learning | [Diagnostic logging in Azure Machine Learning](../../machine-learning/monitor-resource-reference.md) |
8686
| Azure Media Services | [Media Services monitoring schemas](/azure/media-services/latest/monitoring/monitor-media-services#schemas) |
8787
| Network security groups |[Log Analytics for network security groups (NSGs)](../../virtual-network/virtual-network-nsg-manage-log.md) |
88+
| Azure Operator Insights | [Monitoring Azure Operator Insights data reference](../../operator-insights/monitor-operator-insights-data-reference.md#schemas) |
8889
| Azure Power BI Embedded | [Logging for Power BI Embedded in Azure](/power-bi/developer/azure-pbie-diag-logs) |
8990
| Recovery Services | [Data model for Azure Backup](../../backup/backup-azure-reports-data-model.md)|
9091
| Azure Service Bus |[Azure Service Bus logs](../../service-bus-messaging/service-bus-diagnostic-logs.md) |

‎articles/operator-insights/TOC.yml

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,4 +59,10 @@
5959
- name: Monitor and troubleshoot SFTP Ingestion Agents
6060
href: troubleshoot-sftp-agent.md
6161
- name: SFTP Ingestion Agent configuration reference
62-
href: sftp-agent-configuration.md
62+
href: sftp-agent-configuration.md
63+
- name: Monitor Azure Operator Insights
64+
href: monitor-operator-insights.md
65+
- name: Reference
66+
items:
67+
- name: Monitoring data
68+
href: monitor-operator-insights-data-reference.md

‎articles/operator-insights/index.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,8 @@ landingContent:
5959
url: dashboards-use.md
6060
- text: Query data in the Data Product
6161
url: data-query.md
62+
- text: Monitor Azure Operator Insights
63+
url: monitor-operator-insights.md
6264

6365
# Card
6466
- title: How to deploy Ingestion Agents for MCC protobuf EDRs

‎articles/operator-insights/monitor-operator-insights-data-reference.md

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
---
2+
title: Monitoring Azure Operator Insights data reference
3+
description: Important reference material needed when you monitor Azure Operator Insights
4+
author: rcdun
5+
ms.author: rdunstan
6+
ms.reviewer: rathishr
7+
ms.topic: reference
8+
ms.service: operator-insights
9+
ms.custom: horz-monitor
10+
ms.date: 12/15/2023
11+
---
12+
<!-- VERSION 2.3
13+
Template for monitoring data reference article for Azure services. This article is support for the main "Monitoring Azure Operator Insights" article for the service. -->
14+
15+
# Monitoring Azure Operator Insights data reference
16+
17+
This article describes the data you can collect in Azure Monitor for Azure Operator Insights. See [Monitoring Azure Operator Insights](monitor-operator-insights.md) for details on how to collect and analyze this monitoring data.
18+
19+
## Metrics
20+
21+
Azure Operator Insights doesn't provide metrics in Azure Monitor.
22+
23+
## Resource logs
24+
25+
This section lists the types of resource logs you can collect for Azure Operator Insights.
26+
27+
|Resource Log Type | Resource Provider / Type Namespace<br/> and link to individual logs |
28+
|-------|-----|
29+
| DataProducts| [Microsoft.NetworkAnalytics/DataProducts](/azure/azure-monitor/reference/supported-logs/microsoft-networkanalytics-dataproducts-logs) |
30+
31+
The DataProducts logs have the following categories:
32+
33+
- Ingestion (`Ingestion`): adding or changing data in the input storage account for a Data Product
34+
- Delete Ingested File (`IngestionDelete`): deleting data from the input storage account for a Data Product
35+
- Digestion (`Digestion`): processing the data available to a Data Product
36+
- Output Storage Read (`ReadStorage`): read access to the output storage account for a Data Product
37+
- Database Query (`DatabaseQuery`): query operations performed on the database of a Data Product
38+
39+
When you configure a diagnostic setting, you can select these categories individually, or select the Audit group. The Audit group contains all the categories except the Digestion category.
40+
41+
For reference, see a list of [all resource logs category types supported in Azure Monitor](/azure/azure-monitor/platform/resource-logs-schema).
42+
43+
## Azure Monitor Logs tables
44+
45+
This section lists all of the Azure Monitor Logs Kusto tables relevant to Azure Operator Insights and available for query by Log Analytics.
46+
47+
|Log type|Table name|Details|
48+
|--------|----------|-------|
49+
|Digestion|[AOIDigestion](/azure/azure-monitor/reference/tables/aoidigestion)| Contains `Digestion` logs|
50+
|Ingestion and storage |[AOIStorage](/azure/azure-monitor/reference/tables/aoistorage)| Contains `Ingestion`, `IngestionDelete` and `ReadStorage` |
51+
|Database queries|[AOIDatabaseQuery](/azure/azure-monitor/reference/tables/aoidatabasequery)| Contains `DatabaseQuery` |
52+
53+
54+
### Diagnostics tables
55+
56+
Azure Operator Insights uses the tables listed in [Azure Monitor Logs tables](#azure-monitor-logs-tables) to store resource log information. It doesn't use the Azure Diagnostics table.
57+
58+
## Activity log
59+
60+
The following table lists the operations that Azure Operator Insights can record in the Activity log. This table is a subset of the possible entries you might find in the activity log.
61+
62+
| Namespace | Description |
63+
|:---|:---|
64+
|`Microsoft.NetworkAnalytics`|Logs relating to creating, modifying and deleting Data Product resources|
65+
|`Microsoft.OperationalInsights/workspaces/query/AOI*`|Logs relating to querying Azure Operator Insights data in Azure Monitor|
66+
67+
See [all the possible resource provider operations in the activity log](/azure/role-based-access-control/resource-provider-operations).
68+
69+
For more information on the schema of Activity Log entries, see [Activity Log schema](/azure/azure-monitor/essentials/activity-log-schema).
70+
71+
## Schemas
72+
73+
Azure Operator Insights uses the following schemas for logs.
74+
75+
|Log type|Relates to|Schema link|
76+
|--------|----------|-----------|
77+
|Digestion logs|Processing (digestion) of data|[AOIDigestion](/azure/azure-monitor/reference/tables/aoidigestion)|
78+
|Storage logs|Operations on the Data Product's storage|[AOIStorage](/azure/azure-monitor/reference/tables/aoistorage)|
79+
|Database query logs|Queries run on the Data Product's database|[AOIDatabaseQuery](/azure/azure-monitor/reference/tables/aoidatabasequery)|
80+
81+
## See Also
82+
83+
- See [Monitoring Azure Operator Insights](monitor-operator-insights.md) for a description of monitoring Azure Operator Insights.
84+
- See [Monitoring Azure resources with Azure Monitor](/azure/azure-monitor/essentials/monitor-azure-resource) for details on monitoring Azure resources.

‎articles/operator-insights/monitor-operator-insights.md

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
---
2+
title: Monitoring Azure Operator Insights
3+
description: Start here to learn how to monitor Azure Operator Insights
4+
author: rcdun
5+
ms.author: rdunstan
6+
ms.reviewer: rathishr
7+
ms.service: operator-insights
8+
ms.topic: conceptual
9+
ms.custom: horz-monitor
10+
ms.date: 12/15/2023
11+
---
12+
13+
<!-- VERSION 2.3 2022_05_17
14+
Template for the main monitoring article for Azure services. -->
15+
16+
# Monitoring Azure Operator Insights
17+
18+
When you have critical applications and business processes relying on Azure resources, you want to monitor those resources for their availability, performance, and operation.
19+
20+
Azure Operator Insights Data Products use [Azure Monitor](/azure/azure-monitor/overview). They collect the same kinds of monitoring data as other Azure resources that are described in [Monitoring data from Azure resources](/azure/azure-monitor/essentials/monitor-azure-resource#monitoring-data-from-Azure-resources). See [Monitoring Azure Operator Insights data reference](monitor-operator-insights-data-reference.md) for detailed information on the monitoring data created by Data Products.
21+
22+
> [!TIP]
23+
> If you're unfamiliar with the features of Azure Monitor common to all Azure services that use it, read [Monitoring Azure resources with Azure Monitor](/azure/azure-monitor/essentials/monitor-azure-resource).
24+
25+
Ingestion agents also collect monitoring data that you or Microsoft Support can use for troubleshooting.
26+
27+
## Metrics for Data Products: Overview, collection and analysis
28+
29+
Azure Operator Insights doesn't provide metrics in Azure Monitor.
30+
31+
## Activity logs for Data Products: Overview, collection and analysis
32+
33+
The [Activity log](/azure/azure-monitor/essentials/activity-log) is a type of platform log in Azure that provides insight into subscription-level events. For Azure Operator Insights, the Activity log includes activities like creating a Data Product or changing its settings.
34+
35+
The Activity log is collected and stored automatically by Azure. You can:
36+
37+
- View the Activity log in the **Activity Log** for your Data Product.
38+
- Route the Activity Log to a Log Analytics workspace, which offers a rich query interface. See [Send to Log Analytics workspace](../azure-monitor/essentials/activity-log.md#send-to-log-analytics-workspace).
39+
- Route the Activity Log to other locations or download it. See [Azure Monitor activity log](../azure-monitor/essentials/activity-log.md).
40+
41+
## Resource logs for Data Products: Overview, collection and analysis
42+
43+
Resource logs provide an insight into operations that were performed within an Azure resource. This is known as the *data plane*. For Data Products, resource logs include ingestion (activity on files uploaded to Azure Operator Insights), digestion (processing the data in those files), and management of the processed data.
44+
45+
Resource logs aren't collected and stored until you create a *diagnostic setting* that routes them to one or more locations. We recommend routing them to a Log Analytics workspace, which stores the logs in [Azure Monitor Logs](../azure-monitor/logs/data-platform-logs.md). Log Analytics allows you to analyze the logs of all your Azure resources together in Azure Monitor Logs and take advantage of all the features available to Azure Monitor Logs including [log queries](../azure-monitor/logs/log-query-overview.md) and [log alerts](../azure-monitor/alerts/alerts-log.md).
46+
47+
For instructions on using getting started with Log Analytics and creating a diagnostic setting, see [Get started with resource logs for Data Products](#get-started-with-resource-logs-for-data-products). For more information about the data available, see [Data Product information in Azure Monitor Logs](#data-product-information-in-azure-monitor-logs).
48+
49+
### Get started with resource logs for Data Products
50+
51+
To start monitoring a Data Product with Azure Monitor Logs and Log Analytics:
52+
53+
1. Create a Log Analytics workspace by following [Create a Log Analytics workspace](../azure-monitor/logs/quick-create-workspace.md).
54+
1. In the **Diagnostic setting** view of your Data Product, create a diagnostic setting that routes the logs that you want to collect to the Log Analytics workspace. To use the example query in this procedure, include **Database Query** (in addition to any other category of logs that you want to collect).
55+
- For instructions, see [Create diagnostic setting to collect platform logs and metrics in Azure](/azure/azure-monitor/platform/diagnostic-settings). You can use the Azure portal, CLI, or PowerShell.
56+
- The categories of logs for Azure Operator Insights are listed in [Azure Operator Insights monitoring data reference](monitor-operator-insights-data-reference.md#resource-logs).
57+
1. To use the example query in this procedure, run a query on the data in your Data Product by following [Query data in the Data Product](data-query.md). This step ensures that Azure Monitor Logs has some data for your data product.
58+
1. Return to your Data Product resource and select **Logs** from the Azure Operator Insights menu to access Log Analytics.
59+
1. Run the following query to view the log for the query that you ran on your Data Product, replacing _[email protected]_ with the email address you used when you ran the query. You can also adapt the sample queries in [Sample Kusto queries](#sample-kusto-queries).
60+
```kusto
61+
AOIDatabaseQuery
62+
| where User has_cs "[email protected]"
63+
| take 100
64+
```
65+
66+
> [!IMPORTANT]
67+
> When you select **Logs** from the Azure Operator Insights menu, Log Analytics is opened with the query scope set to the current Data Product. This means that log queries will only include data from that resource. If you want to run a query that includes data from other Data Products or data from other Azure services, select **Logs** from the **Azure Monitor** menu. See [Log query scope and time range in Azure Monitor Log Analytics](/azure/azure-monitor/logs/scope) for details.
68+
69+
### Data Product information in Azure Monitor Logs
70+
71+
For a full list of the types of resource logs collected for Azure Operator Insights, see [Monitoring Azure Operator Insights data reference: Resource logs](monitor-operator-insights-data-reference.md#resource-logs).
72+
73+
Data in Azure Monitor Logs is stored in tables where each table has its own set of unique properties. For a list of the Azure Operator Insights tables used by Azure Monitor Logs and queryable by Log Analytics, see [Monitoring Azure Operator Insights data reference: Azure Monitor Logs tables](monitor-operator-insights-data-reference.md#azure-monitor-logs-tables).
74+
75+
All resource logs in Azure Monitor have the same fields followed by service-specific fields. The common schema is outlined in [Azure Monitor resource log schema](/azure/azure-monitor/essentials/resource-logs-schema) The schemas for Azure Operator Insights resource logs are found in the [Azure Operator Insights Data Reference: Schemas](monitor-operator-insights-data-reference.md#schemas).
76+
77+
### Sample Kusto queries
78+
79+
You can use the following example queries in a Log Analytics workspace to help you monitor your Data Products:
80+
81+
- Get all logs about rows that weren't digested successfully:
82+
83+
```kusto
84+
AOIDigestion
85+
| where Message startswith_cs "Failed to decode row"
86+
| take 100
87+
```
88+
89+
- Get a breakdown of the number of files that weren't digested, grouped by the top-level directory that they were uploaded to (typically the SiteId):
90+
91+
```kusto
92+
AOIDigestion
93+
| where Message startswith_cs "Failed to digest file"
94+
| parse FilePath with Source:string "/" *
95+
| summarize count() by Source
96+
```
97+
98+
- List all the queries run on a Quality of Experience - MCC Data Product by a particular user:
99+
100+
```kusto
101+
AOIDatabaseQuery
102+
| where DatabaseName has_cs "edrdp" and User has_cs "[email protected]"
103+
| take 100
104+
```
105+
106+
- List all the ingestion operations performed on input storage of a Data Product:
107+
108+
```kusto
109+
AOIStorage
110+
| where Category has_cs "Ingestion"
111+
| take 100
112+
```
113+
114+
- List all delete operations performed on input storage of a Data Product:
115+
116+
```kusto
117+
AOIStorage
118+
| where Category has_cs "IngestionDelete"
119+
| take 100
120+
```
121+
122+
- List all Read operations performed on storage of a Data Product:
123+
124+
```kusto
125+
AOIStorage
126+
| where Category has_cs "ReadStorage"
127+
| take 100
128+
```
129+
130+
For a list of common queries for Azure Operator Insights, see the [Log Analytics queries interface](/azure/azure-monitor/logs/queries).
131+
132+
## Monitoring for ingestion agents
133+
134+
Azure Operator Insights also requires ingestion agents deployed in your network.
135+
136+
Ingestion agents that we provide automatically collect metrics and logs for troubleshooting. Metrics and logs are stored on the VM on which you installed the agent, and aren't uploaded to Azure Monitor. For details, see the troubleshooting guidance for [MCC EDR Ingestion Agents](troubleshoot-mcc-edr-agent.md) or [SFTP Ingestion Agents](troubleshoot-sftp-agent.md).
137+
138+
## Next steps
139+
140+
- For a reference of the Azure Monitor data created by Azure Operator Insights, see [Monitoring Azure Operator Insights data reference](monitor-operator-insights-data-reference.md).
141+
- For more information about metrics and logs for MCC EDR ingestion agents, see [Monitor and troubleshoot MCC EDR Ingestion Agents for Azure Operator Insights](troubleshoot-mcc-edr-agent.md).
142+
- For more information about metrics and logs for SFTP ingestion agents, see [Monitor and troubleshoot SFTP Ingestion Agents for Azure Operator Insights](troubleshoot-sftp-agent.md).
143+
- For background on Azure Monitor, see [Monitoring Azure resources with Azure Monitor](/azure/azure-monitor/essentials/monitor-azure-resource) .

0 commit comments

Comments
 (0)