Merge pull request #267371 from v-thepet/adf-monitor

Azure Monitor horizontals - Azure Data Factory

Author: rmca14

articles/data-factory/.openpublishing.redirection.data-factory.json

@@ -1071,6 +1071,21 @@
             "redirect_url": "/azure/data-factory/connector-microsoft-fabric-lakehouse",
             "redirect_document_id": false
           },
+          {
+            "source_path_from_root": "/articles/data-factory/monitor-using-azure-monitor.md",
+            "redirect_url": "/azure/data-factory/monitor-data-factory",
+            "redirect_document_id": false
+          },
+          {
+            "source_path_from_root": "/articles/data-factory/monitor-metrics-alerts.md",
+            "redirect_url": "/azure/data-factory/monitor-data-factory",
+            "redirect_document_id": false
+          },
+          {
+            "source_path_from_root": "/articles/data-factory/monitor-schema-logs-events.md",
+            "redirect_url": "/azure/data-factory/monitor-data-factory-reference",
+            "redirect_document_id": false
+          },
           {
             "source_path_from_root": "/articles/data-factory/connector-troubleshoot-google-adwords.md",
             "redirect_url": "/azure/data-factory/connector-troubleshoot-google-ads",

articles/data-factory/TOC.yml

@@ -914,18 +914,14 @@ items:
       displayName: user properties
     - name: Monitor with Azure Monitor
       items:
-      - name: Overview
-        href: monitor-using-azure-monitor.md
-      - name: Azure Data Factory metrics and alerts
-        href: monitor-metrics-alerts.md
+      - name: Monitor
+        href: monitor-data-factory.md
       - name: Configure diagnostics settings and workspace
         href: monitor-configure-diagnostics.md
-      - name: Setup diagnostics logs via the Azure Monitor REST API
+      - name: Set up diagnostics logs via the Azure Monitor REST API
         href: monitor-logs-rest.md
       - name: Monitor SSIS operations with Azure Monitor
         href: monitor-ssis.md
-      - name: Schema of logs and events
-        href: monitor-schema-logs-events.md
     - name: Monitor with SDKs
       href: monitor-programmatically.md
     - name: Pipeline failure and error handling
@@ -971,9 +967,9 @@ items:
         - name: Diagnostic tool for self-hosted integration runtime
           href: self-hosted-integration-runtime-diagnostic-tool.md
           displayName: self hosted
-        - name: Monitoring self-hosted integration runtime in Azure
+        - name: Monitor self-hosted integration runtime in Azure
           href: monitor-shir-in-azure.md
-        - name: Configuring self-hosted integration runtime for log analytics collection
+        - name: Configure self-hosted integration runtime for log analytics collection
           href: how-to-configure-shir-for-log-analytics-collection.md
       displayName: self hosted, runtime
     - name: Azure-SSIS integration runtime
@@ -1396,6 +1392,8 @@ items:
     href: /cli/azure/datafactory
   - name: Compute optimized data flows retired
     href: compute-optimized-data-flow-retire.md
+  - name: Monitoring data reference
+    href: monitor-data-factory-reference.md
 - name: Resources
   items:
   - name: Whitepapers

articles/data-factory/monitor-schema-logs-events.md renamed to articles/data-factory/monitor-data-factory-reference.md

@@ -1,21 +1,110 @@
 ---
-title: Schema of logs and events 
-description: Learn about the schema used by Azure Data Factory logs and events for monitoring.
-author: nabhishek
-ms.author: abnarain
-ms.reviewer: jburchel
+title: Monitoring data reference for Azure Data Factory
+description: This article contains important reference material you need when you monitor Azure Data Factory.
+ms.date: 03/19/2024
+ms.custom: horz-monitor
+ms.topic: reference
+author: jonburchel
+ms.author: jburchel
 ms.service: data-factory
-ms.subservice: monitoring
-ms.topic: conceptual
-ms.date: 10/20/2023
 ---
 
-# Schema of logs and events
+# Azure Data Factory monitoring data reference
 
-This article describes the schema used by Azure Data Factory logs and events for monitoring.
+[!INCLUDE [horz-monitor-ref-intro](~/articles/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-intro.md)]
 
-## Monitor schema
-The following lists of attributes are used for monitoring.
+See [Monitor Data Factory](monitor-data-factory.md) for details on the data you can collect for Azure Data Factory and how to use it.
+
+[!INCLUDE [horz-monitor-ref-metrics-intro](~/articles/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-metrics-intro.md)]
+
+### Supported metrics for Microsoft.DataFactory/datafactories
+The following table lists the metrics available for the Microsoft.DataFactory/datafactories resource type.
+[!INCLUDE [horz-monitor-ref-metrics-tableheader](~/articles/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-metrics-tableheader.md)]
+[!INCLUDE [Microsoft.DataFactory/datafactories](~/azure-reference-other-repo/azure-monitor-ref/supported-metrics/includes/microsoft-datafactory-datafactories-metrics-include.md)]
+
+### Supported metrics for Microsoft.DataFactory/factories
+The following table lists the metrics available for the Microsoft.DataFactory/factories resource type.
+[!INCLUDE [horz-monitor-ref-metrics-tableheader](~/articles/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-metrics-tableheader.md)]
+[!INCLUDE [Microsoft.DataFactory/factories](~/azure-reference-other-repo/azure-monitor-ref/supported-metrics/includes/microsoft-datafactory-factories-metrics-include.md)]
+
+> [!NOTE]
+> Except for _PipelineElapsedTimeRuns_, only events from completed, triggered activity and pipeline runs are emitted. In-progress and debug runs aren't emitted. However, events from all SSIS package executions are emitted, including those that are completed and in progress, regardless of their invocation methods. For example, you can invoke package executions on Azure-enabled SQL Server Data Tools, via T-SQL on SQL Server Management Studio, SQL Server Agent, or other designated tools, and as triggered or debug runs of Execute SSIS Package activities in Data Factory pipelines.
+
+[!INCLUDE [horz-monitor-ref-metrics-dimensions-intro](~/articles/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-metrics-dimensions-intro.md)]
+[!INCLUDE [horz-monitor-ref-metrics-dimensions](~/articles/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-metrics-dimensions.md)]
+
+### Microsoft.DataFactory/datafactories
+
+- pipelineName
+- activityName
+
+### Microsoft.DataFactory/factories
+- ActivityType
+- PipelineName
+- FailureType
+- Name
+- IntegrationRuntimeName
+- ContainerName
+- DagFile
+- DagId
+- ComputeNodeSize
+- Job
+- Operator
+- Pool
+- TaskId
+- State
+- NodeName
+- CancelledBy
+- RunId
+
+[!INCLUDE [horz-monitor-ref-resource-logs](~/articles/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-resource-logs.md)]
+
+### Supported resource logs for Microsoft.DataFactory/factories
+[!INCLUDE [Microsoft.DataFactory/factories](~/azure-reference-other-repo/azure-monitor-ref/supported-logs/includes/microsoft-datafactory-factories-logs-include.md)]
+
+[!INCLUDE [horz-monitor-ref-logs-tables](~/articles/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-logs-tables.md)]
+### Data factories
+
+Microsoft.DataFactory/factories
+
+- [AzureActivity](/azure/azure-monitor/reference/tables/AzureActivity#columns)
+- [AzureMetrics](/azure/azure-monitor/reference/tables/AzureMetrics#columns)
+- [AzureDiagnostics](/azure/azure-monitor/reference/tables/AzureDiagnostics#columns)
+- [ADFActivityRun](/azure/azure-monitor/reference/tables/ADFActivityRun#columns)
+- [ADFPipelineRun](/azure/azure-monitor/reference/tables/ADFPipelineRun#columns)
+- [ADFTriggerRun](/azure/azure-monitor/reference/tables/ADFTriggerRun#columns)
+- [ADFSandboxActivityRun](/azure/azure-monitor/reference/tables/ADFSandboxActivityRun#columns)
+- [ADFSandboxPipelineRun](/azure/azure-monitor/reference/tables/ADFSandboxPipelineRun#columns)
+- [ADFSSISIntegrationRuntimeLogs](/azure/azure-monitor/reference/tables/ADFSSISIntegrationRuntimeLogs#columns)
+- [ADFSSISPackageEventMessageContext](/azure/azure-monitor/reference/tables/ADFSSISPackageEventMessageContext#columns)
+- [ADFSSISPackageEventMessages](/azure/azure-monitor/reference/tables/ADFSSISPackageEventMessages#columns)
+- [ADFSSISPackageExecutableStatistics](/azure/azure-monitor/reference/tables/ADFSSISPackageExecutableStatistics#columns)
+- [ADFSSISPackageExecutionComponentPhases](/azure/azure-monitor/reference/tables/ADFSSISPackageExecutionComponentPhases#columns)
+- [ADFSSISPackageExecutionDataStatistics](/azure/azure-monitor/reference/tables/ADFSSISPackageExecutionDataStatistics#columns)
+
+## Monitor and Log Analytics schemas for logs and events
+
+This section lists the attributes that the Azure Monitor and Log Analytics schemas use for monitoring logs and events.
+
+Log Analytics inherits the schema from Azure Monitor with the following exceptions:
+
+- The first letter in each column name is capitalized. For example, the column name **correlationId** in Monitor is **CorrelationId** in Log Analytics.
+- There's no **Level** column.
+- The dynamic **properties** column is preserved as the following dynamic JSON blob type.
+
+    | Azure Monitor column | Log Analytics column | Type |
+    | --- | --- | --- |
+    | $.properties.UserProperties | UserProperties | Dynamic |
+    | $.properties.Annotations | Annotations | Dynamic |
+    | $.properties.Input | Input | Dynamic |
+    | $.properties.Output | Output | Dynamic |
+    | $.properties.Error.errorCode | ErrorCode | int |
+    | $.properties.Error.message | ErrorMessage | string |
+    | $.properties.Error | Error | Dynamic |
+    | $.properties.Predecessors | Predecessors | Dynamic |
+    | $.properties.Parameters | Parameters | Dynamic |
+    | $.properties.SystemParameters | SystemParameters | Dynamic |
+    | $.properties.Tags | Tags | Dynamic |
 
 ### Activity-run log attributes
 
@@ -445,28 +534,11 @@ Here are the log attributes of data movements through each leg of data flow pipe
 | **executionPath**            | String | The path of execution for data flow task                           | `\Transformation\Data Flow Task` |
 | **resourceId**               | String | The unique ID of your data factory resource                                 | `/SUBSCRIPTIONS/<subscriptionID>/RESOURCEGROUPS/<resourceGroupName>/PROVIDERS/MICROSOFT.DATAFACTORY/FACTORIES/<dataFactoryName>` |
 
-## Log Analytics schema
-
-Log Analytics inherits the schema from Azure Monitor with the following exceptions:
-
-* The first letter in each column name is capitalized. For example, the column name "correlationId" in Monitor is "CorrelationId" in Log Analytics.
-* There's no "Level" column.
-* The dynamic "properties" column is preserved as the following dynamic JSON blob type.
-
-    | Azure Monitor column | Log Analytics column | Type |
-    | --- | --- | --- |
-    | $.properties.UserProperties | UserProperties | Dynamic |
-    | $.properties.Annotations | Annotations | Dynamic |
-    | $.properties.Input | Input | Dynamic |
-    | $.properties.Output | Output | Dynamic |
-    | $.properties.Error.errorCode | ErrorCode | int |
-    | $.properties.Error.message | ErrorMessage | string |
-    | $.properties.Error | Error | Dynamic |
-    | $.properties.Predecessors | Predecessors | Dynamic |
-    | $.properties.Parameters | Parameters | Dynamic |
-    | $.properties.SystemParameters | SystemParameters | Dynamic |
-    | $.properties.Tags | Tags | Dynamic |
+[!INCLUDE [horz-monitor-ref-activity-log](~/articles/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-activity-log.md)]
+- [Microsoft.DataFactory resource provider operations](/azure/role-based-access-control/resource-provider-operations#microsoftdatafactory)
 
 ## Related content
 
-[Monitor programmatically using SDKs](monitor-programmatically.md)
+- [Monitor Data Factory](monitor-data-factory.md)
+- [Monitor Azure resources with Azure Monitor](/azure/azure-monitor/essentials/monitor-azure-resource)
+- [Monitor programmatically using SDKs](monitor-programmatically.md)