Merge pull request #275624 from cdpark/horz-monitor-datafactory

User Story 246543: Q&M: Curate 27 SOX Azure Monitor horizontals articles to reduce duplication - Data Factory

Author: JamesJBarnett

articles/data-factory/diagnostic-logs-and-metrics-for-workflow-orchestration-manager.md

@@ -118,75 +118,4 @@ Data Factory offers comprehensive metrics for Airflow integration runtimes, allo
 
 ## Airflow metrics
 
-The following table lists the metrics available for Workflow Orchestration Manager. The table headings are:
-
-- **Metric**: The metric display name as it appears in the Azure portal.
-- **Name in REST API**: The metric name as referred to in the REST API.
-- **Description**: A description of the metric.
-- **Unit**: Unit of measure.
-- **Aggregation**: The default aggregation type. Valid values are Average, Minimum, Maximum, Total, and Count.
-- **Dimensions**: Dimensions available for the metric.
-- **Time grains**: Intervals at which the metric is sampled. For example, PT1M indicates that the metric is sampled every minute, PT30M every 30 minutes, PT1H every hour, and so on.
-- **DS export**: Whether the metric is exportable to Azure Monitor Logs via diagnostic settings.
-
-|Metric|Name in REST API|Description|Unit|Aggregation|Dimensions|Time grains|DS export|
-|---|---|---|---|---|---|---|
-|**Airflow Integration Runtime Celery Task Timeout Error** |`AirflowIntegrationRuntimeCeleryTaskTimeoutError` |Number of `AirflowTaskTimeout` errors raised when publishing Task to Celery Broker. |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime Collect DB Dags** |`AirflowIntegrationRuntimeCollectDBDags` |Milliseconds taken for fetching all Serialized DAGs from database. |Milliseconds |Average |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime Cpu Percentage** |`AirflowIntegrationRuntimeCpuPercentage` |CPU usage percentage of the Airflow integration runtime. |Percent |Average |`IntegrationRuntimeName`, `ContainerName`|PT1M |No|
-|**Airflow Integration Runtime Memory Usage** |`AirflowIntegrationRuntimeCpuUsage` |Millicores consumed by Airflow integration runtime, indicating the CPU resources used in thousandths of a CPU core. |Millicores |Average |`IntegrationRuntimeName`, `ContainerName`|PT1M |Yes|
-|**Airflow Integration Runtime Dag Bag Size** |`AirflowIntegrationRuntimeDagBagSize` |Number of DAGs found when the scheduler ran a scan based on its configuration. |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime Dag Callback Exceptions** |`AirflowIntegrationRuntimeDagCallbackExceptions` |Number of exceptions raised from DAG callbacks. When exceptions occur, it means DAG callback isn't working. |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime DAG File Refresh Error** |`AirflowIntegrationRuntimeDAGFileRefreshError` |Number of failures loading any DAG files. |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime DAG Processing Import Errors** |`AirflowIntegrationRuntimeDAGProcessingImportErrors` |Number of errors from trying to parse DAG files. |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime DAG Processing Last Duration** |`AirflowIntegrationRuntimeDAGProcessingLastDuration` |Seconds taken to load the specific DAG file. |Milliseconds |Average |`IntegrationRuntimeName`, `DagFile`|PT1M |No|
-|**Airflow Integration Runtime DAG Processing Last Run Seconds Ago** |`AirflowIntegrationRuntimeDAGProcessingLastRunSecondsAgo` |Seconds since <dag_file> was last processed. |Seconds |Average |`IntegrationRuntimeName`, `DagFile`|PT1M |No|
-|**Airflow Integration Runtime DAG ProcessingManager Stalls** |`AirflowIntegrationRuntimeDAGProcessingManagerStalls` |Number of stalled `DagFileProcessorManager`. |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime DAG Processing Processes** |`AirflowIntegrationRuntimeDAGProcessingProcesses` |Relative number of currently running DAG parsing processes. (For example, this delta is negative when, since the last metric was sent, processes were completed.) |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime DAG Processing Processor Timeouts** |`AirflowIntegrationRuntimeDAGProcessingProcessorTimeouts` |Number of file processors that were killed because they took too long. |Seconds |Average |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime DAG Processing Total Parse Time** |`AirflowIntegrationRuntimeDAGProcessingTotalParseTime` |Seconds taken to scan and import `dag_processing.file_path_queue_size` DAG files. |Seconds |Average |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime DAG Run Dependency Check** |`AirflowIntegrationRuntimeDAGRunDependencyCheck` |Milliseconds taken to check DAG dependencies. |Milliseconds |Average |`IntegrationRuntimeName`, `DagId`|PT1M |No|
-|**Airflow Integration Runtime DAG Run Duration Failed** |`AirflowIntegrationRuntimeDAGRunDurationFailed` |Seconds taken for a `DagRun` to reach failed state. |Milliseconds |Average |`IntegrationRuntimeName`, `DagId`|PT1M |No|
-|**Airflow Integration Runtime DAG Run Duration Success** |`AirflowIntegrationRuntimeDAGRunDurationSuccess` |Seconds taken for a `DagRun` to reach success state. |Milliseconds |Average |`IntegrationRuntimeName`, `DagId`|PT1M |No|
-|**Airflow Integration Runtime DAG Run First Task Scheduling Delay** |`AirflowIntegrationRuntimeDAGRunFirstTaskSchedulingDelay` |Seconds elapsed between the first task `start_date` and the `DagRun` expected start. |Milliseconds |Average |`IntegrationRuntimeName`, `DagId`|PT1M |No|
-|**Airflow Integration Runtime DAG Run Schedule Delay** |`AirflowIntegrationRuntimeDAGRunScheduleDelay` |Seconds of delay between the scheduled `DagRun` start date and the actual `DagRun` start date. |Milliseconds |Average |`IntegrationRuntimeName`, `DagId`|PT1M |No|
-|**Airflow Integration Runtime Executor Open Slots** |`AirflowIntegrationRuntimeExecutorOpenSlots` |Number of open slots on the executor. |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime Executor Queued Tasks** |`AirflowIntegrationRuntimeExecutorQueuedTasks` |Number of queued tasks on the executor. |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime Executor Running Tasks** |`AirflowIntegrationRuntimeExecutorRunningTasks` |Number of running tasks on the executor. |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime Job End** |`AirflowIntegrationRuntimeJobEnd` |Number of ended <job_name> job, for example, `SchedulerJob` and `LocalTaskJob`. |Count |Total |`IntegrationRuntimeName`, `Job`|PT1M |No|
-|**Airflow Integration Runtime Heartbeat Failure** |`AirflowIntegrationRuntimeJobHeartbeatFailure` |Number of failed Heartbeats for a <job_name> job, for example, `SchedulerJob` and `LocalTaskJob`. |Count |Total |`IntegrationRuntimeName`, `Job`|PT1M |No|
-|**Airflow Integration Runtime Job Start** |`AirflowIntegrationRuntimeJobStart` |Number of started <job_name> jobs, for example, `SchedulerJob` and `LocalTaskJob`. |Count |Total |`IntegrationRuntimeName`, `Job`|PT1M |No|
-|**Airflow Integration Runtime Memory Percentage** |`AirflowIntegrationRuntimeMemoryPercentage` |Memory Percentage used by Airflow integration runtime environments. |Percent |Average |`IntegrationRuntimeName`, `ContainerName`|PT1M |Yes|
-|**Airflow Integration Runtime Node Count** |`AirflowIntegrationRuntimeNodeCount` | |Count |Average |`IntegrationRuntimeName`, `ComputeNodeSize`|PT1M |Yes|
-|**Airflow Integration Runtime Operator Failures** |`AirflowIntegrationRuntimeOperatorFailures` |Total operator failures. |Count |Total |`IntegrationRuntimeName`, `Operator`|PT1M |No|
-|**Airflow Integration Runtime Operator Successes** |`AirflowIntegrationRuntimeOperatorSuccesses` |Total operator successes. |Count |Total |`IntegrationRuntimeName`, `Operator`|PT1M |No|
-|**Airflow Integration Runtime Pool Open Slots** |`AirflowIntegrationRuntimePoolOpenSlots` |Number of open slots in the pool. |Count |Total |`IntegrationRuntimeName`, `Pool`|PT1M |No|
-|**Airflow Integration Runtime Pool Queued Slots** |`AirflowIntegrationRuntimePoolQueuedSlots` |Number of queued slots in the pool. |Count |Total |`IntegrationRuntimeName`, `Pool`|PT1M |No|
-|**Airflow Integration Runtime Pool Running Slots** |`AirflowIntegrationRuntimePoolRunningSlots` |Number of running slots in the pool. |Count |Total |`IntegrationRuntimeName`, `Pool`|PT1M |No|
-|**Airflow Integration Runtime Pool Starving Tasks** |`AirflowIntegrationRuntimePoolStarvingTasks` |Number of starving tasks in the pool. |Count |Total |`IntegrationRuntimeName`, `Pool`|PT1M |No|
-|**Airflow Integration Runtime Scheduler Critical Section Busy** |`AirflowIntegrationRuntimeSchedulerCriticalSectionBusy` |Count of times a scheduler process tried to get a lock on the critical section (needed to send tasks to the executor) and found it locked by another process. |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime Scheduler Critical Section Duration** |`AirflowIntegrationRuntimeSchedulerCriticalSectionDuration` |Milliseconds spent in the critical section of a scheduler loop. Only a single scheduler can enter this loop at a time. |Milliseconds |Average |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime Scheduler Failed SLA Email Attempts** |`AirflowIntegrationRuntimeSchedulerFailedSLAEmailAttempts` |Number of failed SLA miss email notification attempts. |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime Scheduler Heartbeats** |`AirflowIntegrationRuntimeSchedulerHeartbeat` |Scheduler heartbeats. |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime Scheduler Orphaned Tasks Adopted** |`AirflowIntegrationRuntimeSchedulerOrphanedTasksAdopted` |Number of orphaned tasks adopted by the Scheduler. |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime Scheduler Orphaned Tasks Cleared** |`AirflowIntegrationRuntimeSchedulerOrphanedTasksCleared` |Number of orphaned tasks cleared by the Scheduler. |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime Scheduler Tasks Executable** |`AirflowIntegrationRuntimeSchedulerTasksExecutable` |Number of tasks that are ready for execution (set to queued) with respect to pool limits, DAG concurrency, executor state, and priority. |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime Scheduler Tasks Killed Externally** |`AirflowIntegrationRuntimeSchedulerTasksKilledExternally` |Number of tasks killed externally. |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime Scheduler Tasks Running** |`AirflowIntegrationRuntimeSchedulerTasksRunning` | |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime Scheduler Tasks Starving** |`AirflowIntegrationRuntimeSchedulerTasksStarving` |Number of tasks that can't be scheduled because of no open slot in the pool. |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime Started Task Instances** |`AirflowIntegrationRuntimeStartedTaskInstances` | |Count |Total |`IntegrationRuntimeName`, `DagId`, `TaskId`|PT1M |No|
-|**Airflow Integration Runtime Task Instance Created Using Operator** |`AirflowIntegrationRuntimeTaskInstanceCreatedUsingOperator` |Number of task instances created for a specific operator. |Count |Total |`IntegrationRuntimeName`, `Operator`|PT1M |No|
-|**Airflow Integration Runtime Task Instance Duration** |`AirflowIntegrationRuntimeTaskInstanceDuration` | |Milliseconds |Average |`IntegrationRuntimeName`, `DagId`, `TaskID`|PT1M |No|
-|**Airflow Integration Runtime Task Instance Failures** |`AirflowIntegrationRuntimeTaskInstanceFailures` |Overall task instances failures. |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime Task Instance Finished** |`AirflowIntegrationRuntimeTaskInstanceFinished` |Overall task instances finished. |Count |Total |`IntegrationRuntimeName`, `DagId`, `TaskId`, `State`|PT1M |No|
-|**Airflow Integration Runtime Task Instance Previously Succeeded** |`AirflowIntegrationRuntimeTaskInstancePreviouslySucceeded` |Number of previously succeeded task instances. |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime Task Instance Successes** |`AirflowIntegrationRuntimeTaskInstanceSuccesses` |Overall task instance successes. |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime Task Removed From DAG** |`AirflowIntegrationRuntimeTaskRemovedFromDAG` |Number of tasks removed for a specific DAG. (That is, the task no longer exists in DAG.) |Count |Total |`IntegrationRuntimeName`, `DagId`|PT1M |No|
-|**Airflow Integration Runtime Task Restored To DAG** |`AirflowIntegrationRuntimeTaskRestoredToDAG` |Number of tasks restored for a specific DAG. (That is, a task instance that was previously in a REMOVED state in the database is added to a DAG file.) |Count |Total |`IntegrationRuntimeName`, `DagId`|PT1M |No|
-|**Airflow Integration Runtime Triggers Blocked Main Thread** |`AirflowIntegrationRuntimeTriggersBlockedMainThread` |Number of triggers that blocked the main thread (likely because they weren't fully asynchronous). |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime Triggers Failed** |`AirflowIntegrationRuntimeTriggersFailed` |Number of triggers that errored before they could fire an event. |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime Triggers Running** |`AirflowIntegrationRuntimeTriggersRunning` |Number of triggers currently running for a triggerer (described by hostname). |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime Triggers Succeeded** |`AirflowIntegrationRuntimeTriggersSucceeded` |Number of triggers that fired at least one event. |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-|**Airflow Integration Runtime Zombie Tasks Killed** |`AirflowIntegrationRuntimeZombiesKilled` |Zombie tasks killed. |Count |Total |`IntegrationRuntimeName`|PT1M |No|
-
-For more information, see [Supported metrics for Microsoft.DataFactory/factories](/azure/azure-monitor/reference/supported-metrics/microsoft-datafactory-factories-metrics).
+To see the metrics available for Workflow Orchestration Manager, view the *Airflow* metrics listed in the [Supported metrics](monitor-data-factory-reference.md#supported-metrics-for-microsoftdatafactoryfactories) table.

articles/data-factory/how-to-configure-shir-for-log-analytics-collection.md

@@ -63,8 +63,9 @@ In the interface, when first configuring it, a suggested counter set will be rec
 
 ## View Events and Performance counter data in Log Analytics
 
-Consult this tutorial on [How to query data in Log Analytics](../azure-monitor/logs/log-analytics-tutorial.md).
-The two tables where the telemetry is saved are called Perf and Event respectively. The following query will check the row count to see if we have data flowing in. This would confirm if the instrumentation described above is working.
+To learn how to analyze monitoring data in the Azure Monitor Logs / Log Analytics store by using the Kusto query language (KQL), see [Kusto queries](monitor-data-factory.md#kusto-queries).
+
+The two tables where the telemetry is saved are called Perf and Event respectively. The following query checks the row count to see if we have data flowing in. This confirms if the instrumentation described above is working.
 
 ### Sample KQL queries
 

articles/data-factory/monitor-configure-diagnostics.md

@@ -20,17 +20,8 @@ Create or add diagnostic settings for your data factory.
 
 1. Give your setting a name, select **Send to Log Analytics**, and then select a workspace from **Log Analytics workspace**.
 
-    * In _Azure-Diagnostics_ mode, diagnostic logs flow into the _AzureDiagnostics_ table.
-    * In _Resource-Specific_ mode, diagnostic logs from Azure Data Factory flow into the following tables:
-      - _ADFActivityRun_
-      - _ADFPipelineRun_
-      - _ADFTriggerRun_
-      - _ADFSSISIntegrationRuntimeLogs_
-      - _ADFSSISPackageEventMessageContext_
-      - _ADFSSISPackageEventMessages_
-      - _ADFSSISPackageExecutableStatistics_
-      - _ADFSSISPackageExecutionComponentPhases_
-      - _ADFSSISPackageExecutionDataStatistics_
+    * In _Azure-Diagnostics_ mode, diagnostic logs flow into the [AzureDiagnostics](/azure/azure-monitor/reference/tables/AzureDiagnostics) table.
+    * In _Resource-Specific_ mode, diagnostic logs from Azure Data Factory flow into the tables listed in [Supported resource logs for Microsoft.DataFactory/factories](monitor-data-factory-reference.md#supported-resource-logs-for-microsoftdatafactoryfactories).
 
       You can select various logs relevant to your workloads to send to Log Analytics tables. For example: 
         - If you don't use SQL Server Integration Services (SSIS) at all, you don't need to select any SSIS logs. 

articles/data-factory/monitor-data-factory-reference.md

@@ -20,12 +20,12 @@ See [Monitor Data Factory](monitor-data-factory.md) for details on the data you
 ### Supported metrics for Microsoft.DataFactory/datafactories
 The following table lists the metrics available for the Microsoft.DataFactory/datafactories resource type.
 [!INCLUDE [horz-monitor-ref-metrics-tableheader](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-metrics-tableheader.md)]
-[!INCLUDE [Microsoft.DataFactory/datafactories](~/azure-reference-other-repo/azure-monitor-ref/supported-metrics/includes/microsoft-datafactory-datafactories-metrics-include.md)]
+[!INCLUDE [microsoft-datafactory-datafactories-metrics-include](~/reusable-content/ce-skilling/azure/includes/azure-monitor/reference/metrics/microsoft-datafactory-datafactories-metrics-include.md)]
 
 ### Supported metrics for Microsoft.DataFactory/factories
 The following table lists the metrics available for the Microsoft.DataFactory/factories resource type.
 [!INCLUDE [horz-monitor-ref-metrics-tableheader](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-metrics-tableheader.md)]
-[!INCLUDE [Microsoft.DataFactory/factories](~/azure-reference-other-repo/azure-monitor-ref/supported-metrics/includes/microsoft-datafactory-factories-metrics-include.md)]
+[!INCLUDE [microsoft-datafactory-factories-metrics-include](~/reusable-content/ce-skilling/azure/includes/azure-monitor/reference/metrics/microsoft-datafactory-factories-metrics-include.md)]
 
 > [!NOTE]
 > Except for _PipelineElapsedTimeRuns_, only events from completed, triggered activity and pipeline runs are emitted. In-progress and debug runs aren't emitted. However, events from all SSIS package executions are emitted, including those that are completed and in progress, regardless of their invocation methods. For example, you can invoke package executions on Azure-enabled SQL Server Data Tools, via T-SQL on SQL Server Management Studio, SQL Server Agent, or other designated tools, and as triggered or debug runs of Execute SSIS Package activities in Data Factory pipelines.
@@ -60,9 +60,10 @@ The following table lists the metrics available for the Microsoft.DataFactory/fa
 [!INCLUDE [horz-monitor-ref-resource-logs](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-resource-logs.md)]
 
 ### Supported resource logs for Microsoft.DataFactory/factories
-[!INCLUDE [Microsoft.DataFactory/factories](~/azure-reference-other-repo/azure-monitor-ref/supported-logs/includes/microsoft-datafactory-factories-logs-include.md)]
+[!INCLUDE [microsoft-datafactory-factories-logs-include](~/reusable-content/ce-skilling/azure/includes/azure-monitor/reference/logs/microsoft-datafactory-factories-logs-include.md)]
 
 [!INCLUDE [horz-monitor-ref-logs-tables](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-logs-tables.md)]
+
 ### Data factories
 
 Microsoft.DataFactory/factories