draft

v-thepet · v-thepet · commit edf0ecd10fb7 · 2024-03-25T21:38:04.000-07:00
diff --git a/articles/synapse-analytics/monitor-synapse-analytics-reference.md b/articles/synapse-analytics/monitor-synapse-analytics-reference.md
@@ -0,0 +1,157 @@
+---
+title: Monitoring data reference for Azure Synapse Analytics
+description: This article contains important reference material you need when you monitor Azure Synapse Analytics.
+ms.date: 03/25/2024
+ms.custom: horz-monitor
+ms.topic: reference
+author: jonburchel
+ms.author: jburchel
+ms.service: synapse-analytics
+---
+
+# Azure Synapse Analytics monitoring data reference
+
+[!INCLUDE [horz-monitor-ref-intro](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-intro.md)]
+
+See [Monitor Azure Synapse Analytics](monitor-synapse-analytics.md) for details on the data you can collect for Azure Synapse Analytics and how to use it.
+
+[!INCLUDE [horz-monitor-ref-metrics-intro](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-metrics-intro.md)]
+
+### Supported metrics for Microsoft.Synapse/workspaces
+The following table lists the metrics available for the Microsoft.Synapse/workspaces resource type.
+[!INCLUDE [horz-monitor-ref-metrics-tableheader](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-metrics-tableheader.md)]
+[!INCLUDE [Microsoft.Synapse/workspaces](~/azure-reference-other-repo/azure-monitor-ref/supported-metrics/includes/microsoft-synapse-workspaces-metrics-include.md)]
+
+### Supported metrics for Microsoft.Synapse/workspaces/bigDataPools
+The following table lists the metrics available for the Microsoft.Synapse/workspaces/bigDataPools resource type.
+[!INCLUDE [horz-monitor-ref-metrics-tableheader](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-metrics-tableheader.md)]
+[!INCLUDE [Microsoft.Synapse/workspaces/bigDataPools](~/azure-reference-other-repo/azure-monitor-ref/supported-metrics/includes/microsoft-synapse-workspaces-bigdatapools-metrics-include.md)]
+
+### Supported metrics for Microsoft.Synapse/workspaces/kustoPools
+The following table lists the metrics available for the Microsoft.Synapse/workspaces/kustoPools resource type.
+[!INCLUDE [horz-monitor-ref-metrics-tableheader](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-metrics-tableheader.md)]
+[!INCLUDE [Microsoft.Synapse/workspaces/kustoPools](~/azure-reference-other-repo/azure-monitor-ref/supported-metrics/includes/microsoft-synapse-workspaces-kustopools-metrics-include.md)]
+
+### Supported metrics for Microsoft.Synapse/workspaces/scopePools
+The following table lists the metrics available for the Microsoft.Synapse/workspaces/scopePools resource type.
+[!INCLUDE [horz-monitor-ref-metrics-tableheader](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-metrics-tableheader.md)]
+[!INCLUDE [Microsoft.Synapse/workspaces/scopePools](~/azure-reference-other-repo/azure-monitor-ref/supported-metrics/includes/microsoft-synapse-workspaces-scopepools-metrics-include.md)]
+
+### Supported metrics for Microsoft.Synapse/workspaces/sqlPools
+The following table lists the metrics available for the Microsoft.Synapse/workspaces/sqlPools resource type.
+[!INCLUDE [horz-monitor-ref-metrics-tableheader](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-metrics-tableheader.md)]
+[!INCLUDE [Microsoft.Synapse/workspaces/sqlPools](~/azure-reference-other-repo/azure-monitor-ref/supported-metrics/includes/microsoft-synapse-workspaces-sqlpools-metrics-include.md)]
+
+#### Details
+
+- Dedicated SQL pool measures performance in compute data warehouse units (cDWUs). Rather than surfacing details of individual nodes such as memory per node or number of CPUs per node, the intent behind emitting metrics such as `MemoryUsedPercent` and `CPUPercent`  is to show general usage trend over a period of time. These trends help administrators understand how an instance of dedicated SQL pool is utilized, and changes in footprint of memory and/or CPU could be a trigger for actions such as scale-up or scale-down of cDWUs, or investigating queries that might require optimization.
+
+  DWU used represents only a high-level representation of usage across the SQL pool and isn't a comprehensive indicator of utilization. To determine whether to scale up or down, consider all factors that DWU can impact, such as concurrency, memory, `tempdb`, and adaptive cache capacity. [Run your workload at different DWU settings](sql-data-warehouse-manage-compute-overview.md#finding-the-right-size-of-data-warehouse-units) to determine what works best to meet your business objectives.
+
+- Memory percentage reflects utilization even if the data warehouse is idle, not active workload memory consumption. Track this metric along with `tempdb` and Gen2 cache to decide whether you need to scale for more cache capacity to increase workload performance.
+
+- Failed and successful connections are reported for a particular data warehouse, not for the server itself.
+
+[!INCLUDE [horz-monitor-ref-metrics-dimensions-intro](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-metrics-dimensions-intro.md)]
+
+[!INCLUDE [horz-monitor-ref-metrics-dimensions](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-metrics-dimensions.md)]
+
+**Microsoft.Synapse/workspaces**
+
+Result, FailureType, Activity, ActivityType, Pipeline, Trigger,<br>
+EventType, TableName, LinkTableStatus, LinkConnectionName,<br>
+SQLPoolName, SQLDatabaseName, JobName, LogicalName, PartitionId, ProcessorInstance
+
+Use the `Result` dimension of the `IntegrationActivityRunsEnded`, `IntegrationPipelineRunsEnded`, `IntegrationTriggerRunsEnded`, and `BuiltinSqlPoolDataRequestsEnded` metrics to filter by Succeeded, Failed, or Cancelled final state.
+
+**Microsoft.Synapse/workspaces/bigDataPools**
+
+SubmitterId,<br>
+JobState, JobType, JobResult
+
+**Microsoft.Synapse/workspaces/kustoPools**
+Database, SealReason,<br>
+ComponentType, ComponentName,<br>
+ContinuousExportName, Result,<br>
+EventStatus, State, RoleInstance,<br>
+IngestionResultDetails, FailureKind,<br>
+MaterializedViewName, Kind, Result,<br>
+QueryStatus, ComponentType, CommandType
+
+**Microsoft.Synapse/workspaces/scopePools**
+
+JobType, JobResult
+
+**Microsoft.Synapse/workspaces/sqlPools**
+
+IsUserDefined, Result
+
+[!INCLUDE [horz-monitor-ref-resource-logs](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-resource-logs.md)]
+
+### Supported resource logs for Microsoft.Synapse/workspaces
+[!INCLUDE [Microsoft.Synapse/workspaces](~/azure-reference-other-repo/azure-monitor-ref/supported-logs/includes/microsoft-synapse-workspaces-logs-include.md)]
+
+> [!NOTE]  
+> The event **SynapseBuiltinSqlPoolRequestsEnded** is emitted only for queries that read data from storage. It's not emitted for queries that process only metadata.
+
+### Supported resource logs for Microsoft.Synapse/workspaces/bigDataPools
+[!INCLUDE [Microsoft.Synapse/workspaces/bigDataPools](~/azure-reference-other-repo/azure-monitor-ref/supported-logs/includes/microsoft-synapse-workspaces-bigdatapools-logs-include.md)]
+
+### Supported resource logs for Microsoft.Synapse/workspaces/kustoPools
+[!INCLUDE [Microsoft.Synapse/workspaces/kustoPools](~/azure-reference-other-repo/azure-monitor-ref/supported-logs/includes/microsoft-synapse-workspaces-kustopools-logs-include.md)]
+
+### Supported resource logs for Microsoft.Synapse/workspaces/scopePools
+[!INCLUDE [Microsoft.Synapse/workspaces/scopePools](~/azure-reference-other-repo/azure-monitor-ref/supported-logs/includes/microsoft-synapse-workspaces-scopepools-logs-include.md)]
+
+### Supported resource logs for Microsoft.Synapse/workspaces/sqlPools
+[!INCLUDE [Microsoft.Synapse/workspaces/sqlPools](~/azure-reference-other-repo/azure-monitor-ref/supported-logs/includes/microsoft-synapse-workspaces-sqlpools-logs-include.md)]
+
+### Dynamic Management Views (DMVs)
+
+For more information on these logs, see the following information:
+
+- [sys.dm_pdw_exec_requests](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-exec-requests-transact-sql?view=azure-sqldw-latest&preserve-view=true)
+- [sys.dm_pdw_request_steps](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-request-steps-transact-sql?view=azure-sqldw-latest&preserve-view=true)
+- [sys.dm_pdw_dms_workers](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-dms-workers-transact-sql?view=azure-sqldw-latest&preserve-view=true)
+- [sys.dm_pdw_waits](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-waits-transact-sql?view=azure-sqldw-latest&preserve-view=true)
+- [sys.dm_pdw_sql_requests](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-sql-requests-transact-sql?view=azure-sqldw-latest&preserve-view=true)
+
+[!INCLUDE [horz-monitor-ref-logs-tables](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-logs-tables.md)]
+
+### Synapse Workspaces
+Microsoft.Synapse/workspaces
+
+- [AzureActivity](/azure/azure-monitor/reference/tables/AzureActivity#columns)
+- [SynapseRbacOperations](/azure/azure-monitor/reference/tables/SynapseRbacOperations#columns)
+- [SynapseGatewayApiRequests](/azure/azure-monitor/reference/tables/SynapseGatewayApiRequests#columns)
+- [SynapseSqlPoolExecRequests](/azure/azure-monitor/reference/tables/SynapseSqlPoolExecRequests#columns)
+- [SynapseSqlPoolRequestSteps](/azure/azure-monitor/reference/tables/SynapseSqlPoolRequestSteps#columns)
+- [SynapseSqlPoolDmsWorkers](/azure/azure-monitor/reference/tables/SynapseSqlPoolDmsWorkers#columns)
+- [SynapseSqlPoolWaits](/azure/azure-monitor/reference/tables/SynapseSqlPoolWaits#columns)
+- [SynapseSqlPoolSqlRequests](/azure/azure-monitor/reference/tables/SynapseSqlPoolSqlRequests#columns)
+- [SynapseIntegrationPipelineRuns](/azure/azure-monitor/reference/tables/SynapseIntegrationPipelineRuns#columns)
+- [SynapseLinkEvent](/azure/azure-monitor/reference/tables/SynapseLinkEvent#columns)
+- [SynapseIntegrationActivityRuns](/azure/azure-monitor/reference/tables/SynapseIntegrationActivityRuns#columns)
+- [SynapseIntegrationTriggerRuns](/azure/azure-monitor/reference/tables/SynapseIntegrationTriggerRuns#columns)
+- [SynapseBigDataPoolApplicationsEnded](/azure/azure-monitor/reference/tables/SynapseBigDataPoolApplicationsEnded#columns)
+- [SynapseBuiltinSqlPoolRequestsEnded](/azure/azure-monitor/reference/tables/SynapseBuiltinSqlPoolRequestsEnded#columns)
+- [SQLSecurityAuditEvents](/azure/azure-monitor/reference/tables/SQLSecurityAuditEvents#columns)
+- [SynapseScopePoolScopeJobsEnded](/azure/azure-monitor/reference/tables/SynapseScopePoolScopeJobsEnded#columns)
+- [SynapseScopePoolScopeJobsStateChange](/azure/azure-monitor/reference/tables/AzureActivity#columns)
+- [SynapseDXCommand](/azure/azure-monitor/reference/tables/SynapseDXCommand#columns)
+- [SynapseDXFailedIngestion](/azure/azure-monitor/reference/tables/SynapseDXFailedIngestion#columns)
+- [SynapseDXIngestionBatching](/azure/azure-monitor/reference/tables/SynapseDXIngestionBatching#columns)
+- [SynapseDXQuery](/azure/azure-monitor/reference/tables/SynapseDXQuery#columns)
+- [SynapseDXSucceededIngestion](/azure/azure-monitor/reference/tables/SynapseDXSucceededIngestion#columns)
+- [SynapseDXTableUsageStatistics](/azure/azure-monitor/reference/tables/SynapseDXTableUsageStatistics#columns)
+- [SynapseDXTableDetails](/azure/azure-monitor/reference/tables/SynapseDXTableDetails#columns)
+
+[!INCLUDE [horz-monitor-ref-activity-log](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-ref-activity-log.md)]
+
+- [Microsoft.Sql resource provider operations](/azure/role-based-access-control/permissions/databases#microsoftsql)
+- [Microsoft.Synapse resource provider operations](/azure/role-based-access-control/permissions/analytics#microsoftsynapse)
+
+## Related content
+
+- See [Monitor Azure Synapse Analytics](monitor-synapse-analytics.md) for a description of monitoring Synapse Analytics.
+- See [Monitor Azure resources with Azure Monitor](/azure/azure-monitor/essentials/monitor-azure-resource) for details on monitoring Azure resources.
diff --git a/articles/synapse-analytics/monitor-synapse-analytics.md b/articles/synapse-analytics/monitor-synapse-analytics.md
@@ -0,0 +1,106 @@
+---
+title: Monitor Azure Synapse Analytics
+description: Start here to learn how to monitor Azure Synapse Analytics.
+ms.date: 03/25/2024
+ms.custom: horz-monitor
+ms.topic: conceptual
+author: jonburchel
+ms.author: jburchel
+ms.service: synapse-analytics
+---
+
+# Monitor Azure Synapse Analytics
+
+[!INCLUDE [horz-monitor-intro](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-intro.md)]
+
+## Monitoring methods
+
+There are several ways to monitor activities in your Synapse Analytics workspace. You can collect and analyze metrics and logs for built-in and serverless SQL pools, dedicated SQL pools, Azure Spark pools, and Data Explorer pools (preview). You can monitor current and historical activities for SQL, Apache Spark, pipelines and triggers, and integration runtimes.
+
+### Synapse Studio
+
+Open Synapse Studio and navigate to the **Monitor** hub to see a history of all the activities in the workspace and which ones are active.
+
+- Under **Integration**, you can monitor pipelines, triggers, and integration runtimes.
+- Under **Activities**, you can monitor Spark and SQL activities.
+
+For more information about monitoring in Synapse Studio, see [Monitor your Synapse Workspace](get-started-monitor.md).
+
+- For information on monitoring pipeline runs, see [Monitor pipeline runs in Synapse Studio](monitoring/how-to-monitor-pipeline-runs.md).
+- For information on monitoring Apache Spark applications, see [Monitor Apache Spark applications in Synapse Studio](monitoring/apache-spark-applications.md).
+- For information on monitoring SQL pools, see [Use Synapse Studio to monitor your SQL pools](monitoring/how-to-monitor-sql-pools.md).
+- For information on monitoring SQL requests, see [Monitor SQL requests in Synapse Studio](monitoring/how-to-monitor-sql-requests.md).
+
+### DMVs and Query Store
+
+To programmatically monitor Synapse SQL via T-SQL, Synapse Analytics provides a set of Dynamic Management Views (DMVs). These views are useful to troubleshoot and identify performance bottlenecks with your workload. For more information, see [DMVs](sql/query-history-storage-analysis.md#dmvs) and [Monitor your Azure Synapse Analytics dedicated SQL pool workload using DMVs](sql-data-warehouse/sql-data-warehouse-manage-monitor.md). For the list of DMVs that apply to Synapse SQL, see [Dedicated SQL pool Dynamic Management Views (DMVs)](sql/reference-tsql-system-views.md#dedicated-sql-pool-dynamic-management-views-dmvs).
+
+Query Store is a set of internal stores and DMVs that provide insight on query plan choice and performance. Query Store simplifies performance troubleshooting by helping find performance differences caused by query plan changes. For more information, see [Query Store](sql/query-history-storage-analysis.md#query-store).
+
+### Azure portal
+
+You can monitor Synapse Analytics workspaces and pools directly from their Azure portal pages. On the left sidebar menu, you can access the Azure **Activity log**, or select **Alerts**, **Metrics**, **Diagnostic settings**, **Logs**, or **Advisor recommendations** from the **Monitoring** section. This article provides more details about these options.
+
+[!INCLUDE [horz-monitor-resource-types](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-resource-types.md)]
+
+The resource types for Synapse Analytics include:
+
+- Microsoft.Synapse/workspaces
+- Microsoft.Synapse/workspaces/bigDataPools
+- Microsoft.Synapse/workspaces/kustoPools
+- Microsoft.Synapse/workspaces/scopePools
+- Microsoft.Synapse/workspaces/sqlPools
+
+For more information about the resource types for Azure Synapse Analytics, see [Azure Synapse Analytics monitoring data reference](monitor-synapse-analytics-reference.md).
+
+[!INCLUDE [horz-monitor-data-storage](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-data-storage.md)]
+
+Synapse Analytics supports storing monitoring data in Azure Storage or Azure Data Lake Storage Gen 2.
+
+[!INCLUDE [horz-monitor-platform-metrics](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-platform-metrics.md)]
+
+For lists of available platform metrics for Synapse Analytics, see [Synapse Analytics monitoring data reference](monitor-synapse-analytics-reference.md#metrics).
+
+Synapse Analytics Apache Spark pools support Prometheus metrics and Grafana dashboards. For more information, see [Monitor Apache Spark Applications metrics with Prometheus and Grafana](spark/use-prometheus-grafana-to-monitor-apache-spark-application-level-metrics.md).
+
+[!INCLUDE [horz-monitor-resource-logs](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-resource-logs.md)]
+
+For the available resource log categories, their associated Log Analytics tables, and the logs schemas for Synapse Analytics, see [Synapse Analytics monitoring data reference](monitor-synapse-analytics-reference.md#resource-logs).
+
+[!INCLUDE [horz-monitor-activity-log](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-activity-log.md)]
+
+[!INCLUDE [horz-monitor-analyze-data](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-analyze-data.md)]
+
+For a comparison of using Query Store, DMVs, Log Analytics, or Azure Data Explorer to analyze query history and performance, see [Historical query storage and analysis in Azure Synapse Analytics](query-history-storage-analysis.md).
+
+[!INCLUDE [horz-monitor-external-tools](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-external-tools.md)]
+
+[!INCLUDE [horz-monitor-kusto-queries](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-kusto-queries.md)]
+
+### Synapse Link table fail events
+Display sample failed Synapse Link table events.
+
+```kusto
+SynapseLinkEvent
+| where OperationName == "TableFail"
+| limit 100
+```
+[!INCLUDE [horz-monitor-alerts](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-alerts.md)]
+
+### Synapse Analytics alert rules
+
+The following table lists some suggested alerts for Synapse Analytics. These alerts are just examples. You can set alerts for any metric, log entry, or activity log entry that's listed in the [Synapse Analytics monitoring data reference](monitor-synapse-analytics-reference.md).
+
+| Alert type | Condition | Description  |
+|:---|:---|:---|
+| Metric| TempDB 75% | Maximum local tempdb used percentage greater than or equal to 75% of threshold value |
+| Metric| DWU Usage near 100% for 1 hour | Average DWU used percentage greater than 95% for one hour |
+| Log Analytics | SynapseSqlPoolRequestSteps | ShuffleMoveOperation over 10 million rows |
+
+For more details about creating these and other recommended alert rules, see [Create alerts for your Synapse Dedicated SQL Pool](https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/create-alerts-for-your-synapse-dedicated-sql-pool/ba-p/3773256).
+
+[!INCLUDE [horz-monitor-advisor-recommendations](~/reusable-content/ce-skilling/azure/includes/azure-monitor/horizontals/horz-monitor-advisor-recommendations.md)]
+
+## Related content
+- For a reference of the metrics, logs, and other important values created for Synapse Analytics, see [Synapse Analytics monitoring data reference](monitor-synapse-analytics-reference.md).
+- For general details on monitoring Azure resources with Azure Monitor, see [Monitor Azure resources with Azure Monitor](/azure/azure-monitor/essentials/monitor-azure-resource).
diff --git a/articles/synapse-analytics/sql-data-warehouse/sql-data-warehouse-concept-resource-utilization-query-activity.md b/articles/synapse-analytics/sql-data-warehouse/sql-data-warehouse-concept-resource-utilization-query-activity.md
