You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/synapse-analytics/sql-data-warehouse/sql-data-warehouse-concept-resource-utilization-query-activity.md
+11-9Lines changed: 11 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -15,10 +15,11 @@ ms.custom: azure-synapse
15
15
16
16
# Monitoring resource utilization and query activity in Azure Synapse Analytics
17
17
18
-
Azure Synapse Analytics provides a rich monitoring experience within the Azure portal to surface insights regarding your data warehouse workload. The Azure portal is the recommended tool when monitoring your data warehouse as it provides configurable retention periods, alerts, recommendations, and customizable charts and dashboards for metrics and logs. The portal also enables you to integrate with other Azure monitoring services such as Azure Monitor (logs) with Log analytics to provide a holistic monitoring experience for not only your data warehouse but also your entire Azure analytics platform for an integrated monitoring experience. This documentation describes what monitoring capabilities are available to optimize and manage your analytics platform with SQL Analytics.
18
+
Azure Synapse Analytics provides a rich monitoring experience within the Azure portal to surface insights regarding your data warehouse workload. The Azure portal is the recommended tool when monitoring your data warehouse as it provides configurable retention periods, alerts, recommendations, and customizable charts and dashboards for metrics and logs. The portal also enables you to integrate with other Azure monitoring services such as Azure Monitor (logs) with Log analytics to provide a holistic monitoring experience for not only your data warehouse but also your entire Azure analytics platform for an integrated monitoring experience. This documentation describes what monitoring capabilities are available to optimize and manage your analytics platform.
19
19
20
-
## Resource utilization
21
-
The following metrics are available in the Azure portal for SQL Analytics. These metrics are surfaced through [Azure Monitor](https://docs.microsoft.com/azure/azure-monitor/platform/data-collection#metrics).
20
+
## Resource utilization
21
+
22
+
The following metrics are available in the Azure portal for Synapse SQL. These metrics are surfaced through [Azure Monitor](../../azure-monitor/platform/data-collection.md#metrics?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json).
22
23
23
24
24
25
| Metric Name | Description | Aggregation Type |
@@ -43,21 +44,22 @@ The following metrics are available in the Azure portal for SQL Analytics. These
43
44
44
45
Things to consider when viewing metrics and setting alerts:
45
46
46
-
- DWU used represents only a **high-level representation of usage** across the SQL pool and is not meant to be a comprehensive indicator of utilization. To determine whether to scale up or down, consider all factors which can be impacted by DWU such as concurrency, memory, tempdb, and adaptive cache capacity. We recommend [running your workload at different DWU settings](https://docs.microsoft.com/azure/sql-data-warehouse/sql-data-warehouse-manage-compute-overview#finding-the-right-size-of-data-warehouse-units) to determine what works best to meet your business objectives.
47
+
- DWU used represents only a **high-level representation of usage** across the SQL pool and is not meant to be a comprehensive indicator of utilization. To determine whether to scale up or down, consider all factors which can be impacted by DWU such as concurrency, memory, tempdb, and adaptive cache capacity. We recommend [running your workload at different DWU settings](sql-data-warehouse-manage-compute-overview#finding-the-right-size-of-data-warehouse-units) to determine what works best to meet your business objectives.
47
48
- Failed and successful connections are reported for a particular data warehouse - not for the logical server
48
49
- Memory percentage reflects utilization even if the data warehouse is in idle state - it does not reflect active workload memory consumption. Use and track this metric along with others (tempdb, gen2 cache) to make a holistic decision on if scaling for additional cache capacity will increase workload performance to meet your requirements.
49
50
50
-
51
51
## Query activity
52
-
For a programmatic experience when monitoring SQL Analytics via T-SQL, the service provides a set of Dynamic Management Views (DMVs). These views are useful when actively troubleshooting and identifying performance bottlenecks with your workload.
53
52
54
-
To view the list of DMVs that SQL Analytics provides, refer to this [documentation](https://docs.microsoft.com/azure/sql-data-warehouse/sql-data-warehouse-reference-tsql-system-views#sql-data-warehouse-dynamic-management-views-dmvs).
53
+
For a programmatic experience when monitoring Synapse SQL via T-SQL, the service provides a set of Dynamic Management Views (DMVs). These views are useful when actively troubleshooting and identifying performance bottlenecks with your workload.
54
+
55
+
To view the list of DMVs that apply to Synapse SQL, refer to this [documentation](sql-data-warehouse-reference-tsql-system-views#sql-data-warehouse-dynamic-management-views-dmvs).
55
56
56
57
## Metrics and diagnostics logging
57
-
Both metrics and logs can be exported to Azure Monitor, specifically the [Azure Monitor logs](https://docs.microsoft.com/azure/log-analytics/log-analytics-overview) component and can be programmatically accessed through [log queries](https://docs.microsoft.com/azure/log-analytics/log-analytics-tutorial-viewdata). The log latency for SQL Analytics is about 10-15 minutes. For more details on the factors impacting latency, visit the following documentation.
58
58
59
+
Both metrics and logs can be exported to Azure Monitor, specifically the [Azure Monitor logs](../../azure-monitor/log-query/log-query-overview.md?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json) component and can be programmatically accessed through [log queries](../../azure-monitor/log-query/get-started-portal.md?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json). The log latency for Synapse SQL is about 10-15 minutes. For more details on the factors impacting latency, visit the following documentation.
59
60
60
61
## Next steps
62
+
61
63
The following How-to guides describe common scenarios and use cases when monitoring and managing your data warehouse:
62
64
63
-
-[Monitor your data warehouse workload with DMVs](https://docs.microsoft.com/azure/sql-data-warehouse/sql-data-warehouse-manage-monitor)
65
+
-[Monitor your data warehouse workload with DMVs](/sql-data-warehouse/sql-data-warehouse-manage-monitor?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json&view=azure-sqldw-latest)
Copy file name to clipboardExpand all lines: articles/synapse-analytics/sql-data-warehouse/sql-data-warehouse-develop-best-practices-transactions.md
+18-13Lines changed: 18 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
---
2
2
title: Optimizing transactions
3
-
description: Learn how to optimize the performance of your transactional code in SQL Analytics while minimizing risk for long rollbacks.
3
+
description: Learn how to optimize the performance of your transactional code in Synapse SQL while minimizing risk for long rollbacks.
4
4
services: synapse-analytics
5
5
author: XiaoyuMSFT
6
6
manager: craigg
@@ -13,19 +13,22 @@ ms.reviewer: igorstan
13
13
ms.custom: seo-lt-2019, azure-synapse
14
14
---
15
15
16
-
# Optimizing transactions in SQL Analytics
17
-
Learn how to optimize the performance of your transactional code in SQL Analytics while minimizing risk for long rollbacks.
16
+
# Optimizing transactions in Synapse SQL
17
+
18
+
Learn how to optimize the performance of your transactional code in Synapse SQL while minimizing risk for long rollbacks.
18
19
19
20
## Transactions and logging
20
-
Transactions are an important component of a relational database engine. SQL Analytics uses transactions during data modification. These transactions can be explicit or implicit. Single INSERT, UPDATE, and DELETE statements are all examples of implicit transactions. Explicit transactions use BEGIN TRAN, COMMIT TRAN, or ROLLBACK TRAN. Explicit transactions are typically used when multiple modification statements need to be tied together in a single atomic unit.
21
21
22
-
SQL Analytics commits changes to the database using transaction logs. Each distribution has its own transaction log. Transaction log writes are automatic. There is no configuration required. However, whilst this process guarantees the write it does introduce an overhead in the system. You can minimize this impact by writing transactionally efficient code. Transactionally efficient code broadly falls into two categories.
22
+
Transactions are an important component of a relational database engine. Transactions are used during data modification. These transactions can be explicit or implicit. Single INSERT, UPDATE, and DELETE statements are all examples of implicit transactions. Explicit transactions use BEGIN TRAN, COMMIT TRAN, or ROLLBACK TRAN. Explicit transactions are typically used when multiple modification statements need to be tied together in a single atomic unit.
23
+
24
+
Changes to the database are tracked using transaction logs. Each distribution has its own transaction log. Transaction log writes are automatic. There is no configuration required. However, whilst this process guarantees the write it does introduce an overhead in the system. You can minimize this impact by writing transactionally efficient code. Transactionally efficient code broadly falls into two categories.
23
25
24
26
* Use minimal logging constructs whenever possible
25
27
* Process data using scoped batches to avoid singular long running transactions
26
28
* Adopt a partition switching pattern for large modifications to a given partition
27
29
28
30
## Minimal vs. full logging
31
+
29
32
Unlike fully logged operations, which use the transaction log to keep track of every row change, minimally logged operations keep track of extent allocations and meta-data changes only. Therefore, minimal logging involves logging only the information that is required to roll back the transaction after a failure, or for an explicit request (ROLLBACK TRAN). As much less information is tracked in the transaction log, a minimally logged operation performs better than a similarly sized fully logged operation. Furthermore, because fewer writes go the transaction log, a much smaller amount of log data is generated and so is more I/O efficient.
30
33
31
34
The transaction safety limits only apply to fully logged operations.
@@ -36,6 +39,7 @@ The transaction safety limits only apply to fully logged operations.
36
39
>
37
40
38
41
## Minimally logged operations
42
+
39
43
The following operations are capable of being minimally logged:
40
44
41
45
* CREATE TABLE AS SELECT ([CTAS](sql-data-warehouse-develop-ctas.md))
@@ -73,14 +77,13 @@ CTAS and INSERT...SELECT are both bulk load operations. However, both are influe
73
77
It is worth noting that any writes to update secondary or non-clustered indexes will always be fully logged operations.
74
78
75
79
> [!IMPORTANT]
76
-
> A SQL Analytics database has 60 distributions. Therefore, assuming all rows are evenly distributed and landing in a single partition, your batch will need to contain 6,144,000 rows or larger to be minimally logged when writing to a Clustered Columnstore Index. If the table is partitioned and the rows being inserted span partition boundaries, then you will need 6,144,000 rows per partition boundary assuming even data distribution. Each partition in each distribution must independently exceed the 102,400 row threshold for the insert to be minimally logged into the distribution.
77
-
>
80
+
> A Synapse SQL pool database has 60 distributions. Therefore, assuming all rows are evenly distributed and landing in a single partition, your batch will need to contain 6,144,000 rows or larger to be minimally logged when writing to a Clustered Columnstore Index. If the table is partitioned and the rows being inserted span partition boundaries, then you will need 6,144,000 rows per partition boundary assuming even data distribution. Each partition in each distribution must independently exceed the 102,400 row threshold for the insert to be minimally logged into the distribution.
78
81
>
79
82
80
83
Loading data into a non-empty table with a clustered index can often contain a mixture of fully logged and minimally logged rows. A clustered index is a balanced tree (b-tree) of pages. If the page being written to already contains rows from another transaction, then these writes will be fully logged. However, if the page is empty then the write to that page will be minimally logged.
81
84
82
85
## Optimizing deletes
83
-
DELETE is a fully logged operation. If you need to delete a large amount of data in a table or a partition, it often makes more sense to `SELECT` the data you wish to keep, which can be run as a minimally logged operation. To select the data, create a new table with [CTAS](sql-data-warehouse-develop-ctas.md). Once created, use [RENAME](/sql/t-sql/statements/rename-transact-sql) to swap out your old table with the newly created table.
86
+
DELETE is a fully logged operation. If you need to delete a large amount of data in a table or a partition, it often makes more sense to `SELECT` the data you wish to keep, which can be run as a minimally logged operation. To select the data, create a new table with [CTAS](sql-data-warehouse-develop-ctas.md). Once created, use [RENAME](/sql/t-sql/statements/rename-transact-sql?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json&view=azure-sqldw-latest) to swap out your old table with the newly created table.
84
87
85
88
```sql
86
89
-- Delete all sales transactions for Promotions except PromotionKey 2.
@@ -111,7 +114,7 @@ RENAME OBJECT [dbo].[FactInternetSales_d] TO [FactInternetSales];
111
114
```
112
115
113
116
## Optimizing updates
114
-
UPDATE is a fully logged operation. If you need to update a large number of rows in a table or a partition, it can often be far more efficient to use a minimally logged operation such as [CTAS](/sql/t-sql/statements/create-table-as-select-azure-sql-data-warehouse) to do so.
117
+
UPDATE is a fully logged operation. If you need to update a large number of rows in a table or a partition, it can often be far more efficient to use a minimally logged operation such as [CTAS](/sql/t-sql/statements/create-table-as-select-azure-sql-data-warehouse?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json&view=azure-sqldw-latest) to do so.
115
118
116
119
In the example below a full table update has been converted to a CTAS so that minimal logging is possible.
117
120
@@ -172,7 +175,7 @@ DROP TABLE [dbo].[FactInternetSales_old]
172
175
```
173
176
174
177
> [!NOTE]
175
-
> Re-creating large tables can benefit from using SQL Analytics workload management features. For more information, see [Resource classes for workload management](resource-classes-for-workload-management.md).
178
+
> Re-creating large tables can benefit from using Synapse SQL pool workload management features. For more information, see [Resource classes for workload management](resource-classes-for-workload-management.md).
176
179
>
177
180
>
178
181
@@ -400,7 +403,8 @@ END
400
403
```
401
404
402
405
## Pause and scaling guidance
403
-
SQL Analytics lets you [pause, resume, and scale](sql-data-warehouse-manage-compute-overview.md) your SQL pool on demand. When you pause or scale your SQL pool, it is important to understand that any in-flight transactions are terminated immediately; causing any open transactions to be rolled back. If your workload had issued a long running and incomplete data modification prior to the pause or scale operation, then this work will need to be undone. This undoing might impact the time it takes to pause or scale your SQL pool.
406
+
407
+
Synapse SQL lets you [pause, resume, and scale](sql-data-warehouse-manage-compute-overview.md) your SQL pool on demand. When you pause or scale your SQL pool, it is important to understand that any in-flight transactions are terminated immediately; causing any open transactions to be rolled back. If your workload had issued a long running and incomplete data modification prior to the pause or scale operation, then this work will need to be undone. This undoing might impact the time it takes to pause or scale your SQL pool.
404
408
405
409
> [!IMPORTANT]
406
410
> Both `UPDATE` and `DELETE` are fully logged operations and so these undo/redo operations can take significantly longer than equivalent minimally logged operations.
@@ -409,9 +413,10 @@ SQL Analytics lets you [pause, resume, and scale](sql-data-warehouse-manage-comp
409
413
410
414
The best scenario is to let in flight data modification transactions complete prior to pausing or scaling SQL pool. However, this scenario might not always be practical. To mitigate the risk of a long rollback, consider one of the following options:
411
415
412
-
* Rewrite long running operations using [CTAS](/sql/t-sql/statements/create-table-as-select-azure-sql-data-warehouse)
416
+
* Rewrite long running operations using [CTAS](/sql/t-sql/statements/create-table-as-select-azure-sql-data-warehouse?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json&view=azure-sqldw-latest)
413
417
* Break the operation into chunks; operating on a subset of the rows
414
418
415
419
## Next steps
416
-
See [Transactions in SQL Analytics](sql-data-warehouse-develop-transactions.md) to learn more about isolation levels and transactional limits. For an overview of other Best Practices, see [SQL Data Warehouse Best Practices](sql-data-warehouse-best-practices.md).
420
+
421
+
See [Transactions in Synapse SQL](sql-data-warehouse-develop-transactions.md) to learn more about isolation levels and transactional limits. For an overview of other Best Practices, see [SQL Data Warehouse Best Practices](sql-data-warehouse-best-practices.md).
0 commit comments