Skip to content

Commit 99db1fa

Browse files
Merge pull request #293410 from whhender/january-freshness-2025
January freshness 2025 - part 2
2 parents 6938924 + 7e64c48 commit 99db1fa

6 files changed

+82
-82
lines changed

articles/synapse-analytics/sql-data-warehouse/sql-data-warehouse-develop-ctas.md

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,10 @@ description: Explanation and examples of the CREATE TABLE AS SELECT (CTAS) state
44
author: joannapea
55
ms.author: joanpo
66
ms.reviewer: wiassaf
7-
ms.date: 06/09/2022
7+
ms.date: 01/21/2025
88
ms.service: azure-synapse-analytics
99
ms.subservice: sql-dw
10-
ms.topic: conceptual
10+
ms.topic: concept-article
1111
ms.custom:
1212
- azure-synapse
1313
---
@@ -90,7 +90,7 @@ WITH(
9090
);
9191
```
9292

93-
Now you want to create a new copy of this table, with a `Clustered Columnstore Index`, so you can take advantage of the performance of Clustered Columnstore tables. You also want to distribute this table on `ProductKey`, because you're anticipating joins on this column and want to avoid data movement during joins on `ProductKey`. Lastly, you also want to add partitioning on `OrderDateKey`, so you can quickly delete old data by dropping old partitions. Here is the CTAS statement, which copies your old table into a new table.
93+
Now you want to create a new copy of this table, with a `Clustered Columnstore Index`, so you can take advantage of the performance of Clustered Columnstore tables. You also want to distribute this table on `ProductKey`, because you're anticipating joins on this column and want to avoid data movement during joins on `ProductKey`. Lastly, you also want to add partitioning on `OrderDateKey`, so you can quickly delete old data by dropping old partitions. Here's the CTAS statement, which copies your old table into a new table.
9494

9595
```sql
9696
CREATE TABLE FactInternetSales_new
@@ -169,9 +169,9 @@ The value stored for result is different. As the persisted value in the result c
169169

170170
This is important for data migrations. Even though the second query is arguably more accurate, there's a problem. The data would be different compared to the source system, and that leads to questions of integrity in the migration. This is one of those rare cases where the "wrong" answer is actually the right one!
171171

172-
The reason we see a disparity between the two results is due to implicit type casting. In the first example, the table defines the column definition. When the row is inserted, an implicit type conversion occurs. In the second example, there is no implicit type conversion as the expression defines the data type of the column.
172+
The reason we see a disparity between the two results is due to implicit type casting. In the first example, the table defines the column definition. When the row is inserted, an implicit type conversion occurs. In the second example, there's no implicit type conversion as the expression defines the data type of the column.
173173

174-
Notice also that the column in the second example has been defined as a NULLable column, whereas in the first example it has not. When the table was created in the first example, column nullability was explicitly defined. In the second example, it was left to the expression, and by default would result in a NULL definition.
174+
Notice also that the column in the second example has been defined as a NULLable column, whereas in the first example it hasn't. When the table was created in the first example, column nullability was explicitly defined. In the second example, it was left to the expression, and by default would result in a NULL definition.
175175

176176
To resolve these issues, you must explicitly set the type conversion and nullability in the SELECT portion of the CTAS statement. You can't set these properties in 'CREATE TABLE'.
177177
The following example demonstrates how to fix the code:
@@ -194,7 +194,7 @@ Note the following:
194194
* The second part of the ISNULL is a constant, 0.
195195

196196
> [!NOTE]
197-
> For the nullability to be correctly set, it's vital to use ISNULL and not COALESCE. COALESCE is not a deterministic function, and so the result of the expression will always be NULLable. ISNULL is different. It's deterministic. Therefore, when the second part of the ISNULL function is a constant or a literal, the resulting value will be NOT NULL.
197+
> For the nullability to be correctly set, it's vital to use ISNULL and not COALESCE. COALESCE isn't a deterministic function, and so the result of the expression will always be NULLable. ISNULL is different. It's deterministic. Therefore, when the second part of the ISNULL function is a constant or a literal, the resulting value will be NOT NULL.
198198
199199
Ensuring the integrity of your calculations is also important for table partition switching. Imagine you have this table defined as a fact table:
200200

@@ -270,6 +270,6 @@ You can see that type consistency and maintaining nullability properties on a CT
270270

271271
CTAS is one of the most important statements in Synapse SQL. Make sure you thoroughly understand it. See the [CTAS documentation](/sql/t-sql/statements/create-table-as-select-azure-sql-data-warehouse?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json&view=azure-sqldw-latest&preserve-view=true).
272272

273-
## Next steps
273+
## Related content
274274

275275
For more development tips, see the [development overview](sql-data-warehouse-overview-develop.md).

articles/synapse-analytics/sql-data-warehouse/sql-data-warehouse-manage-monitor.md

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,10 @@ description: Learn how to monitor your Azure Synapse Analytics dedicated SQL poo
44
author: WilliamDAssafMSFT
55
ms.author: wiassaf
66
ms.reviewer: kecona
7-
ms.date: 11/09/2022
7+
ms.date: 01/22/2025
88
ms.service: azure-synapse-analytics
99
ms.subservice: sql-dw
10-
ms.topic: conceptual
10+
ms.topic: concept-article
1111
ms.custom: synapse-analytics
1212
---
1313

@@ -25,7 +25,7 @@ GRANT VIEW DATABASE STATE TO myuser;
2525

2626
## Monitor connections
2727

28-
All logins to your data warehouse are logged to [sys.dm_pdw_exec_sessions](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-exec-sessions-transact-sql?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json&view=azure-sqldw-latest&preserve-view=true). This DMV contains the last 10,000 logins. The `session_id` is the primary key and is assigned sequentially for each new login.
28+
All logins to your data warehouse are logged to [sys.dm_pdw_exec_sessions](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-exec-sessions-transact-sql?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json&view=azure-sqldw-latest&preserve-view=true). This DMV contains the last 10,000 logins. The `session_id` is the primary key and is assigned sequentially for each new login.
2929

3030
```sql
3131
-- Other Active Connections
@@ -34,10 +34,10 @@ SELECT * FROM sys.dm_pdw_exec_sessions where status <> 'Closed' and session_id <
3434

3535
## Monitor query execution
3636

37-
All queries executed on SQL pool are logged to [sys.dm_pdw_exec_requests](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-exec-requests-transact-sql?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json&view=azure-sqldw-latest&preserve-view=true). This DMV contains the last 10,000 queries executed. The `request_id` uniquely identifies each query and is the primary key for this DMV. The `request_id` is assigned sequentially for each new query and is prefixed with QID, which stands for query ID. Querying this DMV for a given `session_id` shows all queries for a given login.
37+
All queries executed on SQL pool are logged to [sys.dm_pdw_exec_requests](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-exec-requests-transact-sql?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json&view=azure-sqldw-latest&preserve-view=true). This DMV contains the last 10,000 queries executed. The `request_id` uniquely identifies each query and is the primary key for this DMV. The `request_id` is assigned sequentially for each new query and is prefixed with QID, which stands for query ID. Querying this DMV for a given `session_id` shows all queries for a given login.
3838

3939
> [!NOTE]
40-
> Stored procedures use multiple Request IDs. Request IDs are assigned in sequential order.
40+
> Stored procedures use multiple Request IDs. Request IDs are assigned in sequential order.
4141
4242
Here are steps to follow to investigate query execution plans and times for a particular query.
4343

@@ -59,7 +59,7 @@ ORDER BY total_elapsed_time DESC;
5959

6060
From the preceding query results, **note the Request ID** of the query that you would like to investigate.
6161

62-
Queries in the **Suspended** state can be queued due to a large number of active running queries. These queries also appear in the [sys.dm_pdw_waits](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-waits-transact-sql?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json&view=azure-sqldw-latest&preserve-view=true). In that case, look for waits such as UserConcurrencyResourceType. For information on concurrency limits, see [Memory and concurrency limits](memory-concurrency-limits.md) or [Resource classes for workload management](resource-classes-for-workload-management.md). Queries can also wait for other reasons such as for object locks. If your query is waiting for a resource, see [Investigating queries waiting for resources](#monitor-waiting-queries) further down in this article.
62+
Queries in the **Suspended** state can be queued due to a large number of active running queries. These queries also appear in the [sys.dm_pdw_waits](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-waits-transact-sql?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json&view=azure-sqldw-latest&preserve-view=true). In that case, look for waits such as UserConcurrencyResourceType. For information on concurrency limits, see [Memory and concurrency limits](memory-concurrency-limits.md) or [Resource classes for workload management](resource-classes-for-workload-management.md). Queries can also wait for other reasons such as for object locks. If your query is waiting for a resource, see [Investigating queries waiting for resources](#monitor-waiting-queries) further down in this article.
6363

6464
To simplify the lookup of a query in the [sys.dm_pdw_exec_requests](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-exec-requests-transact-sql?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json&view=azure-sqldw-latest&preserve-view=true) table, use [LABEL](/sql/t-sql/queries/option-clause-transact-sql?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json&view=azure-sqldw-latest&preserve-view=true) to assign a comment to your query, which can be looked up in the `sys.dm_pdw_exec_requests` view.
6565

@@ -90,7 +90,7 @@ WHERE request_id = 'QID####'
9090
ORDER BY step_index;
9191
```
9292

93-
When a DSQL plan is taking longer than expected, the cause can be a complex plan with many DSQL steps or just one step taking a long time. If the plan is many steps with several move operations, consider optimizing your table distributions to reduce data movement. The [Table distribution](sql-data-warehouse-tables-distribute.md) article explains why data must be moved to solve a query. The article also explains some distribution strategies to minimize data movement.
93+
When a DSQL plan is taking longer than expected, the cause can be a complex plan with many DSQL steps or just one step taking a long time. If the plan is many steps with several move operations, consider optimizing your table distributions to reduce data movement. The [Table distribution](sql-data-warehouse-tables-distribute.md) article explains why data must be moved to solve a query. The article also explains some distribution strategies to minimize data movement.
9494

9595
To investigate further details about a single step, inspect the `operation_type` column of the long-running query step and note the **Step Index**:
9696

@@ -146,7 +146,7 @@ DBCC PDW_SHOWEXECUTIONPLAN(55, 238);
146146

147147
## Monitor waiting queries
148148

149-
If you discover that your query is not making progress because it is waiting for a resource, here is a query that shows all the resources a query is waiting for.
149+
If you discover that your query isn't making progress because it's waiting for a resource, here's a query that shows all the resources a query is waiting for.
150150

151151
```sql
152152
-- Find queries
@@ -168,11 +168,11 @@ WHERE waits.request_id = 'QID####'
168168
ORDER BY waits.object_name, waits.object_type, waits.state;
169169
```
170170

171-
If the query is actively waiting on resources from another query, then the state will be **AcquireResources**. If the query has all the required resources, then the state will be **Granted**.
171+
If the query is actively waiting on resources from another query, then the state will be **AcquireResources**. If the query has all the required resources, then the state will be **Granted**.
172172

173173
## Monitor tempdb
174174

175-
The `tempdb` database is used to hold intermediate results during query execution. High utilization of the `tempdb` database can lead to slow query performance. For every DW100c configured, 399 GB of `tempdb` space is allocated (DW1000c would have 3.99 TB of total `tempdb` space). Below are tips for monitoring `tempdb` usage and for decreasing `tempdb` usage in your queries.
175+
The `tempdb` database is used to hold intermediate results during query execution. High utilization of the `tempdb` database can lead to slow query performance. For every DW100c configured, 399 GB of `tempdb` space is allocated (DW1000c would have 3.99 TB of total `tempdb` space). Below are tips for monitoring `tempdb` usage and for decreasing `tempdb` usage in your queries.
176176

177177
### Monitor tempdb with views
178178

@@ -215,11 +215,11 @@ ORDER BY sr.request_id;
215215
> Use [Azure Synapse SQL Distribution Advisor](../sql/distribution-advisor.md) to get recommendations on the distribution method suited for your workloads.
216216
> Use the [Azure Synapse Toolkit](https://github.com/microsoft/Azure_Synapse_Toolbox/tree/master/TSQL_Queries/TempDB) to monitor `tempdb` using T-SQL queries.
217217
218-
If you have a query that is consuming a large amount of memory or have received an error message related to the allocation of `tempdb`, it could be due to a very large [CREATE TABLE AS SELECT (CTAS)](/sql/t-sql/statements/create-table-as-select-azure-sql-data-warehouse) or [INSERT SELECT](/sql/t-sql/statements/insert-transact-sql?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json&view=azure-sqldw-latest&preserve-view=true) statement running that is failing in the final data movement operation. This can usually be identified as a ShuffleMove operation in the distributed query plan right before the final INSERT SELECT. Use [sys.dm_pdw_request_steps](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-request-steps-transact-sql?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json&view=azure-sqldw-latest&preserve-view=true) to monitor ShuffleMove operations.
218+
If you have a query that is consuming a large amount of memory or have received an error message related to the allocation of `tempdb`, it could be due to a very large [CREATE TABLE AS SELECT (CTAS)](/sql/t-sql/statements/create-table-as-select-azure-sql-data-warehouse) or [INSERT SELECT](/sql/t-sql/statements/insert-transact-sql?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json&view=azure-sqldw-latest&preserve-view=true) statement running that is failing in the final data movement operation. This can usually be identified as a ShuffleMove operation in the distributed query plan right before the final INSERT SELECT. Use [sys.dm_pdw_request_steps](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-request-steps-transact-sql?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json&view=azure-sqldw-latest&preserve-view=true) to monitor ShuffleMove operations.
219219

220-
The most common mitigation is to break your CTAS or INSERT SELECT statement into multiple load statements so that the data volume will not exceed the 399 GB per 100DWUc `tempdb` limit. You can also scale your cluster to a larger size to increase how much `tempdb` space you have.
220+
The most common mitigation is to break your CTAS or INSERT SELECT statement into multiple load statements so that the data volume won't exceed the 399 GB per 100DWUc `tempdb` limit. You can also scale your cluster to a larger size to increase how much `tempdb` space you have.
221221

222-
In addition to CTAS and INSERT SELECT statements, large, complex queries running with insufficient memory can spill into `tempdb` causing queries to fail. Consider running with a larger [resource class](resource-classes-for-workload-management.md) to avoid spilling into `tempdb`.
222+
In addition to CTAS and INSERT SELECT statements, large, complex queries running with insufficient memory can spill into `tempdb` causing queries to fail. Consider running with a larger [resource class](resource-classes-for-workload-management.md) to avoid spilling into `tempdb`.
223223

224224
## Monitor memory
225225

@@ -363,6 +363,6 @@ WHERE waiting.state = 'Queued'
363363
ORDER BY Lock_Request_Time DESC;
364364
```
365365

366-
## Next steps
366+
## Related content
367367

368368
- For more information about DMVs, see [System views](../sql/reference-tsql-system-views.md).

0 commit comments

Comments
 (0)