Skip to content

Commit c1cca1d

Browse files
WilliamDAssafMSFTWilliamDAssafMSFT
committed
20211028 1029
1 parent 58efb46 commit c1cca1d

File tree

2 files changed

+48
-61
lines changed

2 files changed

+48
-61
lines changed

‎.vscode/settings.json

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"markdownlint.config": {
3+
"MD028": false,
4+
"MD025": {
5+
"front_matter_title": ""
6+
}
7+
}
8+
}

‎articles/synapse-analytics/sql/query-history-storage-analysis.md

Lines changed: 40 additions & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -1,91 +1,62 @@
11
---
2-
title: Historical Query Storage and Analysis in Azure Synapse Analytics
3-
description: Historic query analysis is one of the crucial needs of data engineers. Azure Synapse Analytics provides four main ways to analyze query history and performance. These include DMVs, Azure Data Explorer, Azure Log Analytics and Query Store.
4-
This article will show you how to use each of these options for your needs.
5-
2+
title: Historical query storage and analysis in Azure Synapse Analytics
3+
description: Historic query analysis is one of the crucial needs of data engineers. Azure Synapse Analytics provides four main ways to analyze query history and performance. These include Query Store, DMVs, Azure Log Analytics, and Azure Data Explorer.
64
author: mariyaali
75
ms.author: mariyaali
6+
ms.reviewer: wiassaf
87
ms.service: synapse-analytics
98
ms.topic: conceptual
10-
ms.date: 10/12/2021
9+
ms.date: 10/28/2021
1110
ms.custom: template-concept
1211
---
1312

14-
# Historical Query Storage and Analysis in Azure Synapse Analytics
13+
# Historical query storage and analysis in Azure Synapse Analytics
1514

16-
Historic query analysis is one of the crucial needs of data engineers. Azure Synapse Analytics provides four main ways to analyze query history and performance. These include DMVs, Azure Data Explorer, Azure Log Analytics and Query Store.
17-
This article will show you how to use each of these options for your needs.
15+
Historic query analysis is one of the crucial needs of data engineers. Azure Synapse Analytics provides four main ways to analyze query history and performance. These include Query Store, DMVs, Azure Log Analytics, and Azure Data Explorer.
1816

19-
## Introduction
20-
Historic query analysis is one of the crucial needs of data engineers. Azure Synapse Analytics provides four main ways to analyze query history and performance. These include DMVs, Azure Data Explorer, Azure Log Analytics and Query Store.
17+
This article will show you how to use each of these options for your needs. Review use cases when it comes to analyzing query history, and the best method for each.
2118

22-
This article will show you how to use each of these options for your needs.
19+
| Customer need | Query Store | DMVs | Log Analytics | Azure Data Explorer |
20+
|------------- | --- | ----- | ------------- |-------------------|
21+
|Out of the box solution | Needs enabling | :heavy_check_mark: | Addition service required | Addition service required|
22+
|Longer analysis periods | 30 days | Up to 10000 rows of history | Customizable | Customizable|
23+
|Crucial metrics availability | Limited | :heavy_check_mark: | Limited | Customizable|
24+
|Use SQL for analysis | :heavy_check_mark: | :heavy_check_mark:| KQL needed | SQL support is limited|
25+
|||||
2326

24-
## Customer needs
25-
This section highlights some use cases when it comes to analyzing query history, and the best method for each.
27+
## Query Store
2628

27-
Customer Need | QS | DMV | Log Analytics | Data Explorer
28-
------------- | --- | ----- | ------------- |-------------------
29-
Out of the box solution | Needs enabling | :heavy_check_mark: | Addition service required | Addition service required
30-
Longer analysis periods | 30 days | Up to 10000 rows of history | Customizable | Customizable
31-
Crucial metrics availability | Limited | :heavy_check_mark: | Limited | Customizable
32-
Use SQL for analysis | :heavy_check_mark: | :heavy_check_mark:| KQL needed | SQL support is limited
29+
The Query Store feature provides insight on query plan choice and performance. It simplifies performance troubleshooting by helping you quickly find performance differences caused by query plan changes.
3330

34-
## Ways to store and analyze query data
35-
36-
### Query Store
37-
38-
Query Store feature provides insight on query plan choice and performance. It simplifies performance troubleshooting by helping you quickly find performance differences caused by query plan changes. Query Store can hold data for until 30 days in.
39-
40-
Query Store is not enabled by default for new Azure Synapse Analytics databases.
41-
42-
To enable Query Store to run the following T-SQL command:
31+
Query Store is not enabled by default for new Azure Synapse Analytics databases. To enable Query Store to run the following T-SQL command:
4332

4433
```sql
4534
ALTER DATABASE <database_name>
4635
SET QUERY_STORE = ON;
4736
```
4837

49-
You can run performance auditing and troubleshooting related tasks by finding last executed queries, execution counts, longest running queries, queries with maximum physical I/O leads. Please refer to [Monitoring Performance By Using the Query Store](/sql/relational-databases/performance/monitoring-performance-by-using-the-query-store.md) for sample queries.
38+
You can run performance auditing and troubleshooting related tasks by finding last executed queries, execution counts, longest running queries, queries with maximum physical I/O leads. Please refer to [Monitoring Performance By Using the Query Store](/sql/relational-databases/performance/monitoring-performance-by-using-the-query-store#performance) for sample queries.
5039

5140
Advantages:
5241
* 30 days of threshold for storage query data.
53-
* Data can be consumed in the same tool that you’d run the query in.
42+
* Data can be consumed in the same tool that you'd run the query in.
5443

5544
Disadvantages:
5645
* Scenarios for analysis are limited in Query Store for Azure Synapse when compared to using DMVs.
5746

58-
### DMV
47+
## DMVs
48+
5949
Dynamic Management Views (DMVs) are extremely useful when it comes to gathering information on query wait times, execution plans, memory, etc.
6050
It is highly recommended to label your query of interest to track it down later.
6151

6252
```sql
6353
-- Query with Label
6454
SELECT *
6555
FROM sys.tables
66-
OPTION (LABEL = 'My Query')
67-
;
56+
OPTION (LABEL = 'My Query');
6857
```
69-
Find more about how to use DMVs here: Monitor your dedicated SQL pool workload using DMVs.
70-
Documentation on supported views is available here:
71-
* [sys.dm_pdw_dms_cores](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-dms-cores-transact-sql.md)
72-
* [sys.dm_pdw_dms_external_work](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-dms-external_work-transact-sql.md)
73-
* [sys.dm_pdw_dms_workers](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-dms-workers-transact-sql.md)
74-
* [sys.dm_pdw_errors](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-dms-errors-transact-sql.md)
75-
* [sys.dm_pdw_exec_connections](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-dms-exec-connections-transact-sql.md)
76-
* [sys.dm_pdw_exec_requests](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-dms-exec-requests-transact-sql.md)
77-
* [sys.dm_pdw_exec_sessions](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-dms-exec-sessions-transact-sql.md)
78-
* [sys.dm_pdw_hadoop_operations](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-dms-hadoop-operations-transact-sql.md)
79-
* [sys.dm_pdw_lock_waits](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-dms-lock-waits-transact-sql.md)
80-
* [sys.dm_pdw_nodes](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-dms-nodes-transact-sql.md)
81-
* [sys.dm_pdw_nodes_database_encryption_keys](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-dms-nodes-database-encryption-keys-transact-sql.md)
82-
* [sys.dm_pdw_os_threads](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-dms-os-threads-transact-sql.md)
83-
* [sys.dm_pdw_request_steps](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-dms-request-steps-transact-sql.md)
84-
* [sys.dm_pdw_resource_waits](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-dms-resource-waits-transact-sql.md)
85-
* [sys.dm_pdw_sql_requests](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-dms-sql-requests-transact-sql.md)
86-
* [sys.dm_pdw_sys_info](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-dms-sys-info-transact-sql.md)
87-
* [sys.dm_pdw_wait_stats](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-dms-wait-stats-transact-sql.md)
88-
* [sys.dm_pdw_waits](/sql/relational-databases/system-dynamic-management-views/sys-dm-pdw-dms-waits-transact-sql.md)
58+
59+
For more information on using DMVs to monitor your Azure Synapse Analytics workload, see [Monitor your dedicated SQL pool workload using DMVs](../sql-data-warehouse/sql-data-warehouse-manage-monitor.md?context=/azure/synapse-analytics/context/context). For documentation on catalog views specific to Azure Synapse Analytics, see [Azure Synapse Analytics Catalog Views](/sql/relational-databases/system-catalog-views/sql-data-warehouse-and-parallel-data-warehouse-catalog-views).
8960

9061
Advantages:
9162
* Data can be consumed in the same querying tool.
@@ -95,13 +66,12 @@ Disadvantages:
9566
* DMVs are limited to 10,000 rows of historic entries.
9667
* Views are reset when pool is paused/resumed.
9768

98-
### Log Analytics
99-
Log Analytics workspaces can be created easily in the Azure portal. For further instructions on how to connect Synapse with Log Analytics, see [Monitor workload - Azure portal](sql-data-warehouse-monitor-workload-portal.md).
69+
## Log Analytics
70+
Log Analytics workspaces can be created easily in the Azure portal. For further instructions on how to connect Synapse with Log Analytics, see [Monitor workload - Azure portal](../sql-data-warehouse/sql-data-warehouse-monitor-workload-portal.md).
10071

101-
Like Azure Data Explorer, Log Analytics uses the Kusto Query Language (KQL). For more information about Kusto syntax, see Kusto query overview.
72+
Like Azure Data Explorer, Log Analytics uses the Kusto Query Language (KQL). For more information about Kusto syntax, see [Kusto query overview](/data-explorer/kusto/query/index.md).
10273

103-
Along with configurable retention period, you choose the workspace you are specifically targeting to query in Log Analytics.
104-
Log Analytics give you the flexibility to store data, run, and save queries.
74+
Along with configurable retention period, you choose the workspace you are specifically targeting to query in Log Analytics. Log Analytics gives you the flexibility to store data, run, and save queries.
10575

10676
Advantages:
10777
* Azure Log Analytics has a customizable log retention policy
@@ -110,12 +80,21 @@ Disadvantages:
11080
* Using KQL adds to the learning curve.
11181
* Limited views can be logged out of the box.
11282

113-
### Data Explorer
114-
Azure Data Explorer (ADX) is a leading data exploration service. This service can be used to analyze historic queries from Azure Synapse Analytics. To setup an Azure Data Factory (ADF) pipeline to copy and store logs to ADX, see [Copy data to or from Azure Data Explorer](/data-factory/connector-azure-data-explorer.md). In ADX, you can run performant Kusto query to analyze your logs. You can combine other strategies here, for example to query and load DMV output to ADX via ADF.
83+
## Azure Data Explorer (ADX)
84+
85+
Azure Data Explorer (ADX) is a leading data exploration service. This service can be used to analyze historic queries from Azure Synapse Analytics. To setup an Azure Data Factory (ADF) pipeline to copy and store logs to ADX, see [Copy data to or from Azure Data Explorer](/data-factory/connector-azure-data-explorer.md). In ADX, you can run performant Kusto query to analyze your logs. You can combine other strategies here, for example to query and load DMV output to ADX via ADF.
11586

11687
Advantages:
11788
* ADX provides a customizable log retention policy.
11889
* Performant query execution against large amount of data, especially queries involving string search.
11990

12091
Disadvantage:
121-
* Using KQL adds to the learning curve.
92+
* Using KQL adds to the learning curve.
93+
94+
## Next steps
95+
96+
- [Azure Data Explorer](/azure/data-explorer/)
97+
- [Azure Data Factory](/azure/data-factory/)
98+
- [Log Analytics in Azure Monitor](/azure/azure-monitor/logs/log-analytics-overview)
99+
- [Azure Synapse SQL architecture](overview-architecture.md)
100+
- [Get Started with Azure Synapse Analytics](../get-started.md)

0 commit comments

Comments
 (0)