Skip to content

Commit 134400a

Browse files
Court72Court72
authored
Merge pull request #177881 from WilliamDAssafMSFT/20211027-new-memory-701-asdb
20211027 OOM for ASDB
2 parents 4791541 + 08f6dfa commit 134400a

File tree

2 files changed

+174
-0
lines changed

2 files changed

+174
-0
lines changed

‎articles/azure-sql/database/troubleshoot-memory-errors-issues.md

Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,172 @@
1+
---
2+
title: Troubleshoot memory issues
3+
titleSuffix: Azure SQL Database
4+
description: Provides steps to troubleshoot out of memory issues in Azure SQL Database
5+
services: sql-database
6+
ms.service: sql-database
7+
ms.subservice: development
8+
ms.topic: troubleshooting
9+
ms.custom:
10+
author: WilliamDAssafMSFT
11+
ms.author: wiassaf
12+
ms.reviewer:
13+
ms.date: 11/03/2021
14+
---
15+
16+
# Troubleshoot out of memory errors with Azure SQL Database
17+
[!INCLUDE[appliesto-sqldb](../includes/appliesto-sqldb.md)]
18+
19+
You may see error messages when the SQL database engine has failed to allocate sufficient memory to run the query. This can be caused by various reasons including the limits of selected service objective, aggregate workload memory demands, and memory demands by the query. For more information on the memory resource limit for Azure SQL Databases, see [Resource management in Azure SQL Database](resource-limits-logical-server.md#memory).
20+
21+
> [!NOTE]
22+
> **This article is focused on Azure SQL Database.** For more on troubleshooting out of memory issues in SQL Server, see [MSSQLSERVER_701](/sql/relational-databases/errors-events/mssqlserver-701-database-engine-error).
23+
24+
Try the following avenues of investigation in response to:
25+
26+
- Error code 701 with error message "There is insufficient system memory in resource pool '%ls' to run this query."
27+
- Error code 802 with error message "There is insufficient memory available in the buffer pool."
28+
29+
## Investigate memory allocation
30+
31+
If out of memory errors persist in Azure SQL Database, consider at least temporarily increasing the service level objective of the database in the Azure portal. If out of memory errors persist, use the following queries to look for unusually high query memory grants that may contribute to an insufficient memory condition. Run the following example queries in the database that experienced the error (not in the `master` database of the Azure SQL logical server).
32+
33+
### Use DMVs to view memory clerks
34+
35+
Start with a broad investigation, if the out of memory error occurred recently, by viewing the allocation of memory to memory clerks. Memory clerks are internal to the database engine for this Azure SQL Database. The top memory clerks in terms of pages allocated might be informative to what type of query or feature of SQL Server is consuming the most memory.
36+
37+
38+
```sql
39+
SELECT [type], [name], pages_kb, virtual_memory_committed_kb
40+
FROM sys.dm_os_memory_clerks
41+
WHERE memory_node_id <> 64 -- ignore Dedicated Admin Connection (DAC) node
42+
ORDER BY pages_kb DESC;
43+
GO
44+
SELECT [type], [name], pages_kb, virtual_memory_committed_kb
45+
FROM sys.dm_os_memory_clerks
46+
WHERE memory_node_id <> 64 -- ignore Dedicated Admin Connection (DAC) node
47+
ORDER BY virtual_memory_committed_kb DESC;
48+
```
49+
50+
- Some common memory clerks, such as MEMORYCLERK_SQLQERESERVATIONS, are best resolved by identifying queries with large memory grants and improving their performance with better indexing and index tuning.
51+
- While OBJECTSTORE_LOCK_MANAGER is unrelated to memory grants, it is expected to be high when queries claim many locks, for example, because of disabled lock escalation or very large transactions.
52+
- Some clerks are expected to be the highest utilization: MEMORYCLERK_SQLBUFFERPOOL is almost always the top clerk, while CACHESTORE_COLUMNSTOREOBJECTPOOL will be high when columnstore indexes are used. Highest utilization by these clerks is expected.
53+
54+
For more information about memory clerk types, see [sys.dm_os_memory_clerks](/sql/relational-databases/system-dynamic-management-views/sys-dm-os-memory-clerks-transact-sql).
55+
56+
### Use DMVs to investigate active queries
57+
58+
In most cases, the query that failed is not the cause of this error.
59+
60+
The following sample query for Azure SQL Database returns important information on transactions that are currently holding or waiting for memory grants. Target the top queries identified for examination and performance tuning, and evaluate whether or not they are executing as intended. Consider the timing of memory-intensive reporting queries or maintenance operations.
61+
62+
```sql
63+
--Active requests with memory grants
64+
SELECT
65+
--Session data
66+
s.[session_id], s.open_transaction_count
67+
--Memory usage
68+
, r.granted_query_memory, mg.grant_time, mg.requested_memory_kb, mg.granted_memory_kb, mg.required_memory_kb, mg.used_memory_kb, mg.max_used_memory_kb
69+
--Query
70+
, query_text = t.text, input_buffer = ib.event_info, query_plan_xml = qp.query_plan, request_row_count = r.row_count, session_row_count = s.row_count
71+
--Session history and status
72+
, s.last_request_start_time, s.last_request_end_time, s.reads, s.writes, s.logical_reads, session_status = s.[status], request_status = r.status
73+
--Session connection information
74+
, s.host_name, s.program_name, s.login_name, s.client_interface_name, s.is_user_process
75+
FROM sys.dm_exec_sessions s
76+
LEFT OUTER JOIN sys.dm_exec_requests AS r
77+
ON r.[session_id] = s.[session_id]
78+
LEFT OUTER JOIN sys.dm_exec_query_memory_grants AS mg
79+
ON mg.[session_id] = s.[session_id]
80+
OUTER APPLY sys.dm_exec_sql_text (r.[sql_handle]) AS t
81+
OUTER APPLY sys.dm_exec_input_buffer(s.[session_id], NULL) AS ib
82+
OUTER APPLY sys.dm_exec_query_plan (r.[plan_handle]) AS qp
83+
WHERE mg.granted_memory_kb > 0
84+
ORDER BY mg.granted_memory_kb desc, mg.requested_memory_kb desc;
85+
```
86+
87+
You may decide to use the KILL statement to stop a currently executing query that is holding or waiting for a large memory grant. Use this statement carefully, especially when business critical processes are running. For more information, see [KILL &#40;Transact-SQL&#41;](/sql/t-sql/language-elements/kill-transact-sql).
88+
89+
90+
### Use Query Store to investigate past query memory usage
91+
92+
While the previous sample query reports only live query results, the following query uses the [Query Store](/sql/relational-databases/performance/monitoring-performance-by-using-the-query-store) to return information on past query execution. This can be helpful in investigating an out of memory error that occurred in the past.
93+
94+
The following sample query for Azure SQL Database return important information on query executions recorded by the Query Store. Target the top queries identified for examination and performance tuning, and evaluate whether or not they are executing as intended. Note the time filter on `qsp.last_execution_time` to restrict results to recent history. You can adjust the TOP clause to produce more or fewer results depending on your environment.
95+
96+
```sql
97+
SELECT TOP 10 PERCENT --limit results
98+
a.plan_id, query_id, plan_group_id, query_sql_text
99+
, query_plan = TRY_CAST(query_plan as XML)
100+
, avg_query_max_used_memory
101+
, min_query_max_used_memory
102+
, max_query_max_used_memory
103+
, last_query_max_used_memory
104+
, last_execution_time
105+
, query_count_executions
106+
FROM (
107+
SELECT
108+
qsp.plan_id, qsp.query_id, qsp.plan_group_id, qsp.query_plan, qsqt.query_sql_text
109+
, last_execution_time = MAX(qsp.last_execution_time)
110+
, query_count_executions = SUM(qsrs.count_executions)
111+
, avg_query_max_used_memory = AVG(qsrs.avg_query_max_used_memory)
112+
, min_query_max_used_memory = MIN(qsrs.min_query_max_used_memory)
113+
, max_query_max_used_memory = MAX(qsrs.max_query_max_used_memory)
114+
, last_query_max_used_memory = MAX(qsrs_latest.last_query_max_used_memory) --only from latest result
115+
FROM sys.query_store_plan AS qsp
116+
INNER JOIN sys.query_store_query AS qsq
117+
ON qsp.query_id = qsq.query_id
118+
INNER JOIN sys.query_store_query_text AS qsqt
119+
ON qsq.query_text_id = qsqt.query_text_id
120+
INNER JOIN sys.query_store_runtime_stats AS qsrs
121+
ON qsp.plan_id = qsrs.plan_id
122+
INNER JOIN (SELECT plan_id
123+
, last_query_max_used_memory
124+
, rownum = ROW_NUMBER() OVER (PARTITION BY plan_id ORDER BY last_execution_time DESC)
125+
FROM sys.query_store_runtime_stats qsrs) AS qsrs_latest
126+
ON qsrs_latest.plan_id = qsp.plan_id
127+
AND qsrs_latest.rownum = 1 --use latest last_query_max_used_memory per plan_id
128+
WHERE DATEADD(hour, -24, sysdatetime()) < qsp.last_execution_time --past 24 hours only
129+
AND qsrs_latest.last_query_max_used_memory > 0
130+
GROUP BY qsp.plan_id, qsp.query_id, qsp.plan_group_id, qsp.query_plan, qsqt.query_sql_text
131+
) AS a
132+
ORDER BY max_query_max_used_memory DESC, avg_query_max_used_memory DESC;
133+
```
134+
135+
### Extended events
136+
In addition to the previous information, it may be helpful to capture a trace of the activities on the server to thoroughly investigate an out of memory issue in Azure SQL Database.
137+
138+
There are two ways to capture traces in SQL Server; Extended Events (XEvents) and Profiler Traces. However, [SQL Server Profiler](/sql/tools/sql-server-profiler/sql-server-profiler) is deprecated trace technology not supported for Azure SQL Database. [Extended Events](/sql/relational-databases/extended-events/extended-events) is the newer tracing technology that allows more versatility and less impact to the observed system, and its interface is integrated into SQL Server Management Studio (SSMS).
139+
140+
Refer to the document that explains how to use the [Extended Events New Session Wizard](/sql/relational-databases/extended-events/quick-start-extended-events-in-sql-server) in SSMS. For Azure SQL databases however, SSMS provides an Extended Events subfolder under each database in Object Explorer. Use an Extended Events session to capture these useful events, and identify the queries generating them:
141+
142+
- Category Errors:
143+
- error_reported
144+
- exchange_spill
145+
- hash_spill_details
146+
147+
- Category Execution:
148+
- excessive_non_grant_memory_used
149+
150+
- Category Memory:
151+
- query_memory_grant_blocking
152+
- query_memory_grant_usage
153+
154+
The capture of memory grant blocks, memory grant spills, or excessive memory grants could be potential clue to a query suddenly taking on more memory than it had in the past, and a potential explanation for an emergent out of memory error in an existing workload.
155+
156+
### In-memory OLTP out of memory
157+
158+
You may encounter `Error code 41805: There is insufficient memory in the resource pool '%ls' to run this operation` if using In-Memory OLTP. Reduce the amount of data in memory-optimized tables and memory-optimized table-valued parameters, or scale up the database to a higher service objective to have more memory. For more information on out of memory issues with SQL Server In-Memory OLTP, see [Resolve Out Of Memory issues](/sql/relational-databases/in-memory-oltp/resolve-out-of-memory-issues).
159+
160+
### Get Azure SQL DB support
161+
162+
If out of memory errors persist in Azure SQL Database, file an Azure support request by selecting **Get Support** on the [Azure Support](https://azure.microsoft.com/support/options) site.
163+
164+
## Next steps
165+
166+
- [Intelligent query processing in SQL databases](/sql/relational-databases/performance/intelligent-query-processing)
167+
- [Query processing architecture guide](/sql/relational-databases/query-processing-architecture-guide)
168+
- [Performance Center for SQL Server Database Engine and Azure SQL Database](/sql/relational-databases/performance/performance-center-for-sql-server-database-engine-and-azure-sql-database.md)
169+
- [Troubleshooting connectivity issues and other errors with Azure SQL Database and Azure SQL Managed Instance](troubleshoot-common-errors-issues.md)
170+
- [Troubleshoot transient connection errors in SQL Database and SQL Managed Instance](troubleshoot-common-connectivity-issues.md)
171+
- [Demonstrating Intelligent Query Processing](https://github.com/Microsoft/sql-server-samples/tree/master/samples/features/intelligent-query-processing)
172+
- [Resource management in Azure SQL Database](resource-limits-logical-server.md#memory).

‎articles/azure-sql/toc.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1512,6 +1512,8 @@
15121512
href: database/troubleshoot-common-errors-issues.md
15131513
- name: Common connection issues
15141514
href: database/troubleshoot-common-connectivity-issues.md
1515+
- name: Troubleshoot out of memory errors
1516+
href: database/troubleshoot-memory-errors-issues.md
15151517
- name: Import/Export service hangs
15161518
href: database/database-import-export-hang.md
15171519
- name: Transaction log errors

0 commit comments

Comments
 (0)