You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: support/azure/azure-kubernetes/availability-performance/high-memory-consumption-disk-intensive-applications.md
+21-21Lines changed: 21 additions & 21 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,37 +1,37 @@
1
1
---
2
2
title: Troubleshoot High Memory Consumption in Disk-Intensive Applications
3
3
description: Helps identify and resolve excessive memory usage due to Linux kernel behaviors on Kubernetes pods.
4
-
ms.date: 04/28/2025
4
+
ms.date: 04/29/2025
5
5
ms.reviewer: claudiogodoy, v-weizhu
6
6
ms.service: azure-kubernetes-service
7
7
ms.custom: sap:Node/node pool availability and performance
8
8
---
9
9
# Troubleshoot high memory consumption in disk-intensive applications
10
10
11
-
Disk input and output operations are costly, and most operating systems implement caching strategies for reading and writing data to the filesystem. [Linux kernel](https://www.kernel.org/doc) usually uses strategies such as the [page cache](https://www.kernel.org/doc/gorman/html/understand/understand013.html) to improve the overall performance. The primary goal of the page cache is to store data that's read from the filesystem in cache, making it available in memory for future read operations.
11
+
Disk input and output operations are costly, and most operating systems implement caching strategies for reading and writing data to the filesystem. The [Linux kernel](https://www.kernel.org/doc) usually uses strategies such as the [page cache](https://www.kernel.org/doc/gorman/html/understand/understand013.html) to improve overall performance. The primary goal of the page cache is to store data read from the filesystem in the cache, making it available in memory for future read operations.
12
12
13
-
This article helps you to identity and avoid high memory consumed by disk-intensive applications due to Linux kernel behaviors on Kubernetes pods.
13
+
This article helps you identity and avoid the high memory consumption caused by disk-intensive applications due to Linux kernel behaviors on Kubernetes pods.
14
14
15
15
## Prerequisites
16
16
17
-
-A tool to connect to the Kubernetes cluster, such as the `kubectl` tool. To install `kubectl` using the [Azure CLI](/cli/azure/install-azure-cli), run the [az aks install-cli](/cli/azure/aks#az-aks-install-cli) command.
17
+
A tool to connect to the Kubernetes cluster, such as the `kubectl` tool. To install `kubectl` using the [Azure CLI](/cli/azure/install-azure-cli), run the [az aks install-cli](/cli/azure/aks#az-aks-install-cli) command.
18
18
19
19
## Symptoms
20
20
21
-
When a disk-intensive application running on a pod perform frequent filesystem operations, high memory consumption might occur.
21
+
When a disk-intensive application running on a pod performs frequent filesystem operations, high memory consumption might occur.
22
22
23
-
The following table outlines the common symptoms of high memory consumption:
23
+
The following table outlines common symptoms of high memory consumption:
24
24
25
25
| Symptom | Description |
26
26
| --- | --- |
27
-
| The [working set](https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#memory) metric too high | This issue occurs when there is a significant difference between the [working set](https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#memory) metric reported by the [Kubernetes Metrics API](https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#metrics-server) and the actual memory consumed by an application. |
28
-
| Out-of-memory (OOM) kill | This issue indicates memory issues exist on your pod. |
29
-
| Increased memory usage after heavy disk activity | After operations such as backups, large file reads/writes, or data imports, memory consumption rises. |
30
-
| Memory usage grows indefinitely | The pod's memory consumption increases over time without reducing, like a memory leak, even if the application itself isn’t leaking memory.|
27
+
| The [working set](https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#memory) metric is too high.| This issue occurs when there's a significant difference between the [working set](https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#memory) metric reported by the [Kubernetes Metrics API](https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#metrics-server) and the actual memory consumed by an application. |
28
+
| Out-of-memory (OOM) kill.| This issue indicates memory issues exist on your pod. |
29
+
| Increased memory usage after heavy disk activity.| After operations such as backups, large file reads/writes, or data imports, memory consumption rises. |
30
+
| Memory usage grows indefinitely.| The pod's memory consumption increases over time without reducing, like a memory leak, even if the application itself isn't leaking memory.|
31
31
32
32
## Troubleshooting checklist
33
33
34
-
### Step 1: Inspect pod working set
34
+
### Step 1: Inspect the pod working set
35
35
36
36
To inspect the working set of pods reported by the Kubernetes Metrics API, run the following [kubectl top pods](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_top/) command:
37
37
@@ -53,7 +53,7 @@ To inspect the memory statistics of the [cgroup](https://www.kernel.org/doc/html
53
53
$ kubectl exec <POD_NAME> -it -- bash
54
54
```
55
55
56
-
2. Navigate to the `cgroup` statistics directory and list memory-related files:
56
+
2. Navigate to the `cgroup` statistics directory and list the memory-related files:
57
57
58
58
```console
59
59
$ ls /sys/fs/cgroup | grep -e memory.stat -e memory.current
@@ -63,9 +63,9 @@ To inspect the memory statistics of the [cgroup](https://www.kernel.org/doc/html
63
63
- `memory.current`: Total memory currently used by the `cgroup` and its descendants.
64
64
- `memory.stat`: This breaks down the cgroup's memory footprint into different types of memory, type-specific details, and other information about the state and past events of the memory management system.
65
65
66
-
All the values listed on those files are in bytes.
66
+
All the values listed in those files are in bytes.
67
67
68
-
3. Get an overview about how the memory consumption is distributed on the pod:
68
+
3. Get an overview of how memory consumption is distributed on the pod:
69
69
70
70
```console
71
71
$ cat /sys/fs/cgroup/memory.current
@@ -95,11 +95,11 @@ The following table describes some memory segments:
95
95
96
96
| Segment | Description |
97
97
|---|---|
98
-
| `anon` | Amount of memory used in anonymous mappings. The majority languages use this segment to allocate memory. |
99
-
| `file` | Amount of memory used to cache filesystem data, including tmpfs and shared memory. |
100
-
| `slab` | Amount of memory used for storing data structures in the Linux kernel. |
98
+
| `anon` | The amount of memory used in anonymous mappings. Most languages use this segment to allocate memory. |
99
+
| `file` | The amount of memory used to cache filesystem data, including tmpfs and shared memory. |
100
+
| `slab` | The amount of memory used to store data structures in the Linux kernel. |
101
101
102
-
Combined with the [Step 2](#step-2-inspect-pod-memory-statistics), the `anon` represents 5197824 bytes which isn't close to the total amount reported by the working set metric. The `slab` memory segment used by the Linux kernel represents 354682456 bytes, which is almost all the memory reported by working set metric on the pod.
102
+
Combined with [Step 2](#step-2-inspect-pod-memory-statistics), `anon` represents 5,197,824 bytes, which isn't close to the total amount reported by the working set metric. The `slab` memory segment used by the Linux kernel represents 354,682,456 bytes, which is almost all the memory reported by the working set metric on the pod.
103
103
104
104
### Step 4: Drop the kernel cache on a debugger pod
105
105
@@ -143,21 +143,21 @@ Combined with the [Step 2](#step-2-inspect-pod-memory-statistics), the `anon` re
143
143
slab 392768
144
144
```
145
145
146
-
If you observe a significant decrease in both working set and `slab` memory segment, you are experiencing the issue where a great amount of memory is used by the Linux kernel on the pod.
146
+
If you observe a significant decrease in both the working set and the `slab` memory segment, you're experiencing an issue with the Linux kernel using a great amount of memory on the pod.
147
147
148
148
## Workaround: Configure appropriate memory limits and requests
149
149
150
150
The only effective workaround for high memory consumption on Kubernetes pods is to set realistic resource [limits and requests](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits). For example:
151
151
152
-
```ymal
152
+
```yaml
153
153
resources:
154
154
requests:
155
155
memory: 30Mi
156
156
limits:
157
157
memory: 60Mi
158
158
```
159
159
160
-
By configuring appropriate memory limits and requests in the Kubernetes or specification, you can ensure that Kubernetes manages memory allocation more efficiently, mitigating the impact of excessive kernel-level caching on pod memory usage.
160
+
By configuring appropriate memory limits and requests in Kubernetes or the specification, you can ensure that Kubernetes manages memory allocation more efficiently, mitigating the impact of excessive kernel-level caching on pod memory usage.
161
161
162
162
> [!CAUTION]
163
163
> Misconfigured pod memory limits can lead to problems such as OOMKilled errors.
0 commit comments