You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: support/azure/azure-kubernetes/availability-performance/high-memory-consumption-disk-intensive-applications.md
+34-33Lines changed: 34 additions & 33 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,55 +10,60 @@ ms.custom: sap:Node/node pool availability and performance
10
10
11
11
Disk input and output operations are costly, and most operating systems implement caching strategies for reading and writing data to the filesystem. [Linux kernel](https://www.kernel.org/doc) usually uses strategies such as the [page cache](https://www.kernel.org/doc/gorman/html/understand/understand013.html) to improve the overall performance. The primary goal of the page cache is to store data that's read from the filesystem in cache, making it available in memory for future read operations.
12
12
13
-
When disk-intensive applications perform frequent filesystem operations, high memory consumption might occur. This article helps you to identity and resolve this issue due to Linux kernel behaviors on Kubernetes pods.
13
+
This article helps you to identity and avoid high memory consumed by disk-intensive applications due to Linux kernel behaviors on Kubernetes pods.
14
14
15
15
## Prerequisites
16
16
17
-
- A tool to connect to the Kubernetes cluster, such as the kubectl tool. To install kubectl using the [Azure CLI](/cli/azure/install-azure-cli), run the [az aks install-cli](/cli/azure/aks#az-aks-install-cli) command.
17
+
- A tool to connect to the Kubernetes cluster, such as the `kubectl` tool. To install `kubectl` using the [Azure CLI](/cli/azure/install-azure-cli), run the [az aks install-cli](/cli/azure/aks#az-aks-install-cli) command.
18
18
19
19
## Symptoms
20
20
21
+
When an disk-intensive application running on a pod perform frequent filesystem operations, high memory consumption might occur.
22
+
21
23
The following table outlines the common symptoms of memory saturation:
22
24
23
25
| Symptom | Description |
24
26
| --- | --- |
25
-
|[Working set](https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#memory) metric too high | This issue occurs when there is a significant difference between the working_set metric reported by the [Kubernetes Metrics API](https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#metrics-server) and the actual memory consumed by an application. |
27
+
|The [working set](https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#memory) metric too high | This issue occurs when there is a significant difference between the [working set](https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#memory) metric reported by the [Kubernetes Metrics API](https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#metrics-server) and the actual memory consumed by an application. |
26
28
| Out-of-memory (OOM) kill | This issue indicates memory issues exist on your pod. |
27
29
28
30
## Troubleshooting checklist
29
31
30
32
### Step 1: Inspect pod working set
31
33
32
-
1. Identify which pod is consuming excessive memory by following the guide[Troubleshoot memory saturation in AKS clusters](identify-memory-saturation-aks.md).
33
-
2. Use the following [kubectl top pods](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_top/) command to show the actual [Working_Set](https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#memory) reported by the [Kubernetes metrics API](https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#metrics-server):
34
+
To inspect the working set of pods reported by the Kubernetes Metrics API, run the following [kubectl top pods](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_top/) command:
34
35
35
-
```console
36
-
$ kubectl top pods -A | grep -i "<DEPLOYMENT_NAME>"
37
-
NAME CPU(cores) MEMORY(bytes)
38
-
my-deployment-fc94b7f98-m9z2l 1m 344Mi
39
-
```
36
+
```console
37
+
$ kubectl top pods -A | grep -i "<DEPLOYMENT_NAME>"
38
+
NAME CPU(cores) MEMORY(bytes)
39
+
my-deployment-fc94b7f98-m9z2l 1m 344Mi
40
+
```
41
+
42
+
For detailed steps about how to identify which pod is consuming excessive memory, see [Troubleshoot memory saturation in AKS clusters](identify-memory-saturation-aks.md#step-1-identify-nodes-that-have-memory-saturation).
40
43
41
44
### Step 2: Inspect pod memory statistics
42
45
43
-
Inspect the memory statistics of the [cgroup](https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html) of the pod by following these steps:
46
+
To inspect the memory statistics of the [cgroup](https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html)on the pod that's consuming excessive memory, follow these steps:
44
47
45
48
1. Connect to the pod:
46
49
47
50
```console
48
51
$ kubectl exec <POD_NAME> -it -- bash
49
52
```
50
53
51
-
2. Navigate to the cgroup statistics directory and list memory-related files:
54
+
2. Navigate to the `cgroup` statistics directory and list memory-related files:
52
55
53
56
```console
54
57
$ ls /sys/fs/cgroup | grep -e memory.stat -e memory.current
55
58
memory.current memory.stat
56
59
```
57
60
58
-
- `memory.current`: Total memory currently used by the cgroup and its descendants.
59
-
- `memory.stat`: This breaks down the cgroup's memory footprint into different types of memory, type-specific details, and other information on the state and past events of the memory management system.
61
+
- `memory.current`: Total memory currently used by the `cgroup` and its descendants.
62
+
- `memory.stat`: This breaks down the cgroup's memory footprint into different types of memory, type-specific details, and other information about the state and past events of the memory management system.
63
+
64
+
All the values listed on those files are in bytes.
60
65
61
-
3. All the values listed on those files are in bytes. Get an overview of how the memory consumption is distributed on the `pod`:
66
+
3. Get an overview about how the memory consumption is distributed on the pod:
62
67
63
68
```console
64
69
$ cat /sys/fs/cgroup/memory.current
@@ -78,29 +83,25 @@ Inspect the memory statistics of the [cgroup](https://www.kernel.org/doc/html/la
78
83
...
79
84
```
80
85
81
-
`cAdvisor` uses `memory.current` and `inactive_file` to compute the working set metric. You can replicate the calculation using the following formula:
86
+
`cAdvisor` uses `memory.current` and `inactive_file` to compute the working set metric. You can replicate the calculation using the following formula:
### Step 3: Determine kernel vs. application memory consumption
92
+
### Step 3: Determine kernel and application memory consumption
90
93
91
94
The following table describes some memory segments:
92
95
93
96
| Segment | Description |
94
97
|---|---|
95
-
| anon | Amount of memory used in anonymous mappings. The majority languages use this segment to allocate memory. |
98
+
| `anon` | Amount of memory used in anonymous mappings. The majority languages use this segment to allocate memory. |
96
99
| file | Amount of memory used to cache filesystem data, including tmpfs and shared memory. |
97
-
| slab | Amount of memory used for storing in-kernel data structures. |
100
+
| `slab` | Amount of memory used for storing data structures in the Linux kernel. |
98
101
99
-
The majority of languages use the anon memory segment to allocate resources. In this case, the `anon` represents 5197824 bytes which is not even close to the total amount reported by the working set metric.
102
+
In this case, the `anon` represents 5197824 bytes which is not even close to the total amount reported by the working set metric. The `slab` memory segment used by the Linux kernel represents 354682456 bytes, which is almost all the memory reported by working set metric on the pod.
100
103
101
-
On the other hand, there is one of the segments that Kernel uses the `slab` representing 354682456 bytes, which is almost all the memory reported by working set metric on this pod.
102
-
103
-
### Step 4: Run a node drop cache
104
+
### Step 4: Drop the kernel cache on a debugger pod
104
105
105
106
> [!NOTE]
106
107
> This step might lead to availability and performance issues. Avoid running it in a production environment.
@@ -126,7 +127,7 @@ On the other hand, there is one of the segments that Kernel uses the `slab` repr
126
127
echo 1 > /proc/sys/vm/drop_caches
127
128
```
128
129
129
-
4. Verify if the command in the previous step causes the effect by repeating [Step 1](#step-1-inspect-pod-working-set) and [Step 2](#step-2-inspect-pod-memory-statistics):
130
+
4. Verify if the command in the previous step causes any effect by repeating [Step 1](#step-1-inspect-pod-working-set) and [Step 2](#step-2-inspect-pod-memory-statistics):
130
131
131
132
```console
132
133
$ kubectl top pods -A | grep -i "<DEPLOYMENT_NAME>"
@@ -142,9 +143,9 @@ On the other hand, there is one of the segments that Kernel uses the `slab` repr
142
143
slab 392768
143
144
```
144
145
145
-
If you observe a significant decrease in both working set and slab memory segment, you are experiencing the issue where a great amount of pod's memory is used by the Kernel.
146
+
If you observe a significant decrease in both working set and `slab` memory segment, you are experiencing the issue where a great amount of memory is used by the Linux kernel on the pod.
146
147
147
-
## Workaround: Set appropriate memory limits and requests
148
+
## Workaround: Configure appropriate memory limits and requests
148
149
149
150
The only effective workaround for high memory consumption on Kubernetes pods is to set realistic resource [limits and requests](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits). For example:
150
151
@@ -158,8 +159,8 @@ resources:
158
159
159
160
By configuring appropriate memory limits and requests in the Kubernetes or specification, you can ensure that Kubernetes manages memory allocation more efficiently, mitigating the impact of excessive kernel-level caching on pod memory usage.
160
161
161
-
> [!NOTE]
162
-
> Misconfigured pod memory limits can lead to problems such as OOM-Killed errors.
162
+
> [!CAUTION]
163
+
> Misconfigured pod memory limits can lead to problems such as OOMKilled errors.
0 commit comments