Skip to content

Commit c5714f7

Browse files
mburke5678mburke5678
authored
Merge pull request #76344 from sbeskin-redhat/CNV_33888_swap_feature
CNV-33888: Swap feature
2 parents 46262ce + a935284 commit c5714f7

File tree

3 files changed

+347
-0
lines changed

3 files changed

+347
-0
lines changed

‎_topic_maps/_topic_map.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4245,6 +4245,8 @@ Topics:
42454245
# File: virt-configuring-cluster-realtime-workloads
42464246
- Name: About multi-queue functionality
42474247
File: virt-about-multi-queue
4248+
- Name: Configuring higher VM workload density
4249+
File: virt-configuring-higher-vm-workload-density
42484250
- Name: VM disks
42494251
Dir: virtual_disks
42504252
Topics:

‎modules/virt-using-wasp-agent-to-configure-higher-vm-workload-density.adoc

Lines changed: 317 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,317 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * virt/virtual_machines/virt-configuring-higher-vm-workload-density.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="virt-using-wasp-agent-to-configure-higher-vm-workload-density_{context}"]
7+
= Using `wasp-agent` to configure higher VM workload density
8+
9+
The `wasp-agent` component enables an {product-title} cluster to assign swap resources to virtual machine (VM) workloads.
10+
Swap usage is only supported on worker nodes.
11+
12+
[IMPORTANT]
13+
====
14+
Swap resources can be only assigned to virtual machine workloads (VM pods) of the `Burstable` Quality of Service (QoS) class. VM pods of the `Guaranteed` QoS class and pods of any QoS class that do not belong to VMs cannot swap resources.
15+
16+
For descriptions of QoS classes, see link:https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/[Configure Quality of Service for Pods] (Kubernetes documentation).
17+
====
18+
19+
.Prerequisites
20+
21+
* The `oc` tool is available.
22+
* You are logged into the cluster with the cluster-admin role.
23+
* A memory over-commit ratio is defined.
24+
* The node belongs to a worker pool.
25+
26+
.Procedure
27+
28+
. Create a privileged service account by entering the following commands:
29+
+
30+
[source,terminal]
31+
----
32+
$ oc adm new-project wasp
33+
----
34+
+
35+
[source,terminal]
36+
----
37+
$ oc create sa -n wasp wasp
38+
----
39+
+
40+
[source,terminal]
41+
----
42+
$ oc adm policy add-cluster-role-to-user cluster-admin -n wasp -z wasp
43+
----
44+
+
45+
[source,terminal]
46+
----
47+
$ oc adm policy add-scc-to-user -n wasp privileged -z wasp
48+
----
49+
+
50+
[NOTE]
51+
====
52+
The `wasp-agent` component deploys an OCI hook to enable swap usage for containers on the node level. The low-level nature requires the `DaemonSet` object to be privileged.
53+
====
54+
+
55+
. Deploy `wasp-agent`:
56+
+
57+
.. Create a `DaemonSet` object as follows:
58+
+
59+
[source,yaml]
60+
----
61+
apiVersion: machineconfiguration.openshift.io/v1
62+
kind: DaemonSet
63+
apiVersion: apps/v1
64+
metadata:
65+
name: wasp-agent
66+
namespace: wasp
67+
labels:
68+
app: wasp
69+
tier: node
70+
spec:
71+
selector:
72+
matchLabels:
73+
name: wasp
74+
template:
75+
metadata:
76+
annotations:
77+
description: >-
78+
Configures swap for workloads
79+
labels:
80+
name: wasp
81+
spec:
82+
serviceAccountName: wasp
83+
hostPID: true
84+
hostUsers: true
85+
terminationGracePeriodSeconds: 5
86+
containers:
87+
- name: wasp-agent
88+
image: >-
89+
registry.redhat.io/container-native-virtualization/wasp-agent-rhel9:latest
90+
imagePullPolicy: Always
91+
env:
92+
- name: "FSROOT"
93+
value: "/host"
94+
resources:
95+
requests:
96+
cpu: 100m
97+
memory: 50M
98+
securityContext:
99+
privileged: true
100+
volumeMounts:
101+
- name: host
102+
mountPath: "/host"
103+
volumes:
104+
- name: host
105+
hostPath:
106+
path: "/"
107+
priorityClassName: system-node-critical
108+
updateStrategy:
109+
type: RollingUpdate
110+
rollingUpdate:
111+
maxUnavailable: 10%
112+
maxSurge: 0
113+
status: {}
114+
----
115+
.. Get the latest URL from the link:https://registry.redhat.io/container-native-virtualization/wasp-agent-rhel9:latest[RH portal] and enter the URL in the created `DaemonSet` object.
116+
. Configure the `kubelet` service to permit swap:
117+
.. Create a `KubeletConfiguration` file as shown in the example:
118+
+
119+
.Example of a `KubeletConfiguration` file
120+
[source,yaml]
121+
----
122+
apiVersion: machineconfiguration.openshift.io/v1
123+
kind: KubeletConfig
124+
metadata:
125+
name: custom-config
126+
spec:
127+
machineConfigPoolSelector:
128+
matchLabels:
129+
pools.operator.machineconfiguration.openshift.io/worker: '' # MCP
130+
#machine.openshift.io/cluster-api-machine-role: worker # machine
131+
#node-role.kubernetes.io/worker: '' # node
132+
kubeletConfig:
133+
failSwapOn: false
134+
evictionSoft:
135+
memory.available: "1Gi"
136+
evictionSoftGracePeriod:
137+
memory.available: "10s"
138+
----
139+
+
140+
If the cluster is already using an existing `KubeletConfiguration` file, add the following to the `spec` section:
141+
+
142+
[source,yaml]
143+
----
144+
apiVersion: machineconfiguration.openshift.io/v1
145+
kind: KubeletConfig
146+
metadata:
147+
name: custom-config
148+
# ...
149+
spec
150+
# ...
151+
kubeletConfig:
152+
evictionSoft:
153+
memory.available: 1Gi
154+
evictionSoftGracePeriod:
155+
memory.available: 1m30s
156+
failSwapOn: false
157+
----
158+
.. Run the following command:
159+
+
160+
[source,yaml]
161+
----
162+
$ oc wait mcp worker --for condition=Updated=True
163+
----
164+
. Create a `MachineConfig` object to provision swap as follows:
165+
+
166+
[source,yaml]
167+
----
168+
apiVersion: machineconfiguration.openshift.io/v1
169+
kind: MachineConfig
170+
metadata:
171+
labels:
172+
machineconfiguration.openshift.io/role: worker
173+
name: 90-worker-swap
174+
spec:
175+
config:
176+
ignition:
177+
version: 3.4.0
178+
systemd:
179+
units:
180+
- contents: |
181+
[Unit]
182+
Description=Provision and enable swap
183+
ConditionFirstBoot=no
184+
185+
[Service]
186+
Type=oneshot
187+
Environment=SWAP_SIZE_MB=5000
188+
ExecStart=/bin/sh -c "sudo dd if=/dev/zero of=/var/tmp/swapfile count=$SWAP_SIZE_MB bs=1MiB && sudo chmod 600 /var/tmp/swapfile && sudo mkswap /var/tmp/swapfile && sudo swapon /var/tmp/swapfile && free -h"
189+
190+
[Install]
191+
RequiredBy=kubelet-dependencies.target
192+
enabled: true
193+
name: swap-provision.service
194+
----
195+
+
196+
To have enough swap space for the worst-case scenario, make sure to have at least as much swap space provisioned as overcommitted RAM. Calculate the amount of swap space to be provisioned on a node using the following formula:
197+
+
198+
[source,terminal]
199+
----
200+
NODE_SWAP_SPACE = NODE_RAM * (MEMORY_OVER_COMMIT_PERCENT / 100% - 1)
201+
----
202+
+
203+
Example:
204+
+
205+
[source,terminal]
206+
----
207+
NODE_SWAP_SPACE = 16 GB * (150% / 100% - 1)
208+
= 16 GB * (1.5 - 1)
209+
= 16 GB * (0.5)
210+
= 8 GB
211+
----
212+
+
213+
. Deploy alerting rules as follows:
214+
+
215+
[source,yaml]
216+
----
217+
apiVersion: monitoring.openshift.io/v1
218+
kind: AlertingRule
219+
metadata:
220+
name: wasp-alerts
221+
namespace: openshift-monitoring
222+
spec:
223+
groups:
224+
- name: wasp.rules
225+
rules:
226+
- alert: NodeSwapping
227+
annotations:
228+
description: Node {{ $labels.instance }} is swapping at a rate of {{ printf "%.2f" $value }} MB/s
229+
runbook_url: https://github.com/openshift-virtualization/wasp-agent/tree/main/runbooks/alerts/NodeSwapping.md
230+
summary: A node is swapping memory pages
231+
expr: |
232+
# In MB/s
233+
irate(node_memory_SwapFree_bytes{job="node-exporter"}[5m]) / 1024^2 > 0
234+
for: 1m
235+
labels:
236+
severity: critical
237+
----
238+
. Configure {VirtProductName} to use memory overcommit either by using the {product-title} web console or by editing the HyperConverged custom resource (CR) file as shown in the following example.
239+
+
240+
Example:
241+
+
242+
[source,yaml]
243+
----
244+
apiVersion: hco.kubevirt.io/v1beta1
245+
kind: HyperConverged
246+
metadata:
247+
name: kubevirt-hyperconverged
248+
namespace: openshift-cnv
249+
spec:
250+
higherWorkloadDensity:
251+
memoryOvercommitPercentage: 150
252+
----
253+
. Apply all the configurations to compute nodes in your cluster by entering the following command:
254+
+
255+
[source,terminal]
256+
----
257+
$ oc patch --type=merge \
258+
-f <../manifests/hco-set-memory-overcommit.yaml> \
259+
--patch-file <../manifests/hco-set-memory-overcommit.yaml>
260+
----
261+
+
262+
[NOTE]
263+
====
264+
After applying all configurations, the swap feature is fully available only after all `MachineConfigPool` rollouts are complete.
265+
====
266+
267+
.Verification
268+
269+
. To verify the deployment of `wasp-agent`, run the following command:
270+
+
271+
[source, terminal]
272+
----
273+
$ oc rollout status ds wasp-agent -n wasp
274+
----
275+
+
276+
If the deployment is successful, the following message is displayed:
277+
+
278+
[source, terminal]
279+
----
280+
daemon set "wasp-agent" successfully rolled out
281+
----
282+
283+
. To verify that swap is correctly provisioned, do the following:
284+
.. Run the following command:
285+
+
286+
[source,terminal]
287+
----
288+
$ oc get nodes -l node-role.kubernetes.io/worker
289+
----
290+
.. Select a node from the provided list and run the following command:
291+
+
292+
[source,terminal]
293+
----
294+
$ oc debug node/<selected-node> -- free -m
295+
----
296+
+
297+
If swap is provisioned correctly, an amount greater than zero is displayed, similar to the following:
298+
+
299+
[cols="1,1,1,1,1,1,1"]
300+
|===
301+
| |total |used |free |shared |buff/cache |available
302+
|Mem: |31846 |23155 |1044 |6014 |14483 |8690
303+
|Swap: |8191 |2337 |5854 | | |
304+
|===
305+
306+
. Verify the {VirtProductName} memory overcommitment configuration by running the following command:
307+
+
308+
[source,terminal]
309+
----
310+
$ oc get -n openshift-cnv HyperConverged kubevirt-hyperconverged -o jsonpath="{.spec.higherWorkloadDensity.memoryOvercommitPercentage}"
311+
150
312+
----
313+
+
314+
The returned value, for example `150`, must match the value you had previously configured.
315+
316+
317+

‎virt/virtual_machines/virt-configuring-higher-vm-workload-density.adoc

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
:_mod-docs-content-type: ASSEMBLY
2+
[id="virt-configuring-higher-vm-workload-density"]
3+
= Configuring higher VM workload density
4+
include::_attributes/common-attributes.adoc[]
5+
:context: virt-configuring-higher-vm-workload-density
6+
7+
toc::[]
8+
9+
To increase the number of virtual machines (VMs), you can configure a higher VM workload density in your cluster by overcommitting the amount of memory (RAM).
10+
11+
:FeatureName: Configuring higher workload density
12+
include::snippets/technology-preview.adoc[]
13+
14+
The following workloads are especially suited for higher workload density:
15+
16+
* Many similar workloads
17+
* Underused workloads
18+
19+
[NOTE]
20+
====
21+
While overcommitted memory can lead to a higher workload density, it can also lower workload performance of a highly utilized system.
22+
====
23+
24+
include::modules/virt-using-wasp-agent-to-configure-higher-vm-workload-density.adoc[leveloffset=+1]
25+
26+
27+
28+

0 commit comments

Comments
 (0)