Skip to content

Commit cd74e91

Browse files
author
Shikha Jhala
committed
CNV-40007: Added AAQ Operator docs
1 parent 18c86e9 commit cd74e91

File tree

5 files changed

+174
-0
lines changed

5 files changed

+174
-0
lines changed

‎_topic_maps/_topic_map.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4421,6 +4421,8 @@ Topics:
44214421
#Advanced virtual machine configuration
44224422
- Name: Working with resource quotas for virtual machines
44234423
File: virt-working-with-resource-quotas-for-vms
4424+
- Name: Configuring the Application-Aware Quota Operator
4425+
File: virt-understanding-aaq-operator
44244426
- Name: Specifying nodes for virtual machines
44254427
File: virt-specifying-nodes-for-vms
44264428
- Name: Activating kernel samepage merging (KSM)

‎modules/virt-about-aaq-operator.adoc

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * virt/virtual_machines/advanced_vm_management/virt-understanding-aaq-operator.adoc
4+
5+
:_mod-docs-content-type: CONCEPT
6+
[id="virt-about-aaq-operator_{context}"]
7+
= About the AAQ Operator
8+
9+
The Application-Aware Quota (AAQ) Operator provides more flexible and extensible quota management compared to the native `ResourceQuota` object in the {product-title} platform.
10+
11+
In a multi-tenant cluster environment, where multiple workloads operate on shared infrastructure and resources, using the Kubernetes native `ResourceQuota` object to limit aggregate CPU and memory consumption presents infrastructure overhead and live migration challenges for {VirtProductName} workloads.
12+
13+
{VirtProductName} requires significant compute resource allocation to handle virtual machine (VM) live migrations and manage VM infrastructure overhead. When upgrading {VirtProductName}, you must migrate VMs to upgrade the `virt-launcher` pod. However, migrating a VM in the presence of a resource quota can cause the migration, and subsequently the upgrade, to fail.
14+
15+
With AAQ, you can allocate resources for VMs without interfering with cluster-level activities such as upgrades and node maintenance. The AAQ Operator also supports non-compute resources which eliminates the need to manage both the native resource quota and AAQ API objects separately.
16+
17+
18+
[id="aaq-controller-and-crds_{context}"]
19+
== AAQ Operator controller and custom resources
20+
The AAQ Operator introduces two new API objects defined as custom resource definitions (CRDs) for managing alternative quota implementations across multiple namespaces:
21+
22+
* `ApplicationAwareResourceQuota`: Sets aggregate quota restrictions enforced per namespace. The `ApplicationAwareResourceQuota` API is compatible with the native `ResourceQuota` object and shares the same specification and status definitions.
23+
+
24+
.Example manifest
25+
[source,yaml]
26+
----
27+
apiVersion: aaq.kubevirt.io/v1alpha1
28+
kind: ApplicationAwareResourceQuota
29+
metadata:
30+
name: example-resource-quota
31+
spec:
32+
hard:
33+
requests.memory: 1Gi
34+
limits.memory: 1Gi
35+
requests.cpu/vmi: "1" # <1>
36+
requests.memory/vmi: 1Gi # <2>
37+
# ...
38+
----
39+
<1> The maximum amount of CPU that is allowed for VM workloads in the default namespace.
40+
<2> The maximum amount of RAM that is allowed for VM workloads in the default namespace.
41+
42+
* `ApplicationAwareClusterResourceQuota`: Mirrors the `ApplicationAwareResourceQuota` object at a cluster scope. It is compatible with the native `ClusterResourceQuota` API object and shares the same specification and status definitions. When creating an AAQ cluster quota, you can select multiple namespaces based on annotation selection, label selection, or both by editing the `spec.selector.labels` or `spec.selector.annotations` fields.
43+
+
44+
.Example manifest
45+
[source,yaml]
46+
----
47+
apiVersion: aaq.kubevirt.io/v1alpha1
48+
kind: ApplicationAwareClusterResourceQuota # <1>
49+
metadata:
50+
name: example-resource-quota
51+
spec:
52+
quota:
53+
hard:
54+
requests.memory: 1Gi
55+
limits.memory: 1Gi
56+
requests.cpu/vmi: "1"
57+
requests.memory/vmi: 1Gi
58+
selector:
59+
annotations: null
60+
labels:
61+
matchLabels:
62+
kubernetes.io/metadata.name: default
63+
# ...
64+
----
65+
<1> You can only create an `ApplicationAwareClusterResourceQuota` object if the `spec.allowApplicationAwareClusterResourceQuota` field in the `HyperConverged` custom resource (CR) is set to `true`.
66+
+
67+
[NOTE]
68+
====
69+
If both `spec.selector.labels` and `spec.selector.annotations` fields are set, only namespaces that match both are selected.
70+
====
71+
72+
The AAQ controller uses a scheduling gate mechanism to evaluate whether there is enough of a resource available to run a workload. If so, the scheduling gate is removed from the pod and it is considered ready for scheduling. The quota usage status is updated to indicate how much of the quota is used.
73+
74+
If the CPU and memory requests and limits for the workload exceed the enforced quota usage limit, the pod remains in `SchedulingGated` status until there is enough quota available. The AAQ controller creates an event of type `Warning` with details on why the quota was exceeded. You can view the event details by using the `oc get events` command.
75+
76+
[IMPORTANT]
77+
====
78+
Pods that have the `spec.nodeName` field set to a specific node cannot use namespaces that match the `spec.namespaceSelector` labels defined in the `HyperConverged` CR.
79+
====

‎modules/virt-configuring-aaq-operator.adoc

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * virt/virtual_machines/advanced_vm_management/virt-understanding-aaq-operator.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="virt-configuring-aaq-operator_{context}"]
7+
= Configuring the AAQ Operator by using the CLI
8+
9+
You can configure the AAQ Operator by specifying the fields of the `spec.applicationAwareConfig` object in the `HyperConverged` custom resource (CR).
10+
11+
.Prerequisites
12+
* You have access to the cluster as a user with `cluster-admin` privileges.
13+
* You have installed the OpenShift CLI (`oc`).
14+
15+
.Procedure
16+
* Update the `HyperConverged` CR by running the following command:
17+
+
18+
[source,terminal]
19+
----
20+
$ oc patch hco kubevirt-hyperconverged -n openshift-cnv --type merge -p '{
21+
"spec": {
22+
"applicationAwareConfig": {
23+
"vmiCalcConfigName": "DedicatedVirtualResources",
24+
"namespaceSelector": {
25+
"matchLabels": {
26+
"app": "my-app"
27+
}
28+
},
29+
"allowApplicationAwareClusterResourceQuota": true
30+
}
31+
}
32+
}'
33+
----
34+
+
35+
where:
36+
37+
`vmiCalcConfigName`:: Specifies how resource counting is managed for pods that run virtual machine (VM) workloads. Possible values are:
38+
+
39+
--
40+
* `VmiPodUsage`: Counts compute resources for pods associated with VMs in the same way as native resource quotas and excludes migration-related resources.
41+
* `VirtualResources`: Counts compute resources based on the VM specifications, using the VM RAM size for memory and virtual CPUs for processing.
42+
* `DedicatedVirtualResources` (default): Similar to `VirtualResources`, but separates resource tracking for pods associated with VMs by adding a `/vmi` suffix to CPU and memory resource names. For example, `requests.cpu/vmi` and `requests.memory/vmi`.
43+
--
44+
`namespaceSelector`:: Determines the namespaces for which an AAQ scheduling gate is added to pods when they are created. If a namespace selector is not defined, the AAQ Operator targets namespaces with the `application-aware-quota/enable-gating` label as default.
45+
`allowApplicationAwareClusterResourceQuota`:: If set to `true`, you can create and manage the `ApplicationAwareClusterResourceQuota` object. Setting this attribute to `true` can increase scheduling time.

‎modules/virt-enabling-aaq-operator.adoc

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * virt/virtual_machines/advanced_vm_management/virt-understanding-aaq-operator.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="virt-enabling-aaq-operator_{context}"]
7+
= Enabling the AAQ Operator
8+
9+
To deploy the AAQ Operator, set the `enableApplicationAwareQuota` feature gate to `true` in the `HyperConverged` custom resource (CR).
10+
11+
.Prerequisites
12+
* You have access to the cluster as a user with `cluster-admin` privileges.
13+
* You have installed the OpenShift CLI (`oc`).
14+
15+
.Procedure
16+
* Set the `enableApplicationAwareQuota` feature gate to `true` in the `HyperConverged` CR by running the following command:
17+
+
18+
[source,terminal]
19+
----
20+
$ oc patch hco kubevirt-hyperconverged -n openshift-cnv \
21+
--type json -p '[{"op": "add", "path": "/spec/featureGates/enableApplicationAwareQuota", "value": true}]'
22+
----

‎virt/virtual_machines/advanced_vm_management/virt-understanding-aaq-operator.adoc

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
:_mod-docs-content-type: ASSEMBLY
2+
[id="virt-understanding-aaq-operator"]
3+
= Configuring the Application-Aware Quota (AAQ) Operator
4+
include::_attributes/common-attributes.adoc[]
5+
:context: virt-understanding-aaq-operator
6+
7+
toc::[]
8+
9+
You can use the Application-Aware Quota (AAQ) Operator to customize and manage resource quotas for individual components in an {product-title} cluster.
10+
11+
include::modules/virt-about-aaq-operator.adoc[leveloffset=+1]
12+
13+
include::modules/virt-enabling-aaq-operator.adoc[leveloffset=+1]
14+
15+
include::modules/virt-configuring-aaq-operator.adoc[leveloffset=+1]
16+
17+
[role="_additional-resources"]
18+
[id="additional-resources_{context}"]
19+
== Additional resources
20+
21+
* xref:../../../applications/quotas/quotas-setting-per-project.adoc#quotas-setting-per-project[Resource quotas per project]
22+
* xref:../../../applications/quotas/quotas-setting-across-multiple-projects.adoc#quotas-setting-across-multiple-projects[Resource quotas across multiple projects]
23+
* xref:../../../rest_api/schedule_and_quota_apis/resourcequota-v1.adoc#resourcequota-v1[`ResourceQuota` API reference]
24+
* xref:../../../rest_api/schedule_and_quota_apis/clusterresourcequota-quota-openshift-io-v1.adoc#clusterresourcequota-quota-openshift-io-v1[`ClusterResourceQuota` API reference]
25+
* xref:../../../rest_api/workloads_apis/pod-v1.adoc#spec-schedulinggates[Pod scheduling gates specification]
26+
* xref:../../../nodes/clusters/nodes-containers-events.adoc#nodes-containers-events[Viewing system event information in an {product-title} cluster]

0 commit comments

Comments
 (0)