Skip to content

Commit 788f33f

Browse files
hyderhyder
authored
feat: adds a separate nodepool to run cluster autoscaling (#576)
Resolves #273 Signed-off-by: Ali Mukadam <[email protected]>
1 parent 52b621e commit 788f33f

18 files changed

+891
-21
lines changed

‎docs/clusterautoscaler.adoc‎

Lines changed: 310 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,310 @@
1+
= Using the Oracle Container Engine for Kubernetes Cluster Autoscaler
2+
:idprefix:
3+
:idseparator: -
4+
:sectlinks:
5+
:sectnums:
6+
:toc: auto
7+
8+
9+
:uri-repo: https://github.com/oracle-terraform-modules/terraform-oci-oke
10+
:uri-rel-file-base: link:{uri-repo}/blob/main
11+
:uri-rel-tree-base: link:{uri-repo}/tree/main
12+
:uri-docs: {uri-rel-file-base}/docs
13+
:uri-cluster-autoscaler-parameters: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-are-the-parameters-to-ca
14+
:uri-instructions: {uri-docs}/instructions.adoc
15+
:uri-oci-keys: https://docs.cloud.oracle.com/iaas/Content/API/Concepts/apisigningkey.htm
16+
:uri-oci-ocids: https://docs.cloud.oracle.com/iaas/Content/API/Concepts/apisigningkey.htm#five
17+
:uri-oci-okepolicy: https://docs.cloud.oracle.com/iaas/Content/ContEng/Concepts/contengpolicyconfig.htm#PolicyPrerequisitesService
18+
:uri-oci-cluster-autoscaler: https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengusingclusterautoscaler.htm#Working_with_the_Cluster_Autoscaler
19+
:uri-terraform: https://www.terraform.io
20+
:uri-terraform-oci: https://www.terraform.io/docs/providers/oci/index.html
21+
:uri-terraform-options: {uri-docs}/terraformoptions.adoc
22+
:uri-topology: {uri-docs}/topology.adoc
23+
:uri-upgrade-oke: https://docs.cloud.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengupgradingk8sworkernode.htm
24+
:uri-variables: {uri-rel-file-base}/variables.tf
25+
26+
This section documents how to deploy the Oracle Container Engine for Kubernetes (OKE) Cluster Autoscaler when using this project. At a high level, deploying the Kubernetes Cluster Autoscaler consists of 3 steps:
27+
28+
1. Deploy an _unmanaged_ node pool i.e. a node pool that is not managed by the Kubernetes Cluster Autoscaler. We'll refer to this node pool as the _autoscaler_ node pool.
29+
2. Create a dynamic group and policy to allow worker nodes to manage node pools. The dynamic group uses defined tags to add worker nodes from the autoscaler node pool to the managed node pools.
30+
3. Configure and deploy the Kubernetes Cluster Autoscaler.
31+
32+
== Prerequisites
33+
34+
=== Tag namespace and tag key
35+
36+
. Create a tag namespace named `oke` .
37+
. Create a tag key named `pool` in the `oke` tag namespace:
38+
.. Set the description to "Tag key to indicate node pool purpose"
39+
.. Set the value type to "String"
40+
.. Set the value to `autoscaler` which assigns any node pool with this tag to the Kubernetes Cluster Autoscaler.
41+
42+
To use this list to indicate other purposes besides the Kubernetes Cluster Autoscaler, set the value type to "A list of values" instead of "String" and add your own customer purposes.
43+
44+
45+
. bastion host is created
46+
. operator host is created
47+
. instance_principal is enabled on operator
48+
. set `enable_cluster_autoscaler=true`
49+
50+
== Create required OCI resources
51+
52+
The Kubernetes Cluster Autoscaler is installed to the `autoscaler` node pool and the required Oracle Identity and Access Management policies are created to allow the Kubernetes Cluster Autoscaler to resize the managed node pools based on application workload.
53+
54+
=== Create the `autoscaler` node pool
55+
56+
. Add the following configuration parameters to your `terraform.tfvars`:
57+
58+
+
59+
----
60+
enable_cluster_autoscaler = true
61+
autoscaler_pools = {
62+
# only 1 pool is needed at a time unless:
63+
# 1. you are in the process of upgrading of your cluster
64+
# 2. you are running multiple Kubernetes versions of node pools after a cluster upgrade
65+
asp_v124 = {}
66+
}
67+
----
68+
. The `_v124` suffix to the node pool name indicates the version of Kubernetes running in that node pool.
69+
. Run `terraform apply` to create the `autoscaler` node pools.
70+
. Verify that the `asp_v124` node pool of three nodes has been created. Each node in the pool should have the `oke.pool = autoscaler` tag assigned.
71+
72+
73+
=== Create an OCI dynamic group
74+
75+
. Create a dynamic group `cluster-autoscaler-group` which contains the following matching rule:
76+
77+
+
78+
----
79+
tag.oke.pool.value='autoscaler'
80+
----
81+
82+
=== Create an OCI policy
83+
84+
. Replace `<compartment-name>` in the following policy statements with the name of the compartment in which the managed node pools must be or have been created.
85+
. Create a policy named `cluster-autoscaler-policy` in the root of your tenancy with the policy statements created in the previous step.
86+
87+
+
88+
----
89+
Allow dynamic-group cluster-autoscaler-group to manage cluster-node-pools in compartment <compartment-name>
90+
Allow dynamic-group cluster-autoscaler-group to manage instance-family in compartment <compartment-name>
91+
Allow dynamic-group cluster-autoscaler-group to use subnets in compartment <compartment-name>
92+
Allow dynamic-group cluster-autoscaler-group to read virtual-network-family in compartment <compartment-name>
93+
Allow dynamic-group cluster-autoscaler-group to use vnics in compartment <compartment-name>
94+
Allow dynamic-group cluster-autoscaler-group to inspect compartments in compartment <compartment-name>
95+
----
96+
97+
== Deploy the Kubernetes Cluster Autoscaler
98+
99+
=== Create a Kubernetes manifest
100+
101+
To simplify deployment, only Kubernetes v1.24 and higher are tested with this module. Using the module with an older version of Kubernetes requires amending the OCI provider configuration and applying a valid image tag as documented in the {uri-oci-cluster-autoscaler}[cluster autoscaler documentation].
102+
103+
. On the operator host, a file called `cluster-autoscaler.yaml` will be created. It will only include the list of that should be automatically scaled by the Kubernetes Cluster Autoscaler:
104+
105+
+
106+
----
107+
# node pool not managed by Kubernetes Cluster Autoscaler
108+
np1 = {
109+
shape = "VM.Standard.E4.Flex",
110+
ocpus = 2,
111+
memory = 32,
112+
node_pool_size = 1,
113+
max_node_pool_size = 5,
114+
boot_volume_size = 150,
115+
autoscale = false,
116+
}
117+
118+
# node pool managed by Kubernetes Cluster Autoscaler
119+
np2 = {
120+
shape = "VM.Standard.E4.Flex",
121+
ocpus = 2,
122+
memory = 32,
123+
node_pool_size = 1,
124+
max_node_pool_size = 5,
125+
boot_volume_size = 150,
126+
autoscale = true,
127+
}
128+
----
129+
130+
. The `cluster-autoscaler.yaml` file can be used to configure other {uri-cluster-autoscaler-parameters}[Kubernetes Cluster Autoscaler parameters], if required e.g.
131+
132+
+
133+
----
134+
--scale-down-unneeded-time=5m
135+
----
136+
137+
. The following parameters are not currently supported by the Kubernetes Cluster Autoscaler:
138+
.. `--node-group-auto-discovery`
139+
.. `--node-autoprovisioning-enabled=true`
140+
.. `--gpu-total`
141+
.. `--expander=price`
142+
143+
== Upgrading the OKE cluster when cluster autoscaler is deployed
144+
145+
Assume the following cluster:
146+
147+
- Cluster version: 1.24.1
148+
- Node pools : np1, np2 (1.24.1)
149+
- Autoscaler pools : asp_v124 (1.24.1)
150+
151+
and we need to upgade to 1.25.1.
152+
153+
****
154+
Note that at this time, OKE is not yet supporting Kubernetes 1.25. This is just to illustrate the procedure.
155+
****
156+
157+
=== Upgrading the control plane nodes
158+
159+
. Locate your `kubernetes_version` in your Terraform variable file and change:
160+
161+
+
162+
----
163+
kubernetes_version = "v1.24.1"
164+
----
165+
to
166+
167+
+
168+
----
169+
kubernetes_version = "v1.25.1"
170+
----
171+
172+
. Run terraform:
173+
174+
+
175+
----
176+
terraform apply
177+
----
178+
179+
This will upgrade the control plane nodes. You can verify this in the OCI Console.
180+
181+
****
182+
If you have modified the default resources e.g. security lists, you will need to use a targeted apply:
183+
184+
----
185+
terraform apply --target=module.oke.k8s_cluster
186+
----
187+
****
188+
189+
=== Add new node pools
190+
1. Add new node pools in your list of node pools e.g. change
191+
+
192+
[source,bash]
193+
----
194+
node_pools = {
195+
np1 = {
196+
...
197+
}
198+
np2 = {
199+
...
200+
}
201+
}
202+
----
203+
to
204+
205+
+
206+
----
207+
node_pools = {
208+
np1 = {
209+
...
210+
}
211+
np2 = {
212+
...
213+
}
214+
np3 = {
215+
...
216+
}
217+
np4 = {
218+
...
219+
}
220+
}
221+
----
222+
223+
and run `terraform apply` again. (See note above about targeted apply). If you are using Kubernetes labels or defined tags for your existing applications, you will need to ensure the new node pools also have the same labels. Refer to the `terraform.tfvars.example` file for the format to specify the labels.
224+
225+
When node pools 3 and 4 are created, they will be created with the newer cluster version of Kubernetes. Since you have already upgrade your cluster to `v1.25.1`, node pools 3 and 4 will be running Kubernetes v1.25.1.
226+
227+
=== Add 1 new autoscaler pool
228+
1. Add 1 new autoscaler pool in your autoscaler_pools e.g. change
229+
+
230+
[source,bash]
231+
----
232+
autoscaler_pools = {
233+
asp_v124 = {
234+
...
235+
}
236+
}
237+
----
238+
to
239+
240+
+
241+
----
242+
autoscaler_pools = {
243+
asp_v124 = {
244+
...
245+
}
246+
asp_v125 = {
247+
...
248+
}
249+
}
250+
----
251+
252+
and run `terraform apply` again.
253+
254+
When node pools asp_v125 is created, it will be created with the newer cluster version of Kubernetes. Since you have already upgrade your cluster to `v1.25.1`, node pool asp_v125 will be running Kubernetes v1.25.1.
255+
256+
=== Drain the node pools
257+
258+
. Set `upgrade_nodepool=true`. This will instruct the OKE cluster that some node pools will be drained.
259+
260+
. Provide the list of node pools to drain. This should usually be only the old node pools. You don't need to upgrade all the node pools at once.
261+
262+
+
263+
----
264+
node_pools_to_drain = [ "np1", "np2", "asp_v124"]
265+
----
266+
267+
. Run `terraform apply` (see note above about targeted apply):
268+
269+
+
270+
----
271+
terraform apply
272+
----
273+
274+
. This will ensure that the all the node pools from 1.24 have been drained.
275+
276+
=== Destroy old node pools
277+
278+
When you are ready, you can now delete the old node pools by removing them from the list of node pools:
279+
280+
+
281+
----
282+
node_pools = {
283+
np3 = {
284+
...
285+
}
286+
np4 = {
287+
...
288+
}
289+
asp_v124 = {}
290+
}
291+
----
292+
293+
. Run terraform again:
294+
295+
+
296+
----
297+
terraform apply
298+
----
299+
300+
=== Upgrade the cluster autoscaler
301+
302+
1. Modify the `cluster-autoscaler.yaml`
303+
304+
2. Change the image version to the corresponding Kubernetes version
305+
306+
3. Change the OCIDs of the node pools to manage to those of np3 and np4.
307+
308+
4. Run Terraform apply again.
309+
310+
. This completes the upgrade process. Now, set ```upgrade_nodepool = false``` to prevent draining from current nodes by mistake.

‎locals.tf‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Copyright 2017, 2021 Oracle Corporation and/or affiliates.
1+
# Copyright (c) 2017, 2022 Oracle Corporation and/or its affiliates.
22
# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl
33

44
locals {
@@ -20,9 +20,9 @@ locals {
2020
operator_private_ip = var.create_operator == true ? module.operator[0].operator_private_ip : var.operator_private_ip !="" ? var.operator_private_ip: ""
2121
operator_instance_principal_group_name = var.create_operator == true ? module.operator[0].operator_instance_principal_group_name : ""
2222

23-
vcn_id = var.create_vcn == true ? module.vcn[0].vcn_id : coalesce(var.vcn_id, try(data.oci_core_vcns.vcns[0].virtual_networks[0].id,""))
24-
ig_route_id = var.create_vcn == true ? module.vcn[0].ig_route_id : coalesce(var.ig_route_table_id, try(data.oci_core_route_tables.ig[0].route_tables[0].id,""))
25-
nat_route_id = var.create_vcn == true ? module.vcn[0].nat_route_id : coalesce(var.nat_route_table_id, try(data.oci_core_route_tables.nat[0].route_tables[0].id,""))
23+
vcn_id = var.create_vcn == true ? module.vcn[0].vcn_id : coalesce(var.vcn_id, try(data.oci_core_vcns.vcns[0].virtual_networks[0].id, ""))
24+
ig_route_id = var.create_vcn == true ? module.vcn[0].ig_route_id : coalesce(var.ig_route_table_id, try(data.oci_core_route_tables.ig[0].route_tables[0].id, ""))
25+
nat_route_id = var.create_vcn == true ? module.vcn[0].nat_route_id : coalesce(var.nat_route_table_id, try(data.oci_core_route_tables.nat[0].route_tables[0].id, ""))
2626

2727
ssh_key_arg = var.ssh_private_key_path == "none" ? "" : " -i ${var.ssh_private_key_path}"
2828
validate_drg_input = var.create_drg && (var.drg_id != null) ? tobool("[ERROR]: create_drg variable can not be true if drg_id is provided.]") : true

‎main.tf‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -291,6 +291,10 @@ module "oke" {
291291
freeform_tags = var.freeform_tags["oke"]
292292
defined_tags = var.defined_tags["oke"]
293293

294+
# cluster autoscaler
295+
enable_cluster_autoscaler = var.enable_cluster_autoscaler
296+
autoscaler_pools = var.autoscaler_pools
297+
294298
depends_on = [
295299
module.network
296300
]
@@ -416,6 +420,12 @@ module "extensions" {
416420
nodepool_upgrade_method = var.nodepool_upgrade_method
417421
node_pools_to_drain = var.node_pools_to_drain
418422

423+
# cluster autoscaler
424+
enable_cluster_autoscaler = var.enable_cluster_autoscaler
425+
autoscaler_pools = var.autoscaler_pools
426+
427+
autoscaling_nodepools = module.oke.autoscaling_nodepools
428+
419429
debug_mode = var.debug_mode
420430

421431
depends_on = [

0 commit comments

Comments
 (0)