Skip to content

Commit 8e04c07

Browse files
SkarlsoSkarlso
committed
Added lots more documentation on how to use scaling
1 parent 406dda9 commit 8e04c07

File tree

3 files changed

+172
-22
lines changed

3 files changed

+172
-22
lines changed

‎config/rbac/role.yaml

Lines changed: 0 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -295,23 +295,3 @@ rules:
295295
- get
296296
- patch
297297
- update
298-
- apiGroups:
299-
- infrastructure.cluster.x-k8s.io
300-
resources:
301-
- kubemarkmachinetemplates
302-
verbs:
303-
- create
304-
- delete
305-
- get
306-
- list
307-
- patch
308-
- update
309-
- watch
310-
- apiGroups:
311-
- infrastructure.cluster.x-k8s.io
312-
resources:
313-
- kubemarkmachinetemplates/status
314-
verbs:
315-
- get
316-
- patch
317-
- update

‎controllers/awsmachine_controller.go

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -137,8 +137,6 @@ func (r *AWSMachineReconciler) getObjectStoreService(scope scope.S3Scope) servic
137137
// +kubebuilder:rbac:groups="",resources=secrets;,verbs=get;list;watch
138138
// +kubebuilder:rbac:groups="",resources=namespaces,verbs=get;list;watch
139139
// +kubebuilder:rbac:groups="",resources=events,verbs=get;list;watch;create;update;patch
140-
// +kubebuilder:rbac:groups=infrastructure.cluster.x-k8s.io,resources=kubemarkmachinetemplates,verbs=get;list;watch;create;update;patch;delete
141-
// +kubebuilder:rbac:groups=infrastructure.cluster.x-k8s.io,resources=kubemarkmachinetemplates/status,verbs=get;patch;update
142140

143141
func (r *AWSMachineReconciler) Reconcile(ctx context.Context, req ctrl.Request) (_ ctrl.Result, reterr error) {
144142
log := ctrl.LoggerFrom(ctx)

‎docs/book/src/topics/scale-from-0.md

Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,12 @@
22

33
With the changes introduce into `cluster-api` described in [this](https://github.com/kubernetes-sigs/cluster-api/blob/main/docs/proposals/20210310-opt-in-autoscaling-from-zero.md#upgrade-strategy) proposal, a user can now opt in to scaling nodes from 0.
44

5+
This entails a number of things which I will describe in detail.
6+
7+
The following actions need to be taken to enabled cluster autoscaling:
8+
9+
## Set Capacity field
10+
511
To do that, simply define some values to the new field called `capacity` in the `AWSMachineTemplate` like this:
612

713
```yaml
@@ -25,3 +31,169 @@ status:
2531
2632
To read more about what values are available, consult the proposal. These values can be overridden by selected annotations
2733
on the MachineTemplate.
34+
35+
## Add two necessary annotations to MachineDeployment
36+
37+
There are two annotations which need to be applied to the MachineDeployment like this:
38+
39+
```yaml
40+
apiVersion: cluster.x-k8s.io/v1beta1
41+
kind: MachineDeployment
42+
metadata:
43+
name: "managed-cluster-md-0"
44+
annotations:
45+
cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
46+
cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
47+
```
48+
49+
These are necessary for the autoscaler to be able to pick up the deployment and scale it. Read more about these [here](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/clusterapi/README.md#enabling-autoscaling).
50+
51+
## Install and start cluster-autoscaler
52+
53+
Now comes the tricky part. In order for this to work, you need the cluster-autoscaler binary located [here](https://github.com/kubernetes/autoscaler).
54+
You have to options. Use Helm to install autoscaler, or use the command line ( which is faster in if you are testing ).
55+
56+
In either cases, you need the following options:
57+
- namespace
58+
- cloud-provider
59+
- scale-down-delay-after-add
60+
- scale-down-delay-after-delete
61+
- scale-down-delay-after-failure
62+
- scale-down-unneeded-time
63+
- expander
64+
- kubeconfig
65+
- cloud-config
66+
67+
These last two values are crucial for the autoscaler to work. `cloud-config` is the kubeconfig of the management cluster.
68+
If you are using a service account to access it, you also have an option to define that. Read more about it on the
69+
autoscaler's repository. The second one is the workload cluster. It needs both because the MachineDeployment is in the
70+
control-plane while the actual node and pods are in the workload cluster.
71+
72+
Therefor, you have to install cluster-autoscaler into the _control-plane_ cluster.
73+
74+
I have a handy script to launch autoscaler which looks like this:
75+
76+
```bash
77+
#!/bin/sh
78+
# usage: start-autoscaler management.kubeconfig workload.kubeconfig
79+
cluster-autoscaler \
80+
--cloud-provider=clusterapi \
81+
--v=4 \
82+
--namespace=default \
83+
--max-nodes-total=30 \
84+
--scale-down-delay-after-add=10s \
85+
--scale-down-delay-after-delete=10s \
86+
--scale-down-delay-after-failure=10s \
87+
--scale-down-unneeded-time=23s \
88+
--max-node-provision-time=2m \
89+
--balance-similar-node-groups \
90+
--expander=random \
91+
--kubeconfig=$2 \
92+
--cloud-config=$1
93+
```
94+
95+
Courtesy of [@elmiko](https://github.com/elmiko).
96+
97+
The Helm equivalent is a bit more complex and either needs to mount in the kubeconfig from somewhere or be pointed to it.
98+
99+
## Permissions
100+
101+
This depends on your scenario. Read about it more [here](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler).
102+
Since this is Cluster API Provider AWS, you would need to look for the AWS provider settings [here](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md).
103+
104+
Further, the service account associated with cluster-autoscaler requires permissions to access `get` and `list` the
105+
Cluster API machine template infrastructure objects.
106+
107+
## Putting it together
108+
109+
The whole yaml looks like this:
110+
111+
```yaml
112+
---
113+
apiVersion: cluster.x-k8s.io/v1beta1
114+
kind: Cluster
115+
metadata:
116+
name: "managed-cluster"
117+
spec:
118+
infrastructureRef:
119+
kind: AWSManagedControlPlane
120+
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
121+
name: "managed-cluster-control-plane"
122+
controlPlaneRef:
123+
kind: AWSManagedControlPlane
124+
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
125+
name: "managed-cluster-control-plane"
126+
---
127+
kind: AWSManagedControlPlane
128+
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
129+
metadata:
130+
name: "managed-cluster-control-plane"
131+
spec:
132+
region: "eu-central-1"
133+
version: "v1.22.0"
134+
---
135+
apiVersion: cluster.x-k8s.io/v1beta1
136+
kind: MachineDeployment
137+
metadata:
138+
name: "managed-cluster-md-0"
139+
annotations:
140+
cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
141+
cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
142+
spec:
143+
clusterName: "managed-cluster"
144+
replicas: 0 # _NOTE_ that we set the initial replicas size to *ZERO*.
145+
selector:
146+
matchLabels:
147+
template:
148+
spec:
149+
clusterName: "managed-cluster"
150+
version: "v1.22.0"
151+
bootstrap:
152+
configRef:
153+
name: "managed-cluster-md-0"
154+
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
155+
kind: EKSConfigTemplate
156+
infrastructureRef:
157+
name: "managed-cluster-md-0"
158+
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
159+
kind: AWSMachineTemplate
160+
---
161+
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
162+
kind: AWSMachineTemplate
163+
metadata:
164+
name: "managed-cluster-md-0"
165+
spec:
166+
template:
167+
spec:
168+
instanceType: "t3.small"
169+
iamInstanceProfile: "nodes.cluster-api-provider-aws.sigs.k8s.io"
170+
status:
171+
capacity:
172+
memory: "500m"
173+
cpu: "1"
174+
---
175+
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
176+
kind: EKSConfigTemplate
177+
metadata:
178+
name: "managed-cluster-md-0"
179+
spec:
180+
template: {}
181+
```
182+
183+
## When will it not scale?
184+
185+
There is a document describing under what circumstances it won't be able to scale located [here](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node). Read this carefully.
186+
187+
It has some ramifications when scaling back down to 0. Which will only work if all pods are removed from the node and
188+
the node cannot schedule even the aws-node and kube-proxy pods. There is this tiny manual step of cordoning off the last
189+
node in order to scale back down to 0.
190+
191+
## Conclusion
192+
193+
Once the cluster-autoscaler is running, you will start seeing nodes pop-in as soon as there is some load on the cluster.
194+
To test it, simply create and inflate a deployment like this:
195+
196+
```bash
197+
kubectl create deployment inflate --image=public.ecr.aws/eks-distro/kubernetes/pause:3.2 --kubeconfig workload.kubeconfig
198+
kubectl scale deployment inflate --replicas=50 --kubeconfig workload.kubeconfig
199+
```

0 commit comments

Comments
 (0)