Skip to content

Commit e817ad0

Browse files
jldohmannjldohmann
authored
Merge pull request #49786 from jldohmann/OSDOCS-3888
OSDOCS-3888: add ingress autoscaling feature
2 parents 6f3dd3d + accf220 commit e817ad0

File tree

2 files changed

+259
-1
lines changed

2 files changed

+259
-1
lines changed

‎modules/nw-autoscaling-ingress-controller.adoc

Lines changed: 244 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,244 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * networking/ingress-controller-configuration.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="nw-autoscaling-ingress-controller_{context}"]
7+
= Autoscaling an Ingress Controller
8+
9+
Automatically scale an Ingress Controller to dynamically meet routing performance or availability requirements such as the requirement to increase throughput. The following procedure provides an example for scaling up the default `IngressController`.
10+
11+
.Prerequisites
12+
. You have the OpenShift CLI (`oc`) installed.
13+
. You have access to an {product-title} cluster as a user with the `cluster-admin` role.
14+
. You have the Custom Metrics Autoscaler Operator installed.
15+
16+
.Procedure
17+
. Create a project in the `openshift-ingress-operator` namespace by running the following command:
18+
+
19+
[source,terminal]
20+
----
21+
$ oc project openshift-ingress-operator
22+
----
23+
24+
. Enable OpenShift monitoring for user-defined projects by creating and applying a config map:
25+
26+
.. Create a new `ConfigMap` object, `cluster-monitoring-config.yaml`:
27+
+
28+
.cluster-monitoring-config.yaml
29+
[source,yaml]
30+
----
31+
apiVersion: v1
32+
kind: ConfigMap
33+
metadata:
34+
name: cluster-monitoring-config
35+
namespace: openshift-monitoring
36+
data:
37+
config.yaml: |
38+
enableUserWorkload: true <1>
39+
----
40+
+
41+
<1> When set to `true`, the `enableUserWorkload` parameter enables monitoring for user-defined projects in a cluster.
42+
43+
.. Apply the config map by running the following command:
44+
+
45+
[source,terminal]
46+
----
47+
$ oc apply -f cluster-monitoring-config.yaml
48+
----
49+
50+
. Create a service account to authenticate with Thanos by running the following command:
51+
+
52+
[source,terminal]
53+
----
54+
$ oc create serviceaccount thanos && oc describe serviceaccount thanos
55+
----
56+
+
57+
.Example output
58+
[source,terminal]
59+
----
60+
Name: thanos
61+
Namespace: openshift-ingress-operator
62+
Labels: <none>
63+
Annotations: <none>
64+
Image pull secrets: thanos-dockercfg-b4l9s
65+
Mountable secrets: thanos-dockercfg-b4l9s
66+
Tokens: thanos-token-c422q
67+
Events: <none>
68+
----
69+
70+
. Define a `TriggerAuthentication` object within the `openshift-ingress-operator` namespace using the service account's token.
71+
72+
.. Define the variable `secret` that contains the secret by running the following command:
73+
+
74+
[source,terminal]
75+
----
76+
$ secret=$(oc get secret | grep thanos-token | head -n 1 | awk '{ print $1 }')
77+
----
78+
79+
.. Create the `TriggerAuthentication` object and pass the value of the `secret` variable to the `TOKEN` parameter:
80+
+
81+
[source,terminal]
82+
----
83+
$ oc process TOKEN="$secret" -f - <<EOF | oc apply -f -
84+
apiVersion: template.openshift.io/v1
85+
kind: Template
86+
parameters:
87+
- name: TOKEN
88+
objects:
89+
- apiVersion: keda.sh/v1alpha1
90+
kind: TriggerAuthentication
91+
metadata:
92+
name: keda-trigger-auth-prometheus
93+
spec:
94+
secretTargetRef:
95+
- parameter: bearerToken
96+
name: \${TOKEN}
97+
key: token
98+
- parameter: ca
99+
name: \${TOKEN}
100+
key: ca.crt
101+
EOF
102+
----
103+
104+
. Create and apply a role for reading metrics from Thanos:
105+
106+
.. Create a new role, `thanos-metrics-reader.yaml`, that reads metrics from pods and nodes:
107+
+
108+
.thanos-metrics-reader.yaml
109+
[source,yaml]
110+
----
111+
apiVersion: rbac.authorization.k8s.io/v1
112+
kind: Role
113+
metadata:
114+
name: thanos-metrics-reader
115+
rules:
116+
- apiGroups:
117+
- ""
118+
resources:
119+
- pods
120+
- nodes
121+
verbs:
122+
- get
123+
- apiGroups:
124+
- metrics.k8s.io
125+
resources:
126+
- pods
127+
- nodes
128+
verbs:
129+
- get
130+
- list
131+
- watch
132+
- apiGroups:
133+
- ""
134+
resources:
135+
- namespaces
136+
verbs:
137+
- get
138+
----
139+
140+
.. Apply the new role by running the following command:
141+
+
142+
[source,terminal]
143+
----
144+
$ oc apply -f thanos-metrics-reader.yaml
145+
----
146+
147+
. Add the new role to the service account by entering the following commands:
148+
+
149+
[source,terminal]
150+
----
151+
$ oc adm policy add-role-to-user thanos-metrics-reader -z thanos --role=namespace=openshift-ingress-operator
152+
----
153+
+
154+
[source,terminal]
155+
----
156+
$ oc adm policy -n openshift-ingress-operator add-cluster-role-to-user cluster-monitoring-view -z thanos
157+
----
158+
+
159+
[NOTE]
160+
====
161+
The argument `add-cluster-role-to-user` is only required if you use cross-namespace queries. The following step uses a query from the `kube-metrics` namespace which requires this argument.
162+
====
163+
164+
. Create a new `ScaledObject` YAML file, `ingress-autoscaler.yaml`, that targets the default Ingress Controller deployment:
165+
+
166+
.Example `ScaledObject` definition
167+
[source,yaml]
168+
----
169+
apiVersion: keda.sh/v1alpha1
170+
kind: ScaledObject
171+
metadata:
172+
name: ingress-scaler
173+
spec:
174+
scaleTargetRef: <1>
175+
apiVersion: operator.openshift.io/v1
176+
kind: IngressController
177+
name: default
178+
envSourceContainerName: ingress-operator
179+
minReplicaCount: 1
180+
maxReplicaCount: 20 <2>
181+
cooldownPeriod: 1
182+
pollingInterval: 1
183+
triggers:
184+
- type: prometheus
185+
metricType: AverageValue
186+
metadata:
187+
serverAddress: https://<example-cluster>:9091 <3>
188+
namespace: openshift-ingress-operator <4>
189+
metricName: 'kube-node-role'
190+
threshold: '1'
191+
query: 'sum(kube_node_role{role="worker",service="kube-state-metrics"})' <5>
192+
authModes: "bearer"
193+
authenticationRef:
194+
name: keda-trigger-auth-prometheus
195+
----
196+
<1> The custom resource that you are targeting. In this case, the Ingress Controller.
197+
<2> Optional: The maximum number of replicas. If you omit this field, the default maximum is set to 50 replicas.
198+
<3> The cluster address and port.
199+
<4> The Ingress Operator namespace.
200+
<5> This expression evaluates to however many worker nodes are present in the deployed cluster.
201+
+
202+
[IMPORTANT]
203+
====
204+
If you are using cross-namespace queries, you must target port 9091 and not port 9092 in the `serverAddress` field. You also must have elevated privileges to read metrics from this port.
205+
====
206+
207+
. Apply the custom resource definition by running the following command:
208+
+
209+
[source,terminal]
210+
----
211+
$ oc apply -f ingress-autoscaler.yaml
212+
----
213+
214+
.Verification
215+
* Verify that the default Ingress Controller is scaled out to match the value returned by the `kube-state-metrics` query by running the following commands:
216+
217+
** Use the `grep` command to search the Ingress Controller YAML file for replicas:
218+
+
219+
[source,terminal]
220+
----
221+
$ oc get ingresscontroller/default -o yaml | grep replicas:
222+
----
223+
+
224+
.Example output
225+
[source,terminal]
226+
----
227+
replicas: 3
228+
----
229+
230+
** Get the pods in the `openshift-ingress` project:
231+
+
232+
[source,terminal]
233+
----
234+
$ oc get pods -n openshift-ingress
235+
----
236+
+
237+
.Example output
238+
[source,terminal]
239+
----
240+
NAME READY STATUS RESTARTS AGE
241+
router-default-7b5df44ff-l9pmm 2/2 Running 0 17h
242+
router-default-7b5df44ff-s5sl5 2/2 Running 0 3d22h
243+
router-default-7b5df44ff-wwsth 2/2 Running 0 66s
244+
----

‎networking/ingress-operator.adoc

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,20 @@ include::modules/nw-ingress-setting-a-custom-default-certificate.adoc[leveloffse
4747

4848
include::modules/nw-ingress-custom-default-certificate-remove.adoc[leveloffset=+2]
4949

50+
include::modules/nw-autoscaling-ingress-controller.adoc[leveloffset=+2]
51+
52+
[role="_additional-resources"]
53+
.Additional resources
54+
* xref:../monitoring/enabling-monitoring-for-user-defined-projects.adoc#enabling-monitoring-for-user-defined-projects_enabling-monitoring-for-user-defined-projects[Enabling monitoring for user-defined projects]
55+
56+
* xref:../nodes/pods/nodes-pods-autoscaling-custom.adoc#nodes-pods-autoscaling-custom-install_nodes-pods-autoscaling-custom[Installing the custom metrics autoscaler]
57+
58+
* xref:../nodes/pods/nodes-pods-autoscaling-custom.adoc#nodes-pods-autoscaling-custom-trigger-auth_nodes-pods-autoscaling-custom[Understanding custom metrics autoscaler trigger authentications]
59+
60+
* xref:../nodes/pods/nodes-pods-autoscaling-custom.adoc#nodes-pods-autoscaling-custom-prometheus_nodes-pods-autoscaling-custom[Configuring the custom metrics autoscaler to use {product-title} monitoring]
61+
62+
* xref:../nodes/pods/nodes-pods-autoscaling-custom.adoc#nodes-pods-autoscaling-custom-adding_nodes-pods-autoscaling-custom[Understanding how to add custom metrics autoscalers]
63+
5064
include::modules/nw-scaling-ingress-controller.adoc[leveloffset=+2]
5165

5266
include::modules/nw-configure-ingress-access-logging.adoc[leveloffset=+2]
@@ -98,4 +112,4 @@ ifndef::openshift-rosa,openshift-dedicated[]
98112
== Additional resources
99113

100114
* xref:../networking/configuring-a-custom-pki.adoc#configuring-a-custom-pki[Configuring a custom PKI]
101-
endif::openshift-rosa,openshift-dedicated[]
115+
endif::openshift-rosa,openshift-dedicated[]

0 commit comments

Comments
 (0)