Skip to content

Commit dd82051

Browse files
jeana-redhatjeana-redhat
authored
Merge pull request #51892 from xenolinux/haproxy-reload-interval
OSDOCS-3937: Configure HAProxy reload interval
2 parents c4e9523 + 3e0ad3c commit dd82051

File tree

5 files changed

+81
-45
lines changed

5 files changed

+81
-45
lines changed

‎modules/configuring-haproxy-interval.adoc

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
// Module included in the following assemblies:
2+
// * scalability_and_performance/routing-optimization.adoc
3+
// * post_installation_configuration/network-configuration.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="configuring-haproxy-interval_{context}"]
7+
= Configuring HAProxy reload interval
8+
9+
When you update a route or an endpoint associated with a route, {product-title} router updates the configuration for HAProxy. Then, HAProxy reloads the updated configuration for those changes to take effect. When HAProxy reloads, it generates a new process that handles new connections using the updated configuration.
10+
11+
HAProxy keeps the old process running to handle existing connections until those connections are all closed. When old processes have long-lived connections, these processes can accumulate and consume resources.
12+
13+
The default minimum HAProxy reload interval is five seconds. You can configure an Ingress Controller using its `spec.tuningOptions.reloadInterval` field to set a longer minimum reload interval.
14+
15+
[WARNING]
16+
====
17+
Setting a large value for the minimum HAProxy reload interval can cause latency in observing updates to routes and their endpoints. To lessen the risk, avoid setting a value larger than the tolerable latency for updates.
18+
====
19+
20+
.Procedure
21+
22+
* Change the minimum HAProxy reload interval of the default Ingress Controller to 15 seconds by running the following command:
23+
+
24+
[source, terminal]
25+
----
26+
$ oc -n openshift-ingress-operator patch ingresscontrollers/default --type=merge --patch='{"spec":{"tuningOptions":{"reloadInterval":"15s"}}}'
27+
----

‎modules/ingress-liveness-readiness-startup-probes.adoc

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
// Module included in the following assemblies:
2+
// * scalability_and_performance/routing-optimization.adoc
3+
// * post_installation_configuration/network-configuration.adoc
4+
5+
:_content-type: REFERENCE
6+
[id="ingress-liveness-readiness-startup-probes_{context}"]
7+
= Configuring Ingress Controller liveness, readiness, and startup probes
8+
9+
Cluster administrators can configure the timeout values for the kubelet's liveness, readiness, and startup probes for router deployments that are managed by the {product-title} Ingress Controller (router). The liveness and readiness probes of the router use the default timeout value
10+
of 1 second, which is too short for the kubelet's probes to succeed in some scenarios. Probe timeouts can cause unwanted router restarts that interrupt application connections. The ability to set larger timeout values can reduce the risk of unnecessary and unwanted restarts.
11+
12+
You can update the `timeoutSeconds` value on the `livenessProbe`, `readinessProbe`, and `startupProbe` parameters of the router container.
13+
14+
[cols="3a,8a",options="header"]
15+
|===
16+
|Parameter |Description
17+
18+
|`livenessProbe`
19+
|The `livenessProbe` reports to the kubelet whether a pod is dead and needs to be restarted.
20+
21+
|`readinessProbe`
22+
|The `readinessProbe` reports whether a pod is healthy or unhealthy. When the readiness probe reports an unhealthy pod, then the kubelet marks the pod as not ready to accept traffic. Subsequently, the endpoints for that pod are marked as not ready, and this status propogates to the kube-proxy. On cloud platforms with a configured load balancer, the kube-proxy communicates to the cloud load-balancer not to send traffic to the node with that pod.
23+
24+
|`startupProbe`
25+
|The `startupProbe` gives the router pod up to 2 minutes to initialize before the kubelet begins sending the router liveness and readiness probes. This initialization time can prevent routers with many routes or endpoints from prematurely restarting.
26+
|===
27+
28+
29+
[IMPORTANT]
30+
====
31+
The timeout configuration option is an advanced tuning technique that can be used to work around issues. However, these issues should eventually be diagnosed and possibly a support case or https://issues.redhat.com/secure/CreateIssueDetails!init.jspa?pid=12332330&summary=Summary&issuetype=1&priority=10200&versions=12385624[Jira issue] opened for any issues that causes probes to time out.
32+
====
33+
34+
The following example demonstrates how you can directly patch the default router deployment to set a 5-second timeout for the liveness and readiness probes:
35+
36+
37+
[source, terminal]
38+
----
39+
$ oc -n openshift-ingress patch deploy/router-default --type=strategic --patch='{"spec":{"template":{"spec":{"containers":[{"name":"router","livenessProbe":{"timeoutSeconds":5},"readinessProbe":{"timeoutSeconds":5}}]}}}}'
40+
----
41+
42+
.Verification
43+
[source, terminal]
44+
----
45+
$ oc -n openshift-ingress describe deploy/router-default | grep -e Liveness: -e Readiness:
46+
Liveness: http-get http://:1936/healthz delay=0s timeout=5s period=10s #success=1 #failure=3
47+
Readiness: http-get http://:1936/healthz/ready delay=0s timeout=5s period=10s #success=1 #failure=3
48+
----

‎modules/router-performance-optimizations.adoc

Lines changed: 1 addition & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -2,53 +2,10 @@
22
// * scalability_and_performance/routing-optimization.adoc
33
// * post_installation_configuration/network-configuration.adoc
44

5-
:_content-type: Procedure
5+
:_content-type: CONCEPT
66
[id="router-performance-optimizations_{context}"]
77
= Ingress Controller (router) performance optimizations
88

99
{product-title} no longer supports modifying Ingress Controller deployments by setting environment variables such as `ROUTER_THREADS`, `ROUTER_DEFAULT_TUNNEL_TIMEOUT`, `ROUTER_DEFAULT_CLIENT_TIMEOUT`, `ROUTER_DEFAULT_SERVER_TIMEOUT`, and `RELOAD_INTERVAL`.
1010

1111
You can modify the Ingress Controller deployment, but if the Ingress Operator is enabled, the configuration is overwritten.
12-
13-
== Configuring Ingress Controller liveness, readiness, and startup probes
14-
15-
Cluster administrators can configure the timeout values for the kubelet's liveness, readiness, and startup probes for router deployments that are managed by the {product-title} Ingress Controller (router). The liveness and readiness probes of the router use the default timeout value
16-
of 1 second, which is too short for the kubelet's probes to succeed in some scenarios. Probe timeouts can cause unwanted router restarts that interrupt application connections. The ability to set larger timeout values can reduce the risk of unnecessary and unwanted restarts.
17-
18-
You can update the `timeoutSeconds` value on the `livenessProbe`, `readinessProbe`, and `startupProbe` parameters of the router container.
19-
20-
[cols="3a,8a",options="header"]
21-
|===
22-
|Parameter |Description
23-
24-
|`livenessProbe`
25-
|The `livenessProbe` reports to the kubelet whether a pod is dead and needs to be restarted.
26-
27-
|`readinessProbe`
28-
|The `readinessProbe` reports whether a pod is healthy or unhealthy. When the readiness probe reports an unhealthy pod, then the kubelet marks the pod as not ready to accept traffic. Subsequently, the endpoints for that pod are marked as not ready, and this status propogates to the kube-proxy. On cloud platforms with a configured load balancer, the kube-proxy communicates to the cloud load-balancer not to send traffic to the node with that pod.
29-
30-
|`startupProbe`
31-
|The `startupProbe` gives the router pod up to 2 minutes to initialize before the kubelet begins sending the router liveness and readiness probes. This initialization time can prevent routers with many routes or endpoints from prematurely restarting.
32-
|===
33-
34-
35-
[IMPORTANT]
36-
====
37-
The timeout configuration option is an advanced tuning technique that can be used to work around issues. However, these issues should eventually be diagnosed and possibly a support case or https://issues.redhat.com/secure/CreateIssueDetails!init.jspa?pid=12332330&summary=Summary&issuetype=1&priority=10200&versions=12385624[Jira issue] opened for any issues that causes probes to time out.
38-
====
39-
40-
The following example demonstrates how you can directly patch the default router deployment to set a 5-second timeout for the liveness and readiness probes:
41-
42-
43-
[source, terminal]
44-
----
45-
$ oc -n openshift-ingress patch deploy/router-default --type=strategic --patch='{"spec":{"template":{"spec":{"containers":[{"name":"router","livenessProbe":{"timeoutSeconds":5},"readinessProbe":{"timeoutSeconds":5}}]}}}}'
46-
----
47-
48-
.Verification
49-
[source, terminal]
50-
----
51-
$ oc -n openshift-ingress describe deploy/router-default | grep -e Liveness: -e Readiness:
52-
Liveness: http-get http://:1936/healthz delay=0s timeout=5s period=10s #success=1 #failure=3
53-
Readiness: http-get http://:1936/healthz/ready delay=0s timeout=5s period=10s #success=1 #failure=3
54-
----

‎post_installation_configuration/network-configuration.adoc

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,8 @@ The {product-title} HAProxy router scales to optimize performance.
111111
include::modules/baseline-router-performance.adoc[leveloffset=+2]
112112

113113
include::modules/router-performance-optimizations.adoc[leveloffset=+2]
114+
include::modules/ingress-liveness-readiness-startup-probes.adoc[leveloffset=+3]
115+
include::modules/configuring-haproxy-interval.adoc[leveloffset=+3]
114116

115117
[id="post-installation-osp-fips"]
116118
== Post-installation {rh-openstack} network configuration
@@ -121,4 +123,4 @@ include::modules/installation-osp-configuring-api-floating-ip.adoc[leveloffset=+
121123
include::modules/installation-osp-kuryr-port-pools.adoc[leveloffset=+2]
122124
include::modules/installation-osp-kuryr-settings-active.adoc[leveloffset=+2]
123125
include::modules/nw-osp-enabling-ovs-offload.adoc[leveloffset=+2]
124-
include::modules/nw-osp-hardware-offload-attaching-network.adoc[leveloffset=+2]
126+
include::modules/nw-osp-hardware-offload-attaching-network.adoc[leveloffset=+2]

‎scalability_and_performance/routing-optimization.adoc

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,5 @@ include::modules/baseline-router-performance.adoc[leveloffset=+1]
1313
For more information on Ingress sharding, see xref:../networking/ingress-operator.adoc#nw-ingress-sharding-route-labels_configuring-ingress[Configuring Ingress Controller sharding by using route labels] and xref:../networking/ingress-operator.adoc#nw-ingress-sharding-namespace-labels_configuring-ingress[Configuring Ingress Controller sharding by using namespace labels].
1414

1515
include::modules/router-performance-optimizations.adoc[leveloffset=+1]
16+
include::modules/ingress-liveness-readiness-startup-probes.adoc[leveloffset=+2]
17+
include::modules/configuring-haproxy-interval.adoc[leveloffset=+2]

0 commit comments

Comments
 (0)