Skip to content

Commit ae17e46

Browse files
SergeyKanzhelevTim Bannisteraojea
authored
Document behavior of endpoints with the feature EndpointSliceTerminatingCondition (#36791)
* new behavior of endpoints with the feature gate EndpointSliceTerminatingCondition * Update content/en/docs/concepts/workloads/pods/pod-lifecycle.md Co-authored-by: Tim Bannister <[email protected]> * Update content/en/docs/concepts/workloads/pods/pod-lifecycle.md Co-authored-by: Tim Bannister <[email protected]> * Update content/en/docs/concepts/workloads/pods/pod-lifecycle.md Co-authored-by: Antonio Ojea <[email protected]> * fixing feature gate versions --------- Co-authored-by: Tim Bannister <[email protected]> Co-authored-by: Antonio Ojea <[email protected]>
1 parent 3971a57 commit ae17e46

File tree

3 files changed

+276
-5
lines changed

3 files changed

+276
-5
lines changed

‎content/en/docs/concepts/workloads/pods/pod-lifecycle.md

Lines changed: 23 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -296,7 +296,7 @@ Each probe must define exactly one of these four mechanisms:
296296
The target should implement
297297
[gRPC health checks](https://grpc.io/grpc/core/md_doc_health-checking.html).
298298
The diagnostic is considered successful if the `status`
299-
of the response is `SERVING`.
299+
of the response is `SERVING`.
300300
gRPC probes are an alpha feature and are only available if you
301301
enable the `GRPCContainerProbe`
302302
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/).
@@ -465,14 +465,32 @@ An example flow:
465465
The containers in the Pod receive the TERM signal at different times and in an arbitrary
466466
order. If the order of shutdowns matters, consider using a `preStop` hook to synchronize.
467467
{{< /note >}}
468-
1. At the same time as the kubelet is starting graceful shutdown, the control plane removes that
469-
shutting-down Pod from EndpointSlice (and Endpoints) objects where these represent
468+
1. At the same time as the kubelet is starting graceful shutdown of the Pod, the control plane evaluates whether to remove that shutting-down Pod from EndpointSlice (and Endpoints) objects, where those objects represent
470469
a {{< glossary_tooltip term_id="service" text="Service" >}} with a configured
471470
{{< glossary_tooltip text="selector" term_id="selector" >}}.
472471
{{< glossary_tooltip text="ReplicaSets" term_id="replica-set" >}} and other workload resources
473472
no longer treat the shutting-down Pod as a valid, in-service replica. Pods that shut down slowly
474-
cannot continue to serve traffic as load balancers (like the service proxy) remove the Pod from
475-
the list of endpoints as soon as the termination grace period _begins_.
473+
should not continue to serve regular traffic and should start terminating and finish processing open connections.
474+
Some applications need to go beyond finishing open connections and need more graceful termination -
475+
for example: session draining and completion. Any endpoints that represent the terminating pods
476+
are not immediately removed from EndpointSlices,
477+
and a status indicating [terminating state](/docs/concepts/services-networking/endpoint-slices/#conditions)
478+
is exposed from the EndpointSlice API (and the legacy Endpoints API). Terminating
479+
endpoints always have their `ready` status
480+
as `false` (for backward compatibility with versions before 1.26),
481+
so load balancers will not use it for regular traffic.
482+
If traffic draining on terminating pod is needed, the actual readiness can be checked as a condition `serving`.
483+
You can find more details on how to implement connections draining
484+
in the tutorial [Pods And Endpoints Termination Flow](/docs/tutorials/services/pods-and-endpoint-termination-flow/)
485+
486+
{{<note>}}
487+
If you don't have the `EndpointSliceTerminatingCondition` feature gate enabled
488+
in your cluster (the gate is on by default from Kubernetes 1.22, and locked to default in 1.26), then the Kubernetes control
489+
plane removes a Pod from any relevant EndpointSlices as soon as the Pod's
490+
termination grace period _begins_. The behavior above is described when the
491+
feature gate `EndpointSliceTerminatingCondition` is enabled.
492+
{{</note>}}
493+
476494
1. When the grace period expires, the kubelet triggers forcible shutdown. The container runtime sends
477495
`SIGKILL` to any processes still running in any container in the Pod.
478496
The kubelet also cleans up a hidden `pause` container if that container runtime uses one.

‎content/en/docs/tutorials/services/pods-and-endpoint-termination-flow.md

Lines changed: 221 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,221 @@
1+
---
2+
title: Explore Termination Behavior for Pods And Their Endpoints
3+
content_type: tutorial
4+
weight: 60
5+
---
6+
7+
8+
<!-- overview -->
9+
10+
Once you connected your Application with Service following steps
11+
like those outlined in [Connecting Applications with Services](/docs/tutorials/services/connect-applications-service/),
12+
you have a continuously running, replicated application, that is exposed on a network.
13+
This tutorial helps you look at the termination flow for Pods and to explore ways to implement
14+
graceful connection draining.
15+
16+
<!-- body -->
17+
18+
## Termination process for Pods and their endpoints
19+
20+
There are often cases when you need to terminate a Pod - be it for upgrade or scale down.
21+
In order to improve application availability, it may be important to implement
22+
a proper active connections draining. This tutorial explains the flow of
23+
Pod termination in connection with the corresponding endpoint state and removal.
24+
25+
This tutorial explains the flow of Pod termination in connection with the
26+
corresponding endpoint state and removal by using
27+
a simple nginx web server to demonstrate the concept.
28+
29+
<!-- body -->
30+
31+
## Example flow with endpoint termination
32+
33+
The following is the example of the flow described in the
34+
[Termination of Pods](/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination)
35+
document.
36+
37+
Let's say you have a Deployment containing of a single `nginx` replica
38+
(just for demonstration purposes) and a Service:
39+
40+
{{< codenew file="service/pod-with-graceful-termination.yaml" >}}
41+
42+
```yaml
43+
apiVersion: apps/v1
44+
kind: Deployment
45+
metadata:
46+
name: nginx-deployment
47+
labels:
48+
app: nginx
49+
spec:
50+
replicas: 1
51+
selector:
52+
matchLabels:
53+
app: nginx
54+
template:
55+
metadata:
56+
labels:
57+
app: nginx
58+
spec:
59+
terminationGracePeriodSeconds: 120 # extra long grace period
60+
containers:
61+
- name: nginx
62+
image: nginx:latest
63+
ports:
64+
- containerPort: 80
65+
lifecycle:
66+
preStop:
67+
exec:
68+
# Real life termination may take any time up to terminationGracePeriodSeconds.
69+
# In this example - just hang around for at least the duration of terminationGracePeriodSeconds,
70+
# at 120 seconds container will be forcibly terminated.
71+
# Note, all this time nginx will keep processing requests.
72+
command: [
73+
"/bin/sh", "-c", "sleep 180"
74+
]
75+
76+
---
77+
78+
apiVersion: v1
79+
kind: Service
80+
metadata:
81+
name: nginx-service
82+
spec:
83+
selector:
84+
app: nginx
85+
ports:
86+
- protocol: TCP
87+
port: 80
88+
targetPort: 80
89+
```
90+
91+
Once the Pod and Service are running, you can get the name of any associated EndpointSlices:
92+
93+
```shell
94+
kubectl get endpointslice
95+
```
96+
97+
The output is similar to this:
98+
99+
```none
100+
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
101+
nginx-service-6tjbr IPv4 80 10.12.1.199,10.12.1.201 22m
102+
```
103+
104+
You can see its status, and validate that there is one endpoint registered:
105+
106+
```shell
107+
kubectl get endpointslices -o json -l kubernetes.io/service-name=nginx-service
108+
```
109+
110+
The output is similar to this:
111+
112+
```none
113+
{
114+
"addressType": "IPv4",
115+
"apiVersion": "discovery.k8s.io/v1",
116+
"endpoints": [
117+
{
118+
"addresses": [
119+
"10.12.1.201"
120+
],
121+
"conditions": {
122+
"ready": true,
123+
"serving": true,
124+
"terminating": false
125+
```
126+
127+
Now let's terminate the Pod and validate that the Pod is being terminated
128+
respecting the graceful termination period configuration:
129+
130+
```shell
131+
kubectl delete pod nginx-deployment-7768647bf9-b4b9s
132+
```
133+
134+
All pods:
135+
136+
```shell
137+
kubectl get pods
138+
```
139+
140+
The output is similar to this:
141+
142+
```none
143+
NAME READY STATUS RESTARTS AGE
144+
nginx-deployment-7768647bf9-b4b9s 1/1 Terminating 0 4m1s
145+
nginx-deployment-7768647bf9-rkxlw 1/1 Running 0 8s
146+
```
147+
148+
You can see that the new pod got scheduled.
149+
150+
While the new endpoint is being created for the new Pod, the old endpoint is
151+
still around in the terminating state:
152+
153+
```shell
154+
kubectl get endpointslice -o json nginx-service-6tjbr
155+
```
156+
157+
The output is similar to this:
158+
159+
```none
160+
{
161+
"addressType": "IPv4",
162+
"apiVersion": "discovery.k8s.io/v1",
163+
"endpoints": [
164+
{
165+
"addresses": [
166+
"10.12.1.201"
167+
],
168+
"conditions": {
169+
"ready": false,
170+
"serving": true,
171+
"terminating": true
172+
},
173+
"nodeName": "gke-main-default-pool-dca1511c-d17b",
174+
"targetRef": {
175+
"kind": "Pod",
176+
"name": "nginx-deployment-7768647bf9-b4b9s",
177+
"namespace": "default",
178+
"uid": "66fa831c-7eb2-407f-bd2c-f96dfe841478"
179+
},
180+
"zone": "us-central1-c"
181+
},
182+
{
183+
"addresses": [
184+
"10.12.1.202"
185+
],
186+
"conditions": {
187+
"ready": true,
188+
"serving": true,
189+
"terminating": false
190+
},
191+
"nodeName": "gke-main-default-pool-dca1511c-d17b",
192+
"targetRef": {
193+
"kind": "Pod",
194+
"name": "nginx-deployment-7768647bf9-rkxlw",
195+
"namespace": "default",
196+
"uid": "722b1cbe-dcd7-4ed4-8928-4a4d0e2bbe35"
197+
},
198+
"zone": "us-central1-c"
199+
```
200+
201+
This allows applications to communicate their state during termination
202+
and clients (such as load balancers) to implement a connections draining functionality.
203+
These clients may detect terminating endpoints and implement a special logic for them.
204+
205+
In Kubernetes, endpoints that are terminating always have their `ready` status set as as `false`.
206+
This needs to happen for backward
207+
compatibility, so existing load balancers will not use it for regular traffic.
208+
If traffic draining on terminating pod is needed, the actual readiness can be
209+
checked as a condition `serving`.
210+
211+
When Pod is deleted, the old endpoint will also be deleted.
212+
213+
214+
## {{% heading "whatsnext" %}}
215+
216+
217+
* Learn how to [Connect Applications with Services](/docs/tutorials/services/connect-applications-service/)
218+
* Learn more about [Using a Service to Access an Application in a Cluster](/docs/tasks/access-application-cluster/service-access-application-cluster/)
219+
* Learn more about [Connecting a Front End to a Back End Using a Service](/docs/tasks/access-application-cluster/connecting-frontend-backend/)
220+
* Learn more about [Creating an External Load Balancer](/docs/tasks/access-application-cluster/create-external-load-balancer/)
221+

‎content/en/examples/service/pod-with-graceful-termination.yaml

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: nginx-deployment
5+
labels:
6+
app: nginx
7+
spec:
8+
replicas: 1
9+
selector:
10+
matchLabels:
11+
app: nginx
12+
template:
13+
metadata:
14+
labels:
15+
app: nginx
16+
spec:
17+
terminationGracePeriodSeconds: 120 # extra long grace period
18+
containers:
19+
- name: nginx
20+
image: nginx:latest
21+
ports:
22+
- containerPort: 80
23+
lifecycle:
24+
preStop:
25+
exec:
26+
# Real life termination may take any time up to terminationGracePeriodSeconds.
27+
# In this example - just hang around for at least the duration of terminationGracePeriodSeconds,
28+
# at 120 seconds container will be forcibly terminated.
29+
# Note, all this time nginx will keep processing requests.
30+
command: [
31+
"/bin/sh", "-c", "sleep 180"
32+
]

0 commit comments

Comments
 (0)