Skip to content

Commit 6672535

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #32664 from SergeyKanzhelev/grpcProbesBlog
Add blog article about gRPC probes beta
2 parents 23b6380 + a75c1b2 commit 6672535

File tree

1 file changed

+209
-0
lines changed

1 file changed

+209
-0
lines changed

‎content/en/blog/_posts/2022-05-13-grpc-probes-in-beta.md

Lines changed: 209 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,209 @@
1+
---
2+
layout: blog
3+
title: "Kubernetes 1.24: gRPC container probes in beta"
4+
date: 2022-05-13
5+
slug: grpc-probes-now-in-beta
6+
---
7+
8+
**Author**: Sergey Kanzhelev (Google)
9+
10+
11+
With Kubernetes 1.24 the gRPC probes functionality entered beta and is available by default.
12+
Now you can configure startup, liveness, and readiness probes for your gRPC app
13+
without exposing any HTTP endpoint, nor do you need an executable. Kubernetes can natively connect to your your workload via gRPC and query its status.
14+
15+
## Some history
16+
17+
It's useful to let the system managing your workload check that the app is
18+
healthy, has started OK, and whether the app considers itself good to accept
19+
traffic. Before the gRPC support was added, Kubernetes already allowed you to
20+
check for health based on running an executable from inside the container image,
21+
by making an HTTP request, or by checking whether a TCP connection succeeded.
22+
23+
For most apps, those checks are enough. If your app provides a gRPC endpoint
24+
for a health (or readiness) check, it is easy
25+
to repurpose the `exec` probe to use it for gRPC health checking.
26+
In the blog article [Health checking gRPC servers on Kubernetes](/blog/2018/10/01/health-checking-grpc-servers-on-kubernetes/),
27+
Ahmet Alp Balkan described how you can do that — a mechanism that still works today.
28+
29+
There is a commonly used tool to enable this that was [created](https://github.com/grpc-ecosystem/grpc-health-probe/commit/2df4478982e95c9a57d5fe3f555667f4365c025d)
30+
on August 21, 2018, and with
31+
the first release at [Sep 19, 2018](https://github.com/grpc-ecosystem/grpc-health-probe/releases/tag/v0.1.0-alpha.1).
32+
33+
This approach for gRPC apps health checking is very popular. There are [3,626 Dockerfiles](https://github.com/search?l=Dockerfile&q=grpc_health_probe&type=code)
34+
with the `grpc_health_probe` and [6,621 yaml](https://github.com/search?l=YAML&q=grpc_health_probe&type=Code) files that are discovered with the
35+
basic search on GitHub (at the moment of writing). This is good indication of the tool popularity
36+
and the need to support this natively.
37+
38+
Kubernetes v1.23 introduced an alpha-quality implementation of native support for
39+
querying a workload status using gRPC. Because it was an alpha feature,
40+
this was disabled by default for the v1.23 release.
41+
42+
## Using the feature
43+
44+
We built gRPC health checking in similar way with other probes and believe
45+
it will be [easy to use](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe)
46+
if you are familiar with other probe types in Kubernetes.
47+
The natively supported health probe has many benefits over the workaround involving `grpc_health_probe` executable.
48+
49+
With the native gRPC support you don't need to download and carry `10MB` of an additional executable with your image.
50+
Exec probes are generally slower than a gRPC call as they require instantiating a new process to run an executable.
51+
It also makes the checks less sensible for edge cases when the pod is running at maximum resources and has troubles
52+
instantiating new processes.
53+
54+
There are a few limitations though. Since configuring a client certificate for probes is hard,
55+
services that require client authentication are not supported. The built-in probes are also
56+
not checking the server certificates and ignore related problems.
57+
58+
Built-in checks also cannot be configured to ignore certain types of errors
59+
(`grpc_health_probe` returns different exit codes for different errors),
60+
and cannot be "chained" to run the health check on multiple services in a single probe.
61+
62+
But all these limitations are quite standard for gRPC and there are easy workarounds
63+
for those.
64+
65+
## Try it for yourself
66+
67+
### Cluster-level setup
68+
69+
You can try this feature today. To try native gRPC probes, you can spin up a Kubernetes cluster
70+
yourself with the `GRPCContainerProbe` feature gate enabled, there are many [tools available](/docs/tasks/tools/).
71+
72+
Since the feature gate `GRPCContainerProbe` is enabled by default in 1.24,
73+
many vendors will have this functionality working out of the box.
74+
So you may just create an 1.24 cluster on platform of your choice. Some vendors
75+
allow to enable alpha features on 1.23 clusters.
76+
77+
For example, at the moment of writing, you can spin up the test cluster on GKE for a quick test.
78+
Other vendors may also have similar capabilities, especially if you
79+
are reading this blog post long after the Kubernetes 1.24 release.
80+
81+
On GKE use the following command (note, version is `1.23` and `enable-kubernetes-alpha` are specified).
82+
83+
```shell
84+
gcloud container clusters create test-grpc \
85+
--enable-kubernetes-alpha \
86+
--no-enable-autorepair \
87+
--no-enable-autoupgrade \
88+
--release-channel=rapid \
89+
--cluster-version=1.23
90+
```
91+
92+
You will also need to configure `kubectl` to access the cluster:
93+
94+
```shell
95+
gcloud container clusters get-credentials test-grpc
96+
```
97+
98+
### Trying the feature out
99+
100+
Let's create the pod to test how gRPC probes work. For this test we will use the `agnhost` image.
101+
This is a k8s maintained image with that can be used for all sorts of workload testing.
102+
For example, it has a useful [grpc-health-checking](https://github.com/kubernetes/kubernetes/blob/b2c5bd2a278288b5ef19e25bf7413ecb872577a4/test/images/agnhost/README.md#grpc-health-checking) module
103+
that exposes two ports - one is serving health checking service,
104+
another - http port to react on commands `make-serving` and `make-not-serving`.
105+
106+
Here is an example pod definition. It starts the `grpc-health-checking` module,
107+
exposes ports `5000` and `8080`, and configures gRPC readiness probe:
108+
109+
``` yaml
110+
---
111+
apiVersion: v1
112+
kind: Pod
113+
metadata:
114+
name: test-grpc
115+
spec:
116+
containers:
117+
- name: agnhost
118+
image: k8s.gcr.io/e2e-test-images/agnhost:2.35
119+
command: ["/agnhost", "grpc-health-checking"]
120+
ports:
121+
- containerPort: 5000
122+
- containerPort: 8080
123+
readinessProbe:
124+
grpc:
125+
port: 5000
126+
```
127+
128+
If the file called `test.yaml`, you can create the pod and check it's status.
129+
The pod will be in ready state as indicated by the snippet of the output.
130+
131+
```shell
132+
kubectl apply -f test.yaml
133+
kubectl describe test-grpc
134+
```
135+
136+
The output will contain something like this:
137+
138+
```
139+
Conditions:
140+
Type Status
141+
Initialized True
142+
Ready True
143+
ContainersReady True
144+
PodScheduled True
145+
```
146+
147+
Now let's change the health checking endpoint status to NOT_SERVING.
148+
In order to call the http port of the Pod, let's create a port forward:
149+
150+
```shell
151+
kubectl port-forward test-grpc 8080:8080
152+
```
153+
154+
You can `curl` to call the command...
155+
156+
```shell
157+
curl http://localhost:8080/make-not-serving
158+
```
159+
160+
... and in a few seconds the port status will switch to not ready.
161+
162+
```shell
163+
kubectl describe pod test-grpc
164+
```
165+
166+
The output now will have:
167+
168+
```
169+
Conditions:
170+
Type Status
171+
Initialized True
172+
Ready False
173+
ContainersReady False
174+
PodScheduled True
175+
176+
...
177+
178+
Warning Unhealthy 2s (x6 over 42s) kubelet Readiness probe failed: service unhealthy (responded with "NOT_SERVING")
179+
```
180+
181+
Once it is switched back, in about one second the Pod will get back to ready status:
182+
183+
``` bsh
184+
curl http://localhost:8080/make-serving
185+
kubectl describe test-grpc
186+
```
187+
188+
The output indicates that the Pod went back to being `Ready`:
189+
190+
```
191+
Conditions:
192+
Type Status
193+
Initialized True
194+
Ready True
195+
ContainersReady True
196+
PodScheduled True
197+
```
198+
199+
This new built-in gRPC health probing on Kubernetes makes implementing a health-check via gRPC
200+
much easier than the older approach that relied on using a separate `exec` probe. Read through
201+
the official
202+
[documentation](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe)
203+
to learn more and provide feedback before the feature will be promoted to GA.
204+
205+
## Summary
206+
207+
Kubernetes is a popular workload orchestration platform and we add features based on feedback and demand.
208+
Features like gRPC probes support is a minor improvement that will make life of many app developers
209+
easier and apps more resilient. Try it today and give feedback, before the feature went into GA.

0 commit comments

Comments
 (0)