Skip to content

Commit 1d1c96c

Browse files
author
Lars Ekman
committed
Added blog post kpng-specialized-proxiers
1 parent 383dbc2 commit 1d1c96c

File tree

1 file changed

+241
-0
lines changed

1 file changed

+241
-0
lines changed

‎content/en/blog/_posts/2021-10-18-kpng-specialized-proxiers.md

Lines changed: 241 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,241 @@
1+
---
2+
layout: blog
3+
title: "Use KPNG to Write Specialized kube-proxiers"
4+
date: 2021-10-18
5+
slug: use-kpng-to-write-specialized-kube-proxiers
6+
---
7+
8+
Authors: Lars Ekman, Ericsson
9+
10+
The post will show you how to create a specialized service kube-proxy
11+
style network proxier using Kubernetes Proxy NG
12+
[kpng](https://github.com/kubernetes-sigs/kpng) without interfering
13+
with the existing kube-proxy. The kpng project aims at renewing the
14+
the default Kubernetes Service implementation, the "kube-proxy". An
15+
important feature of kpng is that it can be used as a library to
16+
create proxiers outside K8s. While this is useful for CNI-plugins that
17+
replaces the kube-proxy it also opens the possibility for anyone to
18+
create a proxier for a special purpose.
19+
20+
21+
## Define a service that uses a specialized proxier
22+
23+
```
24+
apiVersion: v1
25+
kind: Service
26+
metadata:
27+
name: kpng-example
28+
labels:
29+
service.kubernetes.io/service-proxy-name: kpng-example
30+
spec:
31+
clusterIP: None
32+
ipFamilyPolicy: RequireDualStack
33+
externalIPs:
34+
- 10.0.0.55
35+
- 1000::55
36+
selector:
37+
app: kpng-alpine
38+
ports:
39+
- port: 6000
40+
```
41+
42+
If the `service.kubernetes.io/service-proxy-name` label is defined the
43+
`kube-proxy` will ignore the service. A custom controller can watch
44+
services with the label set to it's own name, "kpng-example" in
45+
this example, and setup specialized load-balancing.
46+
47+
The `service.kubernetes.io/service-proxy-name` label is [not
48+
new](https://kubernetes.io/docs/reference/labels-annotations-taints/#servicekubernetesioservice-proxy-name),
49+
but so far is has been quite hard to write a specialized proxier.
50+
51+
The common use for a specialized proxier is assumed to be handling
52+
external traffic for some use-case not supported by K8s. In that
53+
case `ClusterIP` is not needed, so we use a "headless" service in this
54+
example.
55+
56+
57+
## Specialized proxier using kpng
58+
59+
A [kpng](https://github.com/kubernetes-sigs/kpng) based proxier
60+
consists of the `kpng` controller handling all the K8s api related
61+
functions, and a "backend" implementing the load-balancing. The
62+
backend can be linked with the `kpng` controller binary or be a
63+
separate program communicating with the controller using gRPC.
64+
65+
```
66+
kpng kube --service-proxy-name=kpng-example to-api
67+
```
68+
69+
This starts the `kpng` controller and tell it to watch only services
70+
with the "kpng-example" service proxy name. The "to-api" parameter
71+
will open a gRPC server for backends.
72+
73+
You can test this yourself outside your cluster. Please see the example
74+
below.
75+
76+
Now we start a backend that simply prints the updates from the
77+
controller.
78+
79+
```
80+
$ kubectl apply -f kpng-example.yaml
81+
$ kpng-json | jq # (this is the backend)
82+
{
83+
"Service": {
84+
"Namespace": "default",
85+
"Name": "kpng-example",
86+
"Type": "ClusterIP",
87+
"IPs": {
88+
"ClusterIPs": {},
89+
"ExternalIPs": {
90+
"V4": [
91+
"10.0.0.55"
92+
],
93+
"V6": [
94+
"1000::55"
95+
]
96+
},
97+
"Headless": true
98+
},
99+
"Ports": [
100+
{
101+
"Protocol": 1,
102+
"Port": 6000,
103+
"TargetPort": 6000
104+
}
105+
]
106+
},
107+
"Endpoints": [
108+
{
109+
"IPs": {
110+
"V6": [
111+
"1100::202"
112+
]
113+
},
114+
"Local": true
115+
},
116+
{
117+
"IPs": {
118+
"V4": [
119+
"11.0.2.2"
120+
]
121+
},
122+
"Local": true
123+
},
124+
{
125+
"IPs": {
126+
"V4": [
127+
"11.0.1.2"
128+
]
129+
}
130+
},
131+
{
132+
"IPs": {
133+
"V6": [
134+
"1100::102"
135+
]
136+
}
137+
}
138+
]
139+
}
140+
```
141+
142+
A real backend would use some mechanism to load-balance traffic from
143+
the external IPs to the endpoints.
144+
145+
146+
147+
## Writing a backend
148+
149+
The `kpng-json` backend looks like this:
150+
151+
```go
152+
package main
153+
import (
154+
"os"
155+
"encoding/json"
156+
"sigs.k8s.io/kpng/client"
157+
)
158+
func main() {
159+
client.Run(jsonPrint)
160+
}
161+
func jsonPrint(items []*client.ServiceEndpoints) {
162+
enc := json.NewEncoder(os.Stdout)
163+
for _, item := range items {
164+
_ = enc.Encode(item)
165+
}
166+
}
167+
```
168+
169+
(yes, that is the entire program)
170+
171+
A real backend would of course be much more complex, but this
172+
illustrates how `kpng` let you focus on load-balancing.
173+
174+
You can have several backends connected to a `kpng` controller, so
175+
during development or debug it can be useful to let something like the
176+
`kpng-json` backend run in parallel with your real backend.
177+
178+
179+
## Example
180+
181+
182+
The complete example can be found [here](https://github.com/kubernetes-sigs/kpng/tree/master/examples/pipe-exec).
183+
184+
As an example we implement an "all-ip" backend. It direct all traffic
185+
for the externalIPs to a local endpoint, regardless of ports and upper
186+
layer protocols. There is a
187+
[KEP](https://github.com/kubernetes/enhancements/pull/2611) for this
188+
function and this example is a much simplified version.
189+
190+
To direct all traffic from an external address to a local POD [only
191+
one iptables rule is
192+
needed](https://github.com/kubernetes/enhancements/pull/2611#issuecomment-895061013),
193+
for instance;
194+
195+
```
196+
ip6tables -t nat -A PREROUTING -d 1000::55/128 -j DNAT --to-destination 1100::202
197+
```
198+
199+
As you can see the addresses are in the call to the backend and all it
200+
have to do is:
201+
202+
* Extract the addresses with `Local: true`
203+
* Setup iptables rules for the `ExternalIPs`
204+
205+
A script doing that may look like:
206+
207+
```
208+
xip=$(cat /tmp/out | jq -r .Service.IPs.ExternalIPs.V6[0])
209+
podip=$(cat /tmp/out | jq -r '.Endpoints[]|select(.Local == true)|select(.IPs.V6 != null)|.IPs.V6[0]')
210+
ip6tables -t nat -A PREROUTING -d $xip/128 -j DNAT --to-destination $podip
211+
```
212+
213+
Assuming the json output above is stored in `/tmp/out` ([jq](https://stedolan.github.io/jq/) is an *awesome* program!).
214+
215+
216+
As this is an example we make it really simple for ourselves by using
217+
a minor variation of the `kpng-json` backend above. Instead of just
218+
printing, a program is called and the json output is passed as `stdin`
219+
to that program. The backend can be tested stand-alone:
220+
221+
```
222+
CALLOUT=jq kpng-callout
223+
```
224+
225+
Where `jq` can be replaced with your own program or script. A script
226+
may look like the example above. For more info and the complete
227+
example please see [https://github.com/kubernetes-sigs/kpng/tree/master/examples/pipe-exec](https://github.com/kubernetes-sigs/kpng/tree/master/examples/pipe-exec).
228+
229+
230+
## Summary
231+
232+
While [kpng](https://github.com/kubernetes-sigs/kpng) is in early
233+
stage of development this post wants to show how you may build your
234+
own specialized K8s proxiers in the future. The only thing your
235+
applications need to do is to add the
236+
`service.kubernetes.io/service-proxy-name` label in the service
237+
manifest.
238+
239+
It is a tedious process to get new features into the `kube-proxy` and
240+
it is not unlikely that they will be rejected, so to write a
241+
specialized proxier may be the only option.

0 commit comments

Comments
 (0)