Skip to content

Commit 1edee93

Browse files
roffemurali-reddy
roffe
authored and
murali-reddy
committed
Documentation how to install kube-router on generic clusters (#281)
* generic deployment work * better docs * more docs * prettify * more docs * more docs * more docs * more docs * tidy up * fixed mount paths * fixed manifests * better clarification of scope
1 parent e51dfbf commit 1edee93

File tree

5 files changed

+639
-0
lines changed

5 files changed

+639
-0
lines changed

‎Documentation/README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,9 @@ Please see the [steps](https://github.com/cloudnativelabs/kube-router/tree/maste
8787
#### kubeadm
8888
Please see the [steps](https://github.com/cloudnativelabs/kube-router/blob/master/Documentation/kubeadm.md) to deploy Kubernetes cluster with Kube-router using [Kubeadm](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/)
8989

90+
#### generic
91+
Please see the [steps](https://github.com/cloudnativelabs/kube-router/blob/master/Documentation/generic.md) to deploy kube-router on generic installed clusters
92+
9093
### deployment
9194

9295
Depending on what functionality of kube-router you want to use, multiple deployment options are possible. You can use the flags `--run-firewall`, `--run-router`, `--run-service-proxy` to selectively enable only required functionality of kube-router.

‎Documentation/generic.md‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# Kube-router on generic cluster
2+
3+
This guide assumes you already have bootstrapped the initial pieces for a Kubernets cluster and is about to switch or setup service & container networking provider
4+
5+
Kube-router relies on kube-controller-manager to allocate pod CIDR for the nodes
6+
7+
Kube-router provides pod networking, network policy and high perfoming IPVS/LVS based service proxy. Depending on you choose to use kube-router for service proxy you have two options listed below the prerequisites
8+
9+
## Prerequisites
10+
11+
kube-router can work as your whole network stack in Kubernetes on-prem & bare metall and works without any cloudproviders
12+
13+
below is the needed configuration to run kube-router in such environments
14+
15+
### Kubelet on each node
16+
17+
kube-router assumes each Kubelet is using `/etc/cni/net.d` as cni conf dir & network plugin `cni`
18+
19+
- --cni-conf-dir=/etc/cni/net.d
20+
- --network-plugin=cni
21+
22+
If you have been using a other CNI providerssuch as weave-net, calico or flannel you will have to remove old configurations from /etc/cni/net.d on each node
23+
24+
## __Switching CNI provider on a running cluster will require you to delete all the running pods and let them recreate and get new adresses assigned from the Kubenet IPAM__
25+
26+
### Kube controller-manager
27+
28+
The following options needs to be set on the controller-manager
29+
30+
```text
31+
--cluster-cidr=${POD_NETWORK} # for example 10.32.0.0/12
32+
--service-cluster-ip-range=${SERVICE_IP_RANGE} # for example 10.50.0.0/22
33+
```
34+
35+
## Kube-router providing pod networking and network policy
36+
37+
Don't forgett to adjust values for Cluster CIDR (pod range) & apiserver adress (must be reachable directly from host networking)
38+
39+
```sh
40+
CLUSTERCIDR=10.32.0.0/12 \
41+
APISERVER=https://cluster01.int.domain.com:6443 \
42+
sh -c 'curl https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/generic-kuberouter.yaml -o - | \
43+
sed -e "s;%APISERVER%;$APISERVER;g" -e "s;%CLUSTERCIDR%;$CLUSTERCIDR;g"' | \
44+
kubectl apply -f -
45+
```
46+
47+
## Kube-router providing service proxy, firewall and pod networking
48+
49+
Don't forgett to adjust values for Cluster CIDR (pod range) & apiserver adress (must be reachable directly from host networking)
50+
51+
```sh
52+
CLUSTERCIDR=10.32.0.0/12 \
53+
APISERVER=https://cluster01.int.domain.com:6443 \
54+
sh -c 'curl https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/generic-kuberouter-all-features.yaml -o - | \
55+
sed -e "s;%APISERVER%;$APISERVER;g" -e "s;%CLUSTERCIDR%;$CLUSTERCIDR;g"' | \
56+
kubectl apply -f -
57+
```
58+
59+
Now since kube-router provides service proxy as well. Run below commands to remove kube-proxy and cleanup any iptables configuration it may have done
60+
61+
Depending on if or how you installed kube-proxy previously these instructions will differ and have to be ran on every node where kube-proxy has run
62+
63+
```sh
64+
kubectl -n kube-system delete ds kube-proxy
65+
docker run --privileged --net=host gcr.io/google_containers/kube-proxy-amd64:v1.7.3 kube-proxy --cleanup-iptables
66+
```

‎daemonset/generic-kuberouter-all-features-advertise-routes.yaml‎

Lines changed: 193 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,193 @@
1+
apiVersion: v1
2+
kind: ConfigMap
3+
metadata:
4+
name: kube-router-cfg
5+
namespace: kube-system
6+
labels:
7+
tier: node
8+
k8s-app: kube-router
9+
data:
10+
cni-conf.json: |
11+
{
12+
"name":"kubernetes",
13+
"type":"bridge",
14+
"bridge":"kube-bridge",
15+
"isDefaultGateway":true,
16+
"ipam": {
17+
"type":"host-local"
18+
}
19+
}
20+
kubeconfig: |
21+
apiVersion: v1
22+
kind: Config
23+
clusterCIDR: "%CLUSTERCIDR%"
24+
clusters:
25+
- name: cluster
26+
cluster:
27+
certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
28+
server: https://%APISERVER%
29+
users:
30+
- name: kube-router
31+
user:
32+
tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
33+
contexts:
34+
- context:
35+
cluster: cluster
36+
user: kube-router
37+
name: kube-router-context
38+
current-context: kube-router-context
39+
40+
---
41+
apiVersion: extensions/v1beta1
42+
kind: DaemonSet
43+
metadata:
44+
labels:
45+
k8s-app: kube-router
46+
tier: node
47+
name: kube-router
48+
namespace: kube-system
49+
spec:
50+
template:
51+
metadata:
52+
labels:
53+
k8s-app: kube-router
54+
tier: node
55+
annotations:
56+
scheduler.alpha.kubernetes.io/critical-pod: ''
57+
spec:
58+
serviceAccountName: kube-router
59+
containers:
60+
- name: kube-router
61+
image: cloudnativelabs/kube-router
62+
imagePullPolicy: Always
63+
args:
64+
- "--run-router=true"
65+
- "--run-firewall=true"
66+
- "--run-service-proxy=true"
67+
- "--kubeconfig=/var/lib/kube-router/kubeconfig"
68+
- "--peer-router-ips=10.1.201.254"
69+
- "--peer-router-asns=64512"
70+
- "--cluster-asn=64512"
71+
- "--advertise-cluster-ip=true"
72+
env:
73+
- name: NODE_NAME
74+
valueFrom:
75+
fieldRef:
76+
fieldPath: spec.nodeName
77+
resources:
78+
requests:
79+
cpu: 250m
80+
memory: 250Mi
81+
securityContext:
82+
privileged: true
83+
volumeMounts:
84+
- name: lib-modules
85+
mountPath: /lib/modules
86+
readOnly: true
87+
- name: cni-conf-dir
88+
mountPath: /etc/cni/net.d
89+
- name: kubeconfig
90+
mountPath: /var/lib/kube-router
91+
readOnly: true
92+
initContainers:
93+
- name: install-cni
94+
image: busybox
95+
imagePullPolicy: Always
96+
command:
97+
- /bin/sh
98+
- -c
99+
- set -e -x;
100+
if [ ! -f /etc/cni/net.d/10-kuberouter.conf ]; then
101+
TMP=/etc/cni/net.d/.tmp-kuberouter-cfg;
102+
cp /etc/kube-router/cni-conf.json ${TMP};
103+
mv ${TMP} /etc/cni/net.d/10-kuberouter.conf;
104+
fi;
105+
if [ ! -f /var/lib/kube-router/kubeconfig ]; then
106+
TMP=/var/lib/kube-router/.tmp-kubeconfig;
107+
cp /etc/kube-router/kubeconfig ${TMP};
108+
mv ${TMP} /var/lib/kube-router/kubeconfig;
109+
fi
110+
volumeMounts:
111+
- mountPath: /etc/cni/net.d
112+
name: cni-conf-dir
113+
- mountPath: /etc/kube-router
114+
name: kube-router-cfg
115+
- name: kubeconfig
116+
mountPath: /var/lib/kube-router
117+
hostNetwork: true
118+
tolerations:
119+
- key: CriticalAddonsOnly
120+
operator: Exists
121+
- effect: NoSchedule
122+
key: node-role.kubernetes.io/master
123+
operator: Exists
124+
volumes:
125+
- name: lib-modules
126+
hostPath:
127+
path: /lib/modules
128+
- name: cni-conf-dir
129+
hostPath:
130+
path: /etc/cni/net.d
131+
- name: kube-router-cfg
132+
configMap:
133+
name: kube-router-cfg
134+
- name: kubeconfig
135+
hostPath:
136+
path: /var/lib/kube-router
137+
138+
---
139+
apiVersion: v1
140+
kind: ServiceAccount
141+
metadata:
142+
name: kube-router
143+
namespace: kube-system
144+
145+
---
146+
kind: ClusterRole
147+
apiVersion: rbac.authorization.k8s.io/v1beta1
148+
metadata:
149+
name: kube-router
150+
namespace: kube-system
151+
rules:
152+
- apiGroups:
153+
- ""
154+
resources:
155+
- namespaces
156+
- pods
157+
- services
158+
- nodes
159+
- endpoints
160+
verbs:
161+
- list
162+
- get
163+
- watch
164+
- apiGroups:
165+
- "networking.k8s.io"
166+
resources:
167+
- networkpolicies
168+
verbs:
169+
- list
170+
- get
171+
- watch
172+
- apiGroups:
173+
- extensions
174+
resources:
175+
- networkpolicies
176+
verbs:
177+
- get
178+
- list
179+
- watch
180+
181+
---
182+
kind: ClusterRoleBinding
183+
apiVersion: rbac.authorization.k8s.io/v1beta1
184+
metadata:
185+
name: kube-router
186+
roleRef:
187+
apiGroup: rbac.authorization.k8s.io
188+
kind: ClusterRole
189+
name: kube-router
190+
subjects:
191+
- kind: ServiceAccount
192+
name: kube-router
193+
namespace: kube-system

0 commit comments

Comments
 (0)