Skip to content

Commit 927b2f5

Browse files
slaskawislaskawi
committed
Updated documentation
1 parent 7e95625 commit 927b2f5

File tree

1 file changed

+161
-23
lines changed

1 file changed

+161
-23
lines changed

‎README.adoc‎

Lines changed: 161 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,47 +1,185 @@
11
= Kubernetes discovery protocol for JGroups
22

3-
This repository contains Kubernetes Ping protocol for JGroups. The protocol can also be used in OpenShift v3 environment (which is also based on Kubernetes).
3+
`KUBE_PING` is a discovery protocol for JGroups cluster nodes managed by Kubernetes.
44

5-
This is the stable branch that will work for both JGroups 3 and 4. We are working on a new version (in master branch) which will contain a lot of improvements for JGroups 4. Stay tuned...
5+
Since Kubernetes is in charge of launching nodes, it knows the IP addresses of all pods it started, and is therefore
6+
the best place to ask for cluster discovery.
67

7-
= How to use it
8+
Discovery is therefore done by asking Kubernetes API for a list of IP addresses of all cluster nodes.
89

9-
In order to run JGroups discovery on Kubernetes one needs to add necessary dependencies to the project:
10+
The protocol spins up a local HTTP Server which is used for sending discovery requests to all instances and wait for
11+
the responses.
1012

11-
.Maven dependencies
13+
A sample configuration looks like this:
14+
15+
.Sample KUBE_PING config
1216
[source,xml]
1317
----
14-
<dependency>
15-
<groupId>org.jgroups.kubernetes</groupId>
16-
<artifactId>kubernetes</artifactId>
17-
<version>${version.kubernetes-ping}</version>
18-
</dependency>
18+
<TCP
19+
bind_addr="loopback,match-interface:eth0"
20+
bind_port="7800"
21+
...
22+
/>
23+
<kubernetes.KUBE_PING
24+
/>
25+
...
26+
----
27+
28+
When a discovery is started, `KUBE_PING` asks Kubernetes for a list of the IP addresses of all pods which are launched,
29+
matching the given _namespace_ and _labels_ (see below).
30+
31+
Let's say Kubernetes launched a cluster of 3 pods with IP addresses `172.17.0.2`, `172.17.0.3` and `172.17.0.5` (all
32+
launched into the same namespace and without any (or the same) labels).
33+
34+
On a discovery request, Kubernetes returns list of 3 IP addresses. JGroups will use embedded HTTP Server exposed on port
35+
8888 by default (see below for configuration) and will send HTTP based requests to each od them.
36+
37+
KUBE_PING therefore sends discovery requests to members at addresses `172.17.0.2:8888`, `172.17.0.3:8888` and
38+
`172.17.0.5:8888`.
39+
40+
41+
== Separating different clusters
42+
43+
If pods with containers in different clusters are launched, we'd get a list of IP addresses of all nodes, not just the
44+
ones in the same cluster, as Kubernetes knows nothing about clusters.
45+
46+
If we start multiple clusters, we have several options to consider. We may use separate namespaces, Network Policies
47+
and/or separate them using different labels (see configuration parameters below).
48+
49+
50+
=== Namespaces
51+
52+
https://kubernetes.io/docs/tasks/administer-cluster/namespaces/[Kubernetes namespaces] provide an easy way to separate
53+
all resources and limit communication between tenants. This is the most effective method of separating clusters from
54+
each other. Additionally https://kubernetes.io/docs/concepts/services-networking/networkpolicies/[Network Policies]
55+
allow to fine-tune this mechanism.
56+
57+
When `KUBE_PING` asks Kubernetes API for Pods, it needs to supply proper namespace to Kubernetes API. This allows
58+
to distinguish a namespace without matching Pods from asking Kubernetes API with wrong tenant.
59+
60+
Having said that, namespace is a required parameter and must be specified when starting application. The easiest way to
61+
set it correctly is to use https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information[Downward API].
62+
63+
Here's an example:
64+
65+
.Configuration with namespace
66+
[source,yaml]
1967
----
68+
apiVersion: v1
69+
items:
70+
- apiVersion: extensions/v1beta1
71+
kind: Deployment
72+
spec:
73+
template:
74+
metadata:
75+
labels:
76+
run: infinispan-server
77+
spec:
78+
containers:
79+
- args:
80+
- cloud
81+
- -Djboss.default.jgroups.stack=kubernetes
82+
env:
83+
- name: OPENSHIFT_KUBE_PING_NAMESPACE
84+
valueFrom:
85+
fieldRef:
86+
apiVersion: v1
87+
fieldPath: metadata.namespace
88+
- name: OPENSHIFT_KUBE_PING_LABELS
89+
value: "cluster=cluster-1"
90+
image: jboss/infinispan-server:9.0.0.Final
91+
name: infinispan-server
92+
ports:
93+
- containerPort: 8080
94+
protocol: TCP
95+
- containerPort: 8181
96+
protocol: TCP
97+
- containerPort: 8888
98+
protocol: TCP
99+
- containerPort: 9990
100+
protocol: TCP
101+
- containerPort: 11211
102+
protocol: TCP
103+
- containerPort: 11222
104+
protocol: TCP
105+
kind: List
106+
metadata: {}
107+
----
108+
109+
=== Labels
110+
111+
Labels allow to separate clusters running inside the _same namespace_.
112+
113+
The easiest way to set labels is to use `OPENSHIFT_KUBE_PING_LABELS` environmental variable and define them in
114+
YAML or JSON configuration file (which can be passed into `kubectl create -f <config>`).
115+
116+
Note, that labels are optional. The cluster will work fine without them.
117+
118+
119+
=== KUBE_PING configuration
120+
121+
[align="left",width="90%",options="header"]
122+
|===============
123+
|Attribute name|Description
124+
| connectTimeout|Max time (in millis) to wait for a connection to the Kubernetes server. If exceeded, an exceptionwill be thrown
125+
| readTimeout|Max time (in millis) to wait for a response from the Kubernetes server
126+
| operationAttempts | Max number of attempts to send discovery requests
127+
| operationSleep|Time (in millis) between operation attempts
128+
| masterProtocol | http (default) or https. Used to send the initial discovery request to the Kubernetes server
129+
| masterHost | The URL of the Kubernetes server
130+
| masterPort | The port on which the Kubernetes server is listening
131+
| apiVersion | The version of the protocol to the Kubernetes server
132+
| namespace | The namespace to be used (leaving this undefined uses `"default"`)
133+
| labels | The labels to use in the discovery request to the Kubernetes server
134+
| clientCertFile | Certificate to access the Kubernetes server
135+
| clientKeyFile | Client key file (store)
136+
| clientKeyPassword | The password to access the client key store
137+
| clientKeyAlgo | The algorithm used by the client
138+
| caCertFile | Client CA certificate
139+
| saTokenFile | Token file
140+
|===============
141+
142+
143+
=== KUBE_PING configuration using environmental variables
144+
145+
It is also possible to set the most critical configuration parameters using environmental variables. This approach
146+
is dedicated to configuration specification using JSON or YAML files.
147+
148+
[align="left",width="90%",options="header"]
149+
|===============
150+
|Environmental variable name|Description
151+
| OPENSHIFT_KUBE_PING_NAMESPACE|Kubernetes/OpenShift namespace
152+
| OPENSHIFT_KUBE_PING_LABELS|Labels used for discovery
153+
| OPENSHIFT_KUBE_PING_SERVER_PORT | Port used for running embedded HTTP server
154+
|===============
20155

21-
It is highly recommended to use JGroups version which is transitively pulled by the Kubernetes discovery protocol (however it is not strictly required).
156+
== Running inside of OpenShift
22157

23-
When deploying on OpenShift v3, the next step requires adding additional privileges using the following command:
158+
OpenShift 3 uses https://docs.openshift.com/enterprise/3.0/dev_guide/service_accounts.html[Service Account]
159+
mechanism to limit Kubernetes API from the Pods.
160+
161+
This requires additional steps when before running the cluster:
24162

25163
.Adding additional privileges
26164
[source,bash]
27165
----
28166
oc policy add-role-to-user view system:serviceaccount:$(oc project -q):default -n $(oc project -q)
29167
----
30168

31-
That's it!
32-
33-
= Additional configuration options
169+
== Maven dependencies
34170

35-
== Environmental variables
171+
In order to run JGroups discovery on Kubernetes one needs to add necessary dependencies to the project:
36172

37-
The configuration is mainly done using environmental variables:
173+
.Maven dependencies
174+
[source,xml]
175+
----
176+
<dependency>
177+
<groupId>org.jgroups.kubernetes</groupId>
178+
<artifactId>kubernetes</artifactId>
179+
<version>${version.kubernetes-ping}</version>
180+
</dependency>
181+
----
38182

39-
|==============================================================================================
40-
|Variable name |Meaning |Default |Example
41-
|OPENSHIFT_KUBE_PING_NAMESPACE |Namespace used for discovery |<not set> |Cluster-1
42-
|OPENSHIFT_KUBE_PING_LABELS |Labels used for discovery |default |my-cluster
43-
|OPENSHIFT_KUBE_PING_SERVER_PORT |Port used for hosting internal server |8888 |my-cluster
44-
|===============================================================================================
45183

46184
= How to tell if it's working?
47185

0 commit comments

Comments
 (0)