Skip to content

Commit 13a6055

Browse files
FilipBFilipB
authored
OSSM-8200: adding doc for multiple control planes + scoping a mesh (openshift-service-mesh#147)
1 parent 21bd142 commit 13a6055

File tree

3 files changed

+306
-0
lines changed

3 files changed

+306
-0
lines changed

‎docs/ossm/README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,3 +8,6 @@ This documentation is specific to the OpenShift Service Mesh product and may dif
88

99
- [Running Red Hat OpenShift Service Mesh (OSSM) 2 and OSSM 3 side by side](./ossm-2-and-ossm-3-side-by-side/README.md)
1010
- [Cert Manager and istio-csr Integration](./cert-manager/README.md)
11+
- [Adding services to a service mesh](./create-mesh/README.md)
12+
- [Installing the Sidecar](./injection/README.md)
13+
- [Multiple Istio Control Planes in a Single Cluster](./multi-control-planes/README.md)

‎docs/ossm/create-mesh/README.md‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
# Scoping the service mesh with DiscoverySelectors
2+
This page describes how the control plane monitors/discovers cluster resources and how to manage its scope.
3+
4+
A service mesh will include a workload that:
5+
1. Has been discovered by the control plane
6+
1. Has been [injected with a Envoy proxy sidecar](../injection/README.md)
7+
8+
9+
By default, the control plane will watch all namespaces within the cluster, meaning that:
10+
- Each proxy instance will receive configuration for all namespaces. This includes information also about workloads that are not enrolled in the mesh.
11+
- Any workload with the appropriate pod or namespace injection label will be injected with a proxy sidecar.
12+
13+
This may not be desirable in a shared cluster, and you may want to limit the scope of the service mesh to only a portion of your cluster. This is particularly important if you plan to have [multiple service meshes within the same cluster](./multi-control-planes/README.md).
14+
15+
### DiscoverySelectors
16+
Discovery selectors provide a mechanism for the mesh administrator to limit the scope of a service mesh. This is done through a Kubernetes [label selector](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors), which defines criteria for which namespaces will be visible to the control plane. Any namespaces not matching are ignored by the control plane entirely.
17+
18+
> **_NOTE:_** Istiod will always open a watch to OpenShift for all namespaces. However, discovery selectors will ignore objects that are not selected very early in its processing, minimizing costs.
19+
20+
> **_NOTE:_** `discoverySelectors` is not a security boundary. Istiod will continue to have access to all namespaces even when you have configured your `discoverySelectors`.
21+
22+
#### Using DiscoverySelectors
23+
The `discoverySelectors` field accepts an array of Kubernetes [selectors](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements). The exact type is `[]LabelSelector`, as defined [here](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements), allowing both simple selectors and set-based selectors. These selectors apply to labels on namespaces.
24+
25+
You can configure each label selector for a variety of use cases, including but not limited to:
26+
27+
- Arbitrary label names/values, for example, all namespaces with label `istio-discovery=enabled`
28+
- A list of namespace labels using set-based selectors which carries OR semantics, for example, all namespaces with label `istio-discovery=enabled` OR `region=us-east1`
29+
- Inclusion and/or exclusion of namespaces, for example, all namespaces with label `istio-discovery=enabled` AND label key `app` equal to `helloworld`
30+
31+
#### Using Discovery Selectors to Scope of a Service Mesh
32+
Assuming you know which namespaces to include as part of the service mesh, as a mesh administrator, you can configure `discoverySelectors` at installation time or post-installation by adding your desired discovery selectors to Istio’s MeshConfig resource. For example, you can configure Istio to discover only the namespaces that have the label `istio-discovery=enabled`.
33+
34+
##### Prerequisites
35+
- The OpenShift Service Mesh operator has been installed
36+
- An Istio CNI resource has been created
37+
- The `istioctl` binary has been installed on your localhost
38+
39+
1. Create the `istio-system` system namespace:
40+
```bash
41+
oc create ns istio-system
42+
```
43+
1. Label the `istio-system` system namespace:
44+
```bash
45+
oc label ns istio-system istio-discovery=enabled
46+
```
47+
1. Prepare `istio.yaml` with `discoverySelectors` configured:
48+
```yaml
49+
kind: Istio
50+
apiVersion: sailoperator.io/v1alpha1
51+
metadata:
52+
name: default
53+
spec:
54+
namespace: istio-system
55+
values:
56+
meshConfig:
57+
discoverySelectors:
58+
- matchLabels:
59+
istio-discovery: enabled
60+
updateStrategy:
61+
type: InPlace
62+
version: v1.23.0
63+
```
64+
1. Apply the Istio CR:
65+
```bash
66+
oc apply -f istio.yaml
67+
```
68+
1. Create first application namespace:
69+
```bash
70+
oc create ns app-ns-1
71+
```
72+
1. Create second application namespace:
73+
```bash
74+
oc create ns app-ns-2
75+
```
76+
1. Label first application namespace to be matched by defined `discoverySelectors` and enable sidecar injection:
77+
```bash
78+
oc label ns app-ns-1 istio-discovery=enabled istio-injection=enabled
79+
```
80+
1. Deploy the sleep application to the first namespaces:
81+
```bash
82+
oc apply -f https://raw.githubusercontent.com/istio/istio/release-1.23/samples/sleep/sleep.yaml -n app-ns-1
83+
```
84+
1. Deploy the sleep application to the second namespaces:
85+
```bash
86+
oc apply -f https://raw.githubusercontent.com/istio/istio/release-1.23/samples/sleep/sleep.yaml -n app-ns-2
87+
```
88+
1. Verify that you don't see any endpoints from the second namespace:
89+
```bash
90+
istioctl pc endpoint deploy/sleep -n app-ns-1
91+
ENDPOINT STATUS OUTLIER CHECK CLUSTER
92+
10.128.2.197:15010 HEALTHY OK outbound|15010||istiod.istio-system.svc.cluster.local
93+
10.128.2.197:15012 HEALTHY OK outbound|15012||istiod.istio-system.svc.cluster.local
94+
10.128.2.197:15014 HEALTHY OK outbound|15014||istiod.istio-system.svc.cluster.local
95+
10.128.2.197:15017 HEALTHY OK outbound|443||istiod.istio-system.svc.cluster.local
96+
10.131.0.32:80 HEALTHY OK outbound|80||sleep.app-ns-1.svc.cluster.local
97+
127.0.0.1:15000 HEALTHY OK prometheus_stats
98+
127.0.0.1:15020 HEALTHY OK agent
99+
unix://./etc/istio/proxy/XDS HEALTHY OK xds-grpc
100+
unix://./var/run/secrets/workload-spiffe-uds/socket HEALTHY OK sds-grpc
101+
```
102+
1. Label second application namespace to be matched by defined `discoverySelectors` and enable sidecar injection:
103+
```bash
104+
oc label ns app-ns-2 istio-discovery=enabled
105+
```
106+
1. Verify that after labeling second namespace it also appears on the list of discovered endpoints:
107+
```bash
108+
istioctl pc endpoint deploy/sleep -n app-ns-1
109+
ENDPOINT STATUS OUTLIER CHECK CLUSTER
110+
10.128.2.197:15010 HEALTHY OK outbound|15010||istiod.istio-system.svc.cluster.local
111+
10.128.2.197:15012 HEALTHY OK outbound|15012||istiod.istio-system.svc.cluster.local
112+
10.128.2.197:15014 HEALTHY OK outbound|15014||istiod.istio-system.svc.cluster.local
113+
10.128.2.197:15017 HEALTHY OK outbound|443||istiod.istio-system.svc.cluster.local
114+
10.131.0.32:80 HEALTHY OK outbound|80||sleep.app-ns-1.svc.cluster.local
115+
10.131.0.33:80 HEALTHY OK outbound|80||sleep.app-ns-2.svc.cluster.local
116+
127.0.0.1:15000 HEALTHY OK prometheus_stats
117+
127.0.0.1:15020 HEALTHY OK agent
118+
unix://./etc/istio/proxy/XDS HEALTHY OK xds-grpc
119+
unix://./var/run/secrets/workload-spiffe-uds/socket HEALTHY OK sds-grpc
120+
```
121+
122+
See [Multiple Istio Control Planes in a Single Cluster](../multi-control-planes/README.md) for another example of `discoverySelectors` usage.

‎docs/ossm/multi-control-planes/README.md‎

Lines changed: 181 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,181 @@
1+
# Multiple Istio Control Planes in a Single Cluster
2+
By default, the control plane will watch all namespaces within the cluster so two control planes would be conflicting each other resulting in undefined behavior.
3+
4+
To resolve this, Istio provides [discoverySelectors](../create-mesh/README.md#discoveryselectors) which together with control plane revisions enables you to install multiple control planes in a single cluster.
5+
6+
## Prerequisites
7+
- The OpenShift Service Mesh operator has been installed
8+
- An Istio CNI resource has been created
9+
- The `istioctl` binary has been installed on your localhost
10+
11+
## Deploying multiple control planes
12+
The cluster will host two control planes installed in two different system namespaces. The mesh application workloads will run in multiple application-specific namespaces, each namespace associated with one or the other control plane based on revision and discovery selector configurations.
13+
14+
1. Create the first system namespace `usergroup-1`:
15+
```bash
16+
oc create ns usergroup-1
17+
```
18+
1. Label the first system namespace:
19+
```bash
20+
oc label ns usergroup-1 usergroup=usergroup-1
21+
```
22+
1. Prepare `istio-1.yaml`:
23+
```yaml
24+
kind: Istio
25+
apiVersion: sailoperator.io/v1alpha1
26+
metadata:
27+
name: usergroup-1
28+
spec:
29+
namespace: usergroup-1
30+
values:
31+
meshConfig:
32+
discoverySelectors:
33+
- matchLabels:
34+
usergroup: usergroup-1
35+
updateStrategy:
36+
type: InPlace
37+
version: v1.23.0
38+
```
39+
1. Create `Istio` resource:
40+
```bash
41+
oc apply -f istio-1.yaml
42+
```
43+
1. Create the second system namespace `usergroup-2`:
44+
```bash
45+
oc create ns usergroup-2
46+
```
47+
1. Label the second system namespace:
48+
```bash
49+
oc label ns usergroup-2 usergroup=usergroup-2
50+
```
51+
1. Prepare `istio-2.yaml`:
52+
```yaml
53+
kind: Istio
54+
apiVersion: sailoperator.io/v1alpha1
55+
metadata:
56+
name: usergroup-2
57+
spec:
58+
namespace: usergroup-2
59+
values:
60+
meshConfig:
61+
discoverySelectors:
62+
- matchLabels:
63+
usergroup: usergroup-2
64+
updateStrategy:
65+
type: InPlace
66+
version: v1.23.0
67+
```
68+
1. Create `Istio` resource:
69+
```bash
70+
oc apply -f istio-2.yaml
71+
```
72+
1. Deploy a policy for workloads in the `usergroup-1` namespace to only accept mutual TLS traffic `peer-auth-1.yaml`:
73+
```yaml
74+
apiVersion: security.istio.io/v1
75+
kind: PeerAuthentication
76+
metadata:
77+
name: "usergroup-1-peerauth"
78+
namespace: "usergroup-1"
79+
spec:
80+
mtls:
81+
mode: STRICT
82+
```
83+
```bash
84+
oc apply -f peer-auth-1.yaml
85+
```
86+
1. Deploy a policy for workloads in the `usergroup-2` namespace to only accept mutual TLS traffic `peer-auth-2.yaml`:
87+
```yaml
88+
apiVersion: security.istio.io/v1
89+
kind: PeerAuthentication
90+
metadata:
91+
name: "usergroup-2-peerauth"
92+
namespace: "usergroup-2"
93+
spec:
94+
mtls:
95+
mode: STRICT
96+
```
97+
```bash
98+
oc apply -f peer-auth-2.yaml
99+
```
100+
1. Verify the control planes are deployed and running:
101+
```bash
102+
oc get pods -n usergroup-1
103+
NAME READY STATUS RESTARTS AGE
104+
istiod-usergroup-1-747fddfb56-xzpkj 1/1 Running 0 5m1s
105+
oc get pods -n usergroup-2
106+
NAME READY STATUS RESTARTS AGE
107+
istiod-usergroup-2-5b9cbb7669-lwhgv 1/1 Running 0 3m41s
108+
```
109+
110+
## Deploy application workloads per usergroup
111+
1. Create three application namespaces:
112+
```bash
113+
oc create ns app-ns-1
114+
oc create ns app-ns-2
115+
oc create ns app-ns-3
116+
```
117+
1. Label each namespace to associate them with their respective control planes:
118+
```bash
119+
oc label ns app-ns-1 usergroup=usergroup-1 istio.io/rev=usergroup-1
120+
oc label ns app-ns-2 usergroup=usergroup-2 istio.io/rev=usergroup-2
121+
oc label ns app-ns-3 usergroup=usergroup-2 istio.io/rev=usergroup-2
122+
```
123+
1. Deploy one `sleep` and `httpbin` application per namespace:
124+
```bash
125+
oc apply -f https://raw.githubusercontent.com/istio/istio/release-1.23/samples/sleep/sleep.yaml -n app-ns-1
126+
oc apply -f https://raw.githubusercontent.com/istio/istio/master/samples/httpbin/httpbin.yaml -n app-ns-1
127+
oc apply -f https://raw.githubusercontent.com/istio/istio/release-1.23/samples/sleep/sleep.yaml -n app-ns-2
128+
oc apply -f https://raw.githubusercontent.com/istio/istio/master/samples/httpbin/httpbin.yaml -n app-ns-2
129+
oc apply -f https://raw.githubusercontent.com/istio/istio/release-1.23/samples/sleep/sleep.yaml -n app-ns-3
130+
oc apply -f https://raw.githubusercontent.com/istio/istio/master/samples/httpbin/httpbin.yaml -n app-ns-3
131+
```
132+
1. Wait a few seconds for the `httpbin` and `sleep` pods to be running with sidecars injected:
133+
```bash
134+
oc get pods -n app-ns-1
135+
NAME READY STATUS RESTARTS AGE
136+
httpbin-7f56dc944b-kpw2x 2/2 Running 0 2m26s
137+
sleep-5577c64d7c-b5wd2 2/2 Running 0 91m
138+
```
139+
Repeat this step for other application namespaces (`app-ns-2`, `app-ns-3`).
140+
> [!TIP]
141+
> `oc wait deployment sleep -n app-ns-1` can be used to wait for a deployment to be ready
142+
143+
## Verify the application to control plane mapping
144+
Now that the applications are deployed, you can use the `istioctl ps` command to confirm that the application workloads are managed by their respective control plane, i.e., `app-ns-1` is managed by `usergroup-1`, `app-ns-2` and `app-ns-3` are managed by `usergroup-2`:
145+
```bash
146+
istioctl ps -i usergroup-1
147+
NAME CLUSTER CDS LDS EDS RDS ECDS ISTIOD VERSION
148+
httpbin-7f56dc944b-kpw2x.app-ns-1 Kubernetes SYNCED (2m23s) SYNCED (2m23s) SYNCED (2m23s) SYNCED (2m23s) IGNORED istiod-usergroup-1-747fddfb56-xzpkj 1.23.0
149+
sleep-5577c64d7c-b5wd2.app-ns-1 Kubernetes SYNCED (66s) SYNCED (66s) SYNCED (66s) SYNCED (66s) IGNORED istiod-usergroup-1-747fddfb56-xzpkj 1.23.0
150+
```
151+
```bash
152+
istioctl ps -i usergroup-2
153+
NAME CLUSTER CDS LDS EDS RDS ECDS ISTIOD VERSION
154+
httpbin-7f56dc944b-g4s57.app-ns-3 Kubernetes SYNCED (2m) SYNCED (2m) SYNCED (2m) SYNCED (2m) IGNORED istiod-usergroup-2-5b9cbb7669-lwhgv 1.23.0
155+
httpbin-7f56dc944b-rzwr5.app-ns-2 Kubernetes SYNCED (2m2s) SYNCED (2m2s) SYNCED (2m) SYNCED (2m2s) IGNORED istiod-usergroup-2-5b9cbb7669-lwhgv 1.23.0
156+
sleep-5577c64d7c-wjnxc.app-ns-3 Kubernetes SYNCED (2m2s) SYNCED (2m2s) SYNCED (2m) SYNCED (2m2s) IGNORED istiod-usergroup-2-5b9cbb7669-lwhgv 1.23.0
157+
sleep-5577c64d7c-xk27f.app-ns-2 Kubernetes SYNCED (2m2s) SYNCED (2m2s) SYNCED (2m) SYNCED (2m2s) IGNORED istiod-usergroup-2-5b9cbb7669-lwhgv 1.23.0
158+
```
159+
## Verify the application connectivity is ONLY within the respective usergroup
160+
1. Send a request from the `sleep` pod in `app-ns-1` in `usergroup-1` to the `httpbin` service in `app-ns-2` in `usergroup-2`. The communication should fail:
161+
```bash
162+
oc -n app-ns-1 exec "$(oc -n app-ns-1 get pod -l app=sleep -o jsonpath={.items..metadata.name})" -c sleep -- curl -sIL http://httpbin.app-ns-2.svc.cluster.local:8000
163+
HTTP/1.1 503 Service Unavailable
164+
content-length: 95
165+
content-type: text/plain
166+
date: Wed, 16 Oct 2024 12:05:37 GMT
167+
server: envoy
168+
```
169+
1. Send a request from the `sleep` pod in `app-ns-2` in `usergroup-2` to the `httpbin` service in `app-ns-3` in `usergroup-2`. The communication should work:
170+
```bash
171+
oc -n app-ns-2 exec "$(oc -n app-ns-2 get pod -l app=sleep -o jsonpath={.items..metadata.name})" -c sleep -- curl -sIL http://httpbin.app-ns-3.svc.cluster.local:8000
172+
HTTP/1.1 200 OK
173+
access-control-allow-credentials: true
174+
access-control-allow-origin: *
175+
content-security-policy: default-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' camo.githubusercontent.com
176+
content-type: text/html; charset=utf-8
177+
date: Wed, 16 Oct 2024 12:06:30 GMT
178+
x-envoy-upstream-service-time: 8
179+
server: envoy
180+
transfer-encoding: chunked
181+
```

0 commit comments

Comments
 (0)