Skip to content

Commit 515b4c9

Browse files
MatthieuFinsebastienmussoFlorentLaloup
authored
[cinder-csi-plugin] Multi region/clouds support for controllerServer (kubernetes#2551)
* feat(cinder-csi-plugin): add support multi cloud openstack cluster (kubernetes#2035) * feat(cinder-csi-plugin-controller): add support --cloud-name arg which permit to manage multiple config Global sections * feat(cinder-csi-plugin-controller): add support of key `cloud` from secret specified in storageClass to reference specific config Global section * feat(cinder-csi-plugin-node): add support --cloud-name arg which permit to specify one of config Global section Signed-off-by: MatthieuFin <[email protected]> * feat(cinder-csi-plugin): add possibility to set custom topology keys (kubernetes#2035) * feat(cinder-csi-plugin-node): Additionnal topology keys `--additionnal-topology` are announced on NodeGetInfo to create proper CSINode object and could used in storage class allowedTopologies field, in combination with csi-provisioner `--feature-gates Topology=true` created PV will have nodeAffinity set with topologies presents in storageClass allowedTopologies and in `--additionnal-topology` argument. Signed-off-by: MatthieuFin <[email protected]> * refactor: 💡 rephrase command flags descriptions Signed-off-by: MatthieuFin <[email protected]> * refactor: 💡 remove useless declaration * refactor: 💡 loop with range instead of C like * docs(cinder-csi-plugin): multi cloud configuration add documentation and examples about multi cloud support configuration Co-authored-by: sebastienmusso <[email protected]> Co-authored-by: FlorentLaloup <[email protected]> Signed-off-by: MatthieuFin <[email protected]> * fix(cinder-csi-plugin): controllerServer ListVolumes with `--max-entries` arg Add unit tests cases on listvolumes function in multicloud configuration with and without `--max-entries` arg Signed-off-by: MatthieuFin <[email protected]> --------- Signed-off-by: MatthieuFin <[email protected]> Co-authored-by: sebastienmusso <[email protected]> Co-authored-by: FlorentLaloup <[email protected]>
1 parent e39dffb commit 515b4c9

File tree

12 files changed

+1679
-159
lines changed

12 files changed

+1679
-159
lines changed

‎cmd/cinder-csi-plugin/main.go‎

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,8 @@ var (
3434
endpoint string
3535
nodeID string
3636
cloudConfig []string
37+
cloudNames []string
38+
additionalTopologies map[string]string
3739
cluster string
3840
httpEndpoint string
3941
provideControllerService bool
@@ -65,6 +67,9 @@ func main() {
6567
klog.Fatalf("Unable to mark flag cloud-config to be required: %v", err)
6668
}
6769

70+
cmd.PersistentFlags().StringSliceVar(&cloudNames, "cloud-name", []string{""}, "Cloud name to instruct CSI driver to read additional OpenStack cloud credentials from the configuration subsections. This option can be specified multiple times to manage multiple OpenStack clouds.")
71+
cmd.PersistentFlags().StringToStringVar(&additionalTopologies, "additional-topology", map[string]string{}, "Additional CSI driver topology keys, for example topology.kubernetes.io/region=REGION1. This option can be specified multiple times to add multiple additional topology keys.")
72+
6873
cmd.PersistentFlags().StringVar(&cluster, "cluster", "", "The identifier of the cluster that the plugin is running in.")
6974
cmd.PersistentFlags().StringVar(&httpEndpoint, "http-endpoint", "", "The TCP network address where the HTTP server for providing metrics for diagnostics, will listen (example: `:8080`). The default is empty string, which means the server is disabled.")
7075

@@ -82,24 +87,28 @@ func handle() {
8287
d := cinder.NewDriver(&cinder.DriverOpts{Endpoint: endpoint, ClusterID: cluster})
8388

8489
openstack.InitOpenStackProvider(cloudConfig, httpEndpoint)
85-
cloud, err := openstack.GetOpenStackProvider()
86-
if err != nil {
87-
klog.Warningf("Failed to GetOpenStackProvider: %v", err)
88-
return
90+
var err error
91+
clouds := make(map[string]openstack.IOpenStack)
92+
for _, cloudName := range cloudNames {
93+
clouds[cloudName], err = openstack.GetOpenStackProvider(cloudName)
94+
if err != nil {
95+
klog.Warningf("Failed to GetOpenStackProvider %s: %v", cloudName, err)
96+
return
97+
}
8998
}
9099

91100
if provideControllerService {
92-
d.SetupControllerService(cloud)
101+
d.SetupControllerService(clouds)
93102
}
94103

95104
if provideNodeService {
96105
//Initialize mount
97106
mount := mount.GetMountProvider()
98107

99108
//Initialize Metadata
100-
metadata := metadata.GetMetadataProvider(cloud.GetMetadataOpts().SearchOrder)
109+
metadata := metadata.GetMetadataProvider(clouds[cloudNames[0]].GetMetadataOpts().SearchOrder)
101110

102-
d.SetupNodeService(cloud, mount, metadata)
111+
d.SetupNodeService(clouds[cloudNames[0]], mount, metadata, additionalTopologies)
103112
}
104113

105114
d.Run()

‎docs/cinder-csi-plugin/multi-region-clouds.md‎

Lines changed: 316 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,316 @@
1+
# Multi Az/Region/Openstack Configuration
2+
3+
### Multi cluster Configuration file
4+
5+
Create a configuration file with a subsection per openstack cluster to manage (pay attention to enable ignore-volume-az in BlockStorage section).
6+
7+
Example of configuration with 3 regions (The default is backward compatible with mono cluster configuration but not mandatory).
8+
```yaml
9+
apiVersion: v1
10+
kind: Secret
11+
metadata:
12+
name: cloud-config
13+
namespace: kube-system
14+
type: Opaque
15+
stringData:
16+
cloud.conf: |-
17+
[BlockStorage]
18+
bs-version=v3
19+
ignore-volume-az=True
20+
21+
[Global]
22+
auth-url="https://auth.cloud.openstackcluster.region-default.local/v3"
23+
username="region-default-username"
24+
password="region-default-password"
25+
region="default"
26+
tenant-id="region-default-tenant-id"
27+
tenant-name="region-default-tenant-name"
28+
domain-name="Default"
29+
30+
[Global "region-one"]
31+
auth-url="https://auth.cloud.openstackcluster.region-one.local/v3"
32+
username="region-one-username"
33+
password="region-one-password"
34+
region="one"
35+
tenant-id="region-one-tenant-id"
36+
tenant-name="region-one-tenant-name"
37+
domain-name="Default"
38+
39+
[Global "region-two"]
40+
auth-url="https://auth.cloud.openstackcluster.region-two.local/v3"
41+
username="region-two-username"
42+
password="region-two-password"
43+
region="two"
44+
tenant-id="region-two-tenant-id"
45+
tenant-name="region-two-tenant-name"
46+
domain-name="Default"
47+
```
48+
49+
50+
51+
### Create region/cloud secrets
52+
53+
Create a secret per openstack cluster which contains a key `cloud` and as value the subsection's name of corresponding openstack cluster in configuration file.
54+
55+
These secrets are referenced in storageClass definitions to identify openstack cluster associated to the storageClass.
56+
57+
```yaml
58+
apiVersion: v1
59+
kind: Secret
60+
metadata:
61+
name: openstack-config-region-one
62+
namespace: kube-system
63+
type: Opaque
64+
stringData:
65+
cloud: region-one
66+
---
67+
apiVersion: v1
68+
kind: Secret
69+
metadata:
70+
name: openstack-config-region-two
71+
namespace: kube-system
72+
type: Opaque
73+
stringData:
74+
cloud: region-two
75+
```
76+
77+
### Create storage Class for dedicated cluster
78+
79+
```yaml
80+
apiVersion: storage.k8s.io/v1
81+
kind: StorageClass
82+
metadata:
83+
annotations:
84+
storageclass.kubernetes.io/is-default-class: "true"
85+
name: sc-region-one
86+
allowVolumeExpansion: true
87+
allowedTopologies:
88+
- matchLabelExpressions:
89+
- key: topology.cinder.csi.openstack.org/zone
90+
values:
91+
- nova
92+
- key: topology.kubernetes.io/region
93+
values:
94+
- region-one
95+
parameters:
96+
csi.storage.k8s.io/controller-publish-secret-name: openstack-config-region-one
97+
csi.storage.k8s.io/controller-publish-secret-namespace: kube-system
98+
csi.storage.k8s.io/node-publish-secret-name: openstack-config-region-one
99+
csi.storage.k8s.io/node-publish-secret-namespace: kube-system
100+
csi.storage.k8s.io/node-stage-secret-name: openstack-config-region-one
101+
csi.storage.k8s.io/node-stage-secret-namespace: kube-system
102+
csi.storage.k8s.io/provisioner-secret-name: openstack-config-region-one
103+
csi.storage.k8s.io/provisioner-secret-namespace: kube-system
104+
provisioner: cinder.csi.openstack.org
105+
reclaimPolicy: Delete
106+
volumeBindingMode: Immediate
107+
---
108+
apiVersion: storage.k8s.io/v1
109+
kind: StorageClass
110+
metadata:
111+
name: sc-region-two
112+
allowVolumeExpansion: true
113+
allowedTopologies:
114+
- matchLabelExpressions:
115+
- key: topology.cinder.csi.openstack.org/zone
116+
values:
117+
- nova
118+
- key: topology.kubernetes.io/region
119+
values:
120+
- region-two
121+
parameters:
122+
csi.storage.k8s.io/controller-publish-secret-name: openstack-config-region-two
123+
csi.storage.k8s.io/controller-publish-secret-namespace: kube-system
124+
csi.storage.k8s.io/node-publish-secret-name: openstack-config-region-two
125+
csi.storage.k8s.io/node-publish-secret-namespace: kube-system
126+
csi.storage.k8s.io/node-stage-secret-name: openstack-config-region-two
127+
csi.storage.k8s.io/node-stage-secret-namespace: kube-system
128+
csi.storage.k8s.io/provisioner-secret-name: openstack-config-region-two
129+
csi.storage.k8s.io/provisioner-secret-namespace: kube-system
130+
provisioner: cinder.csi.openstack.org
131+
reclaimPolicy: Delete
132+
volumeBindingMode: Immediate
133+
```
134+
135+
### Create a csi-cinder-nodeplugin daemonset per cluster openstack
136+
137+
Daemonsets should deploy pods on nodes from proper openstack context. We suppose that the node have a label `topology.kubernetes.io/region` with the openstack cluster name as value (you could manage this with kubespray, manually, whatever, it should be great to implement this in openstack cloud controller manager).
138+
139+
Do as follows:
140+
- Use nodeSelector to match proper nodes labels
141+
- Add cli argument `--additionnal-topology topology.kubernetes.io/region=region-one`, which should match node labels, to container cinder-csi-plugin
142+
- Add cli argument `--cloud-name="region-one"`, which should match configuration file subsection name, to container cinder-csi-plugin.
143+
144+
```yaml
145+
apiVersion: apps/v1
146+
kind: DaemonSet
147+
metadata:
148+
name: csi-cinder-nodeplugin-region-one
149+
namespace: kube-system
150+
spec:
151+
selector:
152+
matchLabels:
153+
app: csi-cinder-nodeplugin-region-one
154+
template:
155+
metadata:
156+
labels:
157+
app: csi-cinder-nodeplugin-region-one
158+
spec:
159+
containers:
160+
- name: node-driver-registrar
161+
...
162+
- name: liveness-probe
163+
...
164+
- name: cinder-csi-plugin
165+
image: docker.io/k8scloudprovider/cinder-csi-plugin:v1.31.0
166+
args:
167+
- /bin/cinder-csi-plugin
168+
- --endpoint=$(CSI_ENDPOINT)
169+
- --cloud-config=$(CLOUD_CONFIG)
170+
- --cloud-name="region-one"
171+
- --additionnal-topology
172+
- topology.kubernetes.io/region=region-one
173+
env:
174+
- name: CSI_ENDPOINT
175+
value: unix://csi/csi.sock
176+
- name: CLOUD_CONFIG
177+
value: /etc/config/cloud.conf
178+
...
179+
volumeMounts:
180+
...
181+
- mountPath: /etc/config
182+
name: secret-cinderplugin
183+
readOnly: true
184+
...
185+
nodeSelector:
186+
topology.kubernetes.io/region: region-one
187+
volumes:
188+
...
189+
- name: secret-cinderplugin
190+
secret:
191+
defaultMode: 420
192+
secretName: cloud-config
193+
...
194+
---
195+
apiVersion: apps/v1
196+
kind: DaemonSet
197+
metadata:
198+
name: csi-cinder-nodeplugin-region-two
199+
namespace: kube-system
200+
spec:
201+
selector:
202+
matchLabels:
203+
app: csi-cinder-nodeplugin-region-two
204+
template:
205+
metadata:
206+
labels:
207+
app: csi-cinder-nodeplugin-region-two
208+
spec:
209+
containers:
210+
- name: node-driver-registrar
211+
...
212+
- name: liveness-probe
213+
...
214+
- name: cinder-csi-plugin
215+
image: docker.io/k8scloudprovider/cinder-csi-plugin:v1.31.0
216+
args:
217+
- /bin/cinder-csi-plugin
218+
- --endpoint=$(CSI_ENDPOINT)
219+
- --cloud-config=$(CLOUD_CONFIG)
220+
- --cloud-name="region-two"
221+
- --additionnal-topology
222+
- topology.kubernetes.io/region=region-two
223+
env:
224+
- name: CSI_ENDPOINT
225+
value: unix://csi/csi.sock
226+
- name: CLOUD_CONFIG
227+
value: /etc/config/cloud.conf
228+
...
229+
volumeMounts:
230+
...
231+
- mountPath: /etc/config
232+
name: secret-cinderplugin
233+
readOnly: true
234+
...
235+
nodeSelector:
236+
topology.kubernetes.io/region: region-two
237+
volumes:
238+
...
239+
- name: secret-cinderplugin
240+
secret:
241+
defaultMode: 420
242+
secretName: cloud-config
243+
...
244+
```
245+
246+
### Configure csi-cinder-controllerplugin deployment
247+
248+
Enable Topology feature-gate on container csi-provisioner of csi-cinder-controllerplugin deployment by adding cli argument ``--feature-gates="Topology=true"
249+
250+
Add cli argument `--cloud-name="region-one"` for each managed openstack cluster, name should match configuration file subsection name, to container `cinder-csi-plugin`.
251+
252+
253+
```yaml
254+
apiVersion: apps/v1
255+
kind: Deployment
256+
metadata:
257+
annotations:
258+
name: csi-cinder-controllerplugin
259+
namespace: kube-system
260+
spec:
261+
selector:
262+
matchLabels:
263+
app: csi-cinder-controllerplugin
264+
template:
265+
metadata:
266+
labels:
267+
app: csi-cinder-controllerplugin
268+
spec:
269+
containers:
270+
- name: csi-provisioner
271+
image: registry.k8s.io/sig-storage/csi-provisioner:v3.0.0
272+
args:
273+
- --csi-address=$(ADDRESS)
274+
- --timeout=3m
275+
- --default-fstype=ext4
276+
- --extra-create-metadata
277+
- --feature-gates
278+
- Topology=true
279+
...
280+
- name: cinder-csi-plugin
281+
image: docker.io/k8scloudprovider/cinder-csi-plugin:v1.31.0
282+
args:
283+
- /bin/cinder-csi-plugin
284+
- --endpoint=$(CSI_ENDPOINT)
285+
- --cloud-config=$(CLOUD_CONFIG)
286+
- --cluster=$(CLUSTER_NAME)
287+
- --cloud-name="region-one"
288+
- --cloud-name="region-two"
289+
env:
290+
- name: CSI_ENDPOINT
291+
value: unix://csi/csi.sock
292+
- name: CLOUD_CONFIG
293+
value: /etc/config/cloud.conf
294+
- name: CLUSTER_NAME
295+
value: kubernetes
296+
volumeMounts:
297+
- mountPath: /etc/config
298+
name: secret-cinderplugin
299+
readOnly: true
300+
...
301+
- name: csi-attacher
302+
...
303+
- name: csi-snapshotter
304+
...
305+
- name: csi-resizer
306+
...
307+
- name: liveness-probe
308+
...
309+
volumes:
310+
- name: secret-cinderplugin
311+
secret:
312+
defaultMode: 420
313+
secretName: cloud-config
314+
...
315+
```
316+

0 commit comments

Comments
 (0)