Skip to content

Commit 789cd55

Browse files
umagnusumagnus
committed
add mount storage doc
1 parent 79f696c commit 789cd55

File tree

4 files changed

+373
-0
lines changed

4 files changed

+373
-0
lines changed

‎deploy/example/mountstorage/README.md

Lines changed: 276 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,276 @@
1+
# Mount an azure blob storage
2+
3+
In case you have the requirement, that your AKS cluster has to access a blob storage with kubelet identity or a dedicated user-assigned managed indentity, the following solution will do this.
4+
5+
You can also use a differnt managed-identity for different persistent volumes (f.e. you have a pod, that should just have write access to some objects while having another pod, that should have write access everywhere.)
6+
7+
8+
## Before you begin
9+
10+
- The Azure CLI version 2.37.0 or later. Run `az --version` to find the version, and run `az upgrade` to upgrade the version. If you need to install or upgrade, see [Install Azure CLI][install-azure-cli].
11+
12+
- Install the aks-preview Azure CLI extension version 0.5.85 or later.
13+
14+
- Ensure, that you are authenticated or run `az login`
15+
16+
- Run `az account set --subscription "mysubscription"` to select the right subscription
17+
18+
- Create a storage account container and upload file, e.g.
19+
```bash
20+
resourcegroup="aks-fuseblob-mi"
21+
storageaccountname="myaksblob"
22+
az storage account create -g "$resourcegroup" -n "$storageaccountname" --access-tier Hot --sku Standard_LRS
23+
az storage container create -n mycontainer --account-name "$storageaccountname" --public-access off
24+
az storage blob upload \
25+
--account-name myaksblob \
26+
--container-name mycontainer \
27+
--name test.htm \
28+
--file test.htm \
29+
--auth-mode key --account-key "$(az storage account keys list --account-name "$storageaccountname" --query '[0].value' -o tsv)"
30+
```
31+
32+
## Using kubelet identity
33+
34+
1. Give kubelet identity access to storage account
35+
```bash
36+
aksnprg="$(az aks list -g "$resourcegroup" --query "[?name == '$aksname'].nodeResourceGroup" -o tsv)"
37+
kloid="$(az identity list -g "$aksnprg" --query "[?name == 'aks-fuseblob-mi-agentpool'].principalId" -o tsv)"
38+
said="$(az storage account list -g "$resourcegroup" --query "[?name == '$storageaccountname'].id" -o tsv)"
39+
az role assignment create --assignee-object-id "$kloid" --role "Storage Blob Data Owner" --scope "$said"
40+
```
41+
42+
1. Get the clientID of kubelet identity
43+
```bash
44+
az identity list -g "$resourcegroup" --query "[?name == 'aks-fuseblob-mi-agentpool'].clientId" -o tsv
45+
```
46+
47+
## Using a dedicated user-assigned managed indentity
48+
You can use a dedicated user-assigned managed indentity to mount the storage.
49+
50+
1. Create user-assigned managed identity and give access to storage account
51+
```bash
52+
az identity create -n myaksblobmi -g "$resourcegroup"
53+
miioid="$(az identity list -g "$resourcegroup" --query "[?name == 'myaksblobmi'].principalId" -o tsv)"
54+
said="$(az storage account list -g "$resourcegroup" --query "[?name == '$storageaccountname'].id" -o tsv)"
55+
az role assignment create --assignee-object-id "$miioid" --role "Storage Blob Data Owner" --scope "$said"
56+
```
57+
58+
1. Assign the user-assigned managed identity to the AKS vm scale set (system nodepool)
59+
```bash
60+
aksnprg="$(az aks list -g "$resourcegroup" --query "[?name == '$aksname'].nodeResourceGroup" -o tsv)"
61+
aksnp="$(az vmss list -g "$aksnprg" --query "[?starts_with(name, 'aks-nodepool1-')].name" -o tsv)"
62+
miid="$(az identity list -g "$resourcegroup" --query "[?name == 'myaksblobmi'].id" -o tsv)"
63+
az vmss identity assign -g "$aksnprg" -n "$aksnp" --identities "$miid"
64+
```
65+
66+
1. Get the clientID of your user-assigned managed identity
67+
```bash
68+
az identity list -g "$resourcegroup" --query "[?name == 'myaksblobmi'].clientId" -o tsv
69+
```
70+
71+
## Mount the azure blob storage
72+
73+
1. Create a ``volume.yaml`` file and set clientID for ``AzureStorageIdentityClientID``. \
74+
Please also check ``resourceGroup`` and ``storageAccount``.
75+
```yml
76+
apiVersion: v1
77+
kind: PersistentVolume
78+
metadata:
79+
name: pv-blob1
80+
spec:
81+
capacity:
82+
storage: 10Gi
83+
accessModes:
84+
- ReadWriteMany
85+
persistentVolumeReclaimPolicy: Retain # If set as "Delete" container would be removed after pvc deletion
86+
storageClassName: azureblob-fuse-premium
87+
mountOptions:
88+
- -o allow_other
89+
- --file-cache-timeout-in-seconds=120
90+
csi:
91+
driver: blob.csi.azure.com
92+
readOnly: false
93+
# make sure this volumeid is unique in the cluster
94+
# `#` is not allowed in self defined volumeHandle
95+
volumeHandle: pv-blob1
96+
volumeAttributes:
97+
protocol: fuse
98+
resourceGroup: aks-fuseblob-mi
99+
storageAccount: myaksblob
100+
containerName: mycontainer
101+
AzureStorageAuthType: MSI
102+
AzureStorageIdentityClientID: "xxxxx-xxxx-xxx-xxx-xxxxxxx"
103+
104+
---
105+
106+
apiVersion: v1
107+
kind: PersistentVolumeClaim
108+
metadata:
109+
name: pvc-blob1
110+
spec:
111+
accessModes:
112+
- ReadWriteMany
113+
resources:
114+
requests:
115+
storage: 10Gi
116+
volumeName: pv-blob1
117+
storageClassName: azureblob-fuse-premium
118+
```
119+
120+
1. Create a ``deployment.yaml`` file.
121+
```yml
122+
apiVersion: apps/v1
123+
kind: Deployment
124+
metadata:
125+
labels:
126+
app: nginx-app1
127+
name: nginx-app1
128+
spec:
129+
replicas: 1
130+
selector:
131+
matchLabels:
132+
app: nginx-app1
133+
template:
134+
metadata:
135+
labels:
136+
app: nginx-app1
137+
spec:
138+
containers:
139+
- image: mcr.microsoft.com/oss/nginx/nginx:1.19.5
140+
name: webapp
141+
imagePullPolicy: Always
142+
resources: {}
143+
ports:
144+
- containerPort: 80
145+
volumeMounts:
146+
- name: pvc-blob1
147+
mountPath: /usr/share/nginx/html
148+
volumes:
149+
- name: pvc-blob1
150+
persistentVolumeClaim:
151+
claimName: pvc-blob1
152+
status: {}
153+
154+
---
155+
156+
apiVersion: v1
157+
kind: Service
158+
metadata:
159+
name: nginx-app1
160+
labels:
161+
run: nginx-app1
162+
spec:
163+
ports:
164+
- port: 80
165+
protocol: TCP
166+
selector:
167+
app: nginx-app1
168+
type: LoadBalancer
169+
```
170+
171+
1. Apply the yaml files
172+
```bash
173+
# create pv and pvc
174+
kubectl apply -f volume.yaml
175+
# check it
176+
kubectl get pv
177+
kubectl get pvc
178+
179+
# create deployment and service
180+
kubectl apply -f deployment.yaml
181+
# check it
182+
kubectl get pods
183+
```
184+
185+
1. Get the url, that you want to open in the browser
186+
```bash
187+
echo "Please surf to: http://$(kubectl get service --field-selector metadata.name==nginx-app1 -o 'jsonpath={.items[*].status.loadBalancer.ingress[*].ip}')/test.htm"
188+
```
189+
190+
# how to add another pv with a dedicated user-assigned identity?
191+
192+
1. Create another user-assigned managed identity and give access to storage account
193+
```bash
194+
az identity create -n myaksblobmi2 -g "$resourcegroup"
195+
miioid="$(az identity list -g "$resourcegroup" --query "[?name == 'myaksblobmi2'].principalId" -o tsv)"
196+
said="$(az storage account list -g "$resourcegroup" --query "[?name == '$storageaccountname'].id" -o tsv)"
197+
az role assignment create --assignee-object-id "$miioid" --role "Storage Blob Data Reader" --scope "$said"
198+
```
199+
200+
1. Assign the user-assigned managed identity to the AKS vm scale set (system nodepool)
201+
```bash
202+
aksnprg="$(az aks list -g "$resourcegroup" --query "[?name == '$aksname'].nodeResourceGroup" -o tsv)"
203+
aksnp="$(az vmss list -g "$aksnprg" --query "[?starts_with(name, 'aks-nodepool1-')].name" -o tsv)"
204+
miid="$(az identity list -g "$resourcegroup" --query "[?name == 'myaksblobmi2'].id" -o tsv)"
205+
az vmss identity assign -g "$aksnprg" -n "$aksnp" --identities "$miid"
206+
```
207+
208+
1. Get the objectID of your user-assigned managed identity
209+
```bash
210+
az identity list -g -g "$resourcegroup" --query "[?name == 'myaksblobmi2'].principalId" -o tsv
211+
```
212+
213+
1. Create a ``volume2.yaml`` file and set objectID for ``AzureStorageIdentityClientID``. \
214+
Please also check ``resourceGroup`` and ``storageAccount``.
215+
```yml
216+
apiVersion: v1
217+
kind: PersistentVolume
218+
metadata:
219+
name: pv-blob2
220+
spec:
221+
capacity:
222+
storage: 10Gi
223+
accessModes:
224+
- ReadWriteMany
225+
persistentVolumeReclaimPolicy: Retain # If set as "Delete" container would be removed after pvc deletion
226+
storageClassName: azureblob-fuse-premium
227+
mountOptions:
228+
- -o allow_other
229+
- --file-cache-timeout-in-seconds=120
230+
csi:
231+
driver: blob.csi.azure.com
232+
readOnly: false
233+
# make sure this volumeid is unique in the cluster
234+
# `#` is not allowed in self defined volumeHandle
235+
volumeHandle: pv-blob2
236+
volumeAttributes:
237+
protocol: fuse
238+
resourceGroup: aks-fuseblob-mi
239+
storageAccount: myaksblob
240+
containerName: mycontainer
241+
AzureStorageAuthType: MSI
242+
AzureStorageIdentityClientID: "xxxxx-xxxx-xxx-xxx-xxxxxxx"
243+
244+
---
245+
246+
apiVersion: v1
247+
kind: PersistentVolumeClaim
248+
metadata:
249+
name: pvc-blob2
250+
spec:
251+
accessModes:
252+
- ReadWriteMany
253+
resources:
254+
requests:
255+
storage: 10Gi
256+
volumeName: pv-blob2
257+
storageClassName: azureblob-fuse-premium
258+
```
259+
260+
1. Apply the ``volume2.yaml`` file
261+
```bash
262+
# create pv and pvc
263+
kubectl apply -f volume2.yaml
264+
# check it
265+
kubectl get pv
266+
kubectl get pvc
267+
```
268+
269+
1. Now you can use the persistent volume claim ``pv-blob2`` in another deployment.
270+
271+
# Security consideration
272+
You should use at least 2 nodepools (a system and a user node pool) and assign the storage identities just to the user nodepool. Identity needs to be assigned where the pvs are mounted.
273+
You can use tains to protect the user assigned identity to schedule only authorized pods on the nodepool, that requires the pvs.
274+
275+
276+
[install-azure-cli]: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli

‎deploy/example/mountstorage/deployment.yaml

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
labels:
5+
app: nginx-app1
6+
name: nginx-app1
7+
spec:
8+
replicas: 1
9+
selector:
10+
matchLabels:
11+
app: nginx-app1
12+
template:
13+
metadata:
14+
labels:
15+
app: nginx-app1
16+
spec:
17+
containers:
18+
- image: mcr.microsoft.com/oss/nginx/nginx:1.19.5
19+
name: webapp
20+
imagePullPolicy: Always
21+
resources: {}
22+
ports:
23+
- containerPort: 80
24+
volumeMounts:
25+
- name: pvc-blob1
26+
mountPath: /usr/share/nginx/html
27+
volumes:
28+
- name: pvc-blob1
29+
persistentVolumeClaim:
30+
claimName: pvc-blob1
31+
status: {}
32+
33+
---
34+
35+
apiVersion: v1
36+
kind: Service
37+
metadata:
38+
name: nginx-app1
39+
labels:
40+
run: nginx-app1
41+
spec:
42+
ports:
43+
- port: 80
44+
protocol: TCP
45+
selector:
46+
app: nginx-app1
47+
type: LoadBalancer

‎deploy/example/mountstorage/test.htm

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
<html>
2+
<head>
3+
<title>Hello world</title>
4+
</head>
5+
<body>
6+
<h1>Hello world</h1>
7+
</body>
8+
</html>

‎deploy/example/mountstorage/volume.yaml

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
apiVersion: v1
2+
kind: PersistentVolume
3+
metadata:
4+
name: pv-blob1
5+
spec:
6+
capacity:
7+
storage: 10Gi
8+
accessModes:
9+
- ReadWriteMany
10+
persistentVolumeReclaimPolicy: Retain # If set as "Delete" container would be removed after pvc deletion
11+
storageClassName: azureblob-fuse-premium
12+
mountOptions:
13+
- -o allow_other
14+
- --file-cache-timeout-in-seconds=120
15+
csi:
16+
driver: blob.csi.azure.com
17+
readOnly: false
18+
# make sure this volumeid is unique in the cluster
19+
# `#` is not allowed in self defined volumeHandle
20+
volumeHandle: pv-blob1
21+
volumeAttributes:
22+
protocol: fuse
23+
resourceGroup: aks-fuseblob-mi
24+
storageAccount: myaksblob
25+
containerName: mycontainer
26+
AzureStorageAuthType: MSI
27+
AzureStorageIdentityClientID: "xxxxxx-xxxx-xxxxxxxxxxx-xxxxxxx-xxxxx"
28+
29+
---
30+
31+
apiVersion: v1
32+
kind: PersistentVolumeClaim
33+
metadata:
34+
name: pvc-blob1
35+
spec:
36+
accessModes:
37+
- ReadWriteMany
38+
resources:
39+
requests:
40+
storage: 10Gi
41+
volumeName: pv-blob1
42+
storageClassName: azureblob-fuse-premium

0 commit comments

Comments
 (0)