Skip to content

Commit 420069e

Browse files
umagnusumagnus
committed
add mount storage doc
1 parent 79f696c commit 420069e

File tree

3 files changed

+287
-0
lines changed

3 files changed

+287
-0
lines changed

‎deploy/example/blobfuse-mi/README.md

Lines changed: 233 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,233 @@
1+
# Mount an azure blob storage
2+
3+
In case you have the requirement, that your AKS cluster has to access a blob storage with kubelet identity or a dedicated user-assigned managed identity, the following solution will do this.
4+
5+
You can also use a different managed-identity for different persistent volumes (f.e. you have a pod, that should just have write access to some objects while having another pod, that should have write access everywhere.)
6+
7+
8+
## Before you begin
9+
10+
- The Azure CLI version 2.37.0 or later. Run `az --version` to find the version, and run `az upgrade` to upgrade the version. If you need to install or upgrade, see [Install Azure CLI][install-azure-cli].
11+
12+
- Install the aks-preview Azure CLI extension version 0.5.85 or later.
13+
14+
- Ensure, that you are authenticated or run `az login`
15+
16+
- Run `az account set --subscription "mysubscription"` to select the right subscription
17+
18+
- Create a storage account container(optional in dynamic provisioning), e.g.
19+
```bash
20+
resourcegroup="blobfuse-mi"
21+
storageaccountname="myaksblob"
22+
az storage account create -g "$resourcegroup" -n "$storageaccountname" --access-tier Hot --sku Standard_LRS
23+
az storage container create -n mycontainer --account-name "$storageaccountname" --public-access off
24+
```
25+
26+
## dynamic provisioning in an existing resource group
27+
28+
1. Grant cluster system assigned identity(control plane identity) `Storage Account Contributor` role to resource group, if mount in an existing storage account, then should also grant identities to storage account
29+
30+
1. Grant kubelet identity `Storage Blob Data Owner` role to resource group to mount blob storage, if mount in an existing storage account, then should also grant identity to storage account
31+
32+
1. Create a storage class in an existing resource group
33+
- Option#1 create storage account by CSI driver, will create a new storage account when `storageAccount` and `containerName` are not provided.
34+
- Option#2 use your own storage account, set storage account name for `storageAccount`, you can also set an existing container name for `containerName` if you want to mount an existing container.
35+
```yml
36+
apiVersion: storage.k8s.io/v1
37+
kind: StorageClass
38+
metadata:
39+
name: blob-fuse
40+
provisioner: blob.csi.azure.com
41+
parameters:
42+
skuName: Premium_LRS
43+
protocol: fuse
44+
resourceGroup: EXISTING_RESOURCE_GROUP_NAME
45+
storageAccount: EXISTING_STORAGE_ACCOUNT_NAME # optional, if use existing storage account
46+
containerName: EXISTING_CONTAINER_NAME # optional, if use existing container
47+
AzureStorageAuthType: MSI
48+
AzureStorageIdentityClientID: "xxxxx-xxxx-xxx-xxx-xxxxxxx"
49+
reclaimPolicy: Delete
50+
volumeBindingMode: Immediate
51+
allowVolumeExpansion: true
52+
mountOptions:
53+
- -o allow_other
54+
- --file-cache-timeout-in-seconds=120
55+
- --use-attr-cache=true
56+
- --cancel-list-on-mount-seconds=10 # prevent billing charges on mounting
57+
- -o attr_timeout=120
58+
- -o entry_timeout=120
59+
- -o negative_timeout=120
60+
- --log-level=LOG_WARNING # LOG_WARNING, LOG_INFO, LOG_DEBUG
61+
- --cache-size-mb=1000 # Default will be 80% of available memory, eviction will happen beyond that.
62+
```
63+
64+
1. Create application
65+
- Create a statefulset with volume mount
66+
```console
67+
kubectl create -f https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/deploy/example/statefulset.yaml
68+
```
69+
70+
- Execute `df -h` command in the container
71+
```console
72+
kubectl exec -it statefulset-blob-0 -- df -h
73+
```
74+
<pre>
75+
Filesystem Size Used Avail Use% Mounted on
76+
...
77+
blobfuse 14G 41M 13G 1% /mnt/blob
78+
...
79+
</pre>
80+
81+
## static provisioning(use an existing storage account)
82+
### Option#1: grant kubelet identity access to storage account
83+
84+
1. Give kubelet identity access to storage account
85+
```bash
86+
aksnprg="$(az aks list -g "$resourcegroup" --query "[?name == '$aksname'].nodeResourceGroup" -o tsv)"
87+
kloid="$(az identity list -g "$aksnprg" --query "[?name == 'blobfuse-mi-agentpool'].principalId" -o tsv)"
88+
said="$(az storage account list -g "$resourcegroup" --query "[?name == '$storageaccountname'].id" -o tsv)"
89+
az role assignment create --assignee-object-id "$kloid" --role "Storage Blob Data Owner" --scope "$said"
90+
```
91+
92+
1. Get the clientID of kubelet identity
93+
```bash
94+
az identity list -g "$resourcegroup" --query "[?name == 'blobfuse-mi-agentpool'].clientId" -o tsv
95+
```
96+
97+
### Option#2: grant a dedicated user-assigned managed identity access to storage account
98+
You can use a dedicated user-assigned managed identity to mount the storage.
99+
100+
1. Create user-assigned managed identity and give access to storage account
101+
```bash
102+
az identity create -n myaksblobmi -g "$resourcegroup"
103+
miioid="$(az identity list -g "$resourcegroup" --query "[?name == 'myaksblobmi'].principalId" -o tsv)"
104+
said="$(az storage account list -g "$resourcegroup" --query "[?name == '$storageaccountname'].id" -o tsv)"
105+
az role assignment create --assignee-object-id "$miioid" --role "Storage Blob Data Owner" --scope "$said"
106+
```
107+
108+
1. Assign the user-assigned managed identity to the AKS vm scale set (system nodepool)
109+
```bash
110+
aksnprg="$(az aks list -g "$resourcegroup" --query "[?name == '$aksname'].nodeResourceGroup" -o tsv)"
111+
aksnp="$(az vmss list -g "$aksnprg" --query "[?starts_with(name, 'aks-nodepool1-')].name" -o tsv)"
112+
miid="$(az identity list -g "$resourcegroup" --query "[?name == 'myaksblobmi'].id" -o tsv)"
113+
az vmss identity assign -g "$aksnprg" -n "$aksnp" --identities "$miid"
114+
```
115+
116+
1. Get the clientID of your user-assigned managed identity
117+
```bash
118+
az identity list -g "$resourcegroup" --query "[?name == 'myaksblobmi'].clientId" -o tsv
119+
```
120+
121+
### Mount the azure blob storage
122+
123+
1. Create storage class
124+
```console
125+
kubectl create -f https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/deploy/example/storageclass-blobfuse.yaml
126+
```
127+
128+
1. Create PV and set clientID for ``AzureStorageIdentityClientID``. Please also check ``resourceGroup`` and ``storageAccount``.
129+
```yml
130+
apiVersion: v1
131+
kind: PersistentVolume
132+
metadata:
133+
name: pv-blob
134+
spec:
135+
capacity:
136+
storage: 10Gi
137+
accessModes:
138+
- ReadWriteMany
139+
persistentVolumeReclaimPolicy: Retain # If set as "Delete" container would be removed after pvc deletion
140+
storageClassName: blob-fuse
141+
mountOptions:
142+
- -o allow_other
143+
- --file-cache-timeout-in-seconds=120
144+
csi:
145+
driver: blob.csi.azure.com
146+
readOnly: false
147+
# make sure this volumeid is unique in the cluster
148+
# `#` is not allowed in self defined volumeHandle
149+
volumeHandle: pv-blob
150+
volumeAttributes:
151+
protocol: fuse
152+
resourceGroup: blobfuse-mi
153+
storageAccount: myaksblob
154+
containerName: mycontainer
155+
AzureStorageAuthType: MSI
156+
AzureStorageIdentityClientID: "xxxxx-xxxx-xxx-xxx-xxxxxxx"
157+
```
158+
159+
1. Create PVC and a deployment with volume mount
160+
```console
161+
kubectl create -f https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/deploy/example/deployment.yaml
162+
# check pod
163+
kubectl get pods
164+
```
165+
166+
## how to add another pv with a dedicated user-assigned identity?
167+
168+
1. Create another user-assigned managed identity and give access to storage account
169+
```bash
170+
az identity create -n myaksblobmi2 -g "$resourcegroup"
171+
miioid="$(az identity list -g "$resourcegroup" --query "[?name == 'myaksblobmi2'].principalId" -o tsv)"
172+
said="$(az storage account list -g "$resourcegroup" --query "[?name == '$storageaccountname'].id" -o tsv)"
173+
az role assignment create --assignee-object-id "$miioid" --role "Storage Blob Data Reader" --scope "$said"
174+
```
175+
176+
1. Assign the user-assigned managed identity to the AKS vm scale set (system nodepool)
177+
```bash
178+
aksnprg="$(az aks list -g "$resourcegroup" --query "[?name == '$aksname'].nodeResourceGroup" -o tsv)"
179+
aksnp="$(az vmss list -g "$aksnprg" --query "[?starts_with(name, 'aks-nodepool1-')].name" -o tsv)"
180+
miid="$(az identity list -g "$resourcegroup" --query "[?name == 'myaksblobmi2'].id" -o tsv)"
181+
az vmss identity assign -g "$aksnprg" -n "$aksnp" --identities "$miid"
182+
```
183+
184+
1. Get the objectID of your user-assigned managed identity
185+
```bash
186+
az identity list -g -g "$resourcegroup" --query "[?name == 'myaksblobmi2'].principalId" -o tsv
187+
```
188+
189+
1. Create storage class
190+
```console
191+
kubectl create -f https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/deploy/example/storageclass-blobfuse.yaml
192+
```
193+
194+
1. Create PV and set objectID for ``AzureStorageIdentityClientID``. \
195+
Please also check ``resourceGroup`` and ``storageAccount``.
196+
```yml
197+
apiVersion: v1
198+
kind: PersistentVolume
199+
metadata:
200+
name: pv-blob
201+
spec:
202+
capacity:
203+
storage: 10Gi
204+
accessModes:
205+
- ReadWriteMany
206+
persistentVolumeReclaimPolicy: Retain # If set as "Delete" container would be removed after pvc deletion
207+
storageClassName: blob-fuse
208+
mountOptions:
209+
- -o allow_other
210+
- --file-cache-timeout-in-seconds=120
211+
csi:
212+
driver: blob.csi.azure.com
213+
readOnly: false
214+
# make sure this volumeid is unique in the cluster
215+
# `#` is not allowed in self defined volumeHandle
216+
volumeHandle: pv-blob
217+
volumeAttributes:
218+
protocol: fuse
219+
resourceGroup: blobfuse-mi
220+
storageAccount: myaksblob
221+
containerName: mycontainer
222+
AzureStorageAuthType: MSI
223+
AzureStorageIdentityClientID: "xxxxx-xxxx-xxx-xxx-xxxxxxx"
224+
```
225+
226+
1. Create PVC
227+
```console
228+
kubectl create -f https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/deploy/example/pvc-blob-csi-static.yaml
229+
# make sure pvc is created and in Bound status after a while
230+
kubectl describe pvc pvc-blob
231+
```
232+
233+
1. Now you can use the persistent volume claim ``pv-blob`` in another deployment.

‎deploy/example/blobfuse-mi/pv-blobfuse-mi.yaml

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
apiVersion: v1
2+
kind: PersistentVolume
3+
metadata:
4+
name: pv-blob
5+
spec:
6+
capacity:
7+
storage: 10Gi
8+
accessModes:
9+
- ReadWriteMany
10+
persistentVolumeReclaimPolicy: Retain # If set as "Delete" container would be removed after pvc deletion
11+
storageClassName: blob-fuse
12+
mountOptions:
13+
- -o allow_other
14+
- --file-cache-timeout-in-seconds=120
15+
csi:
16+
driver: blob.csi.azure.com
17+
readOnly: false
18+
# make sure this volumeid is unique in the cluster
19+
# `#` is not allowed in self defined volumeHandle
20+
volumeHandle: pv-blob
21+
volumeAttributes:
22+
protocol: fuse
23+
resourceGroup: aks-fuseblob-mi
24+
storageAccount: myaksblob
25+
containerName: mycontainer
26+
AzureStorageAuthType: MSI
27+
AzureStorageIdentityClientID: "xxxxxx-xxxx-xxxxxxxxxxx-xxxxxxx-xxxxx"

‎deploy/example/blobfuse-mi/storageclass-blobfuse-mi.yaml

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
---
2+
apiVersion: storage.k8s.io/v1
3+
kind: StorageClass
4+
metadata:
5+
name: blob-fuse
6+
provisioner: blob.csi.azure.com
7+
parameters:
8+
skuName: Premium_LRS
9+
protocol: fuse
10+
resourceGroup: EXISTING_RESOURCE_GROUP_NAME
11+
storageAccount: EXISTING_STORAGE_ACCOUNT_NAME # optional, if use existing storage account
12+
containerName: EXISTING_CONTAINER_NAME # optional, if use existing container
13+
AzureStorageAuthType: MSI
14+
AzureStorageIdentityClientID: "xxxxx-xxxx-xxx-xxx-xxxxxxx"
15+
reclaimPolicy: Delete
16+
volumeBindingMode: Immediate
17+
allowVolumeExpansion: true
18+
mountOptions:
19+
- -o allow_other
20+
- --file-cache-timeout-in-seconds=120
21+
- --use-attr-cache=true
22+
- --cancel-list-on-mount-seconds=10 # prevent billing charges on mounting
23+
- -o attr_timeout=120
24+
- -o entry_timeout=120
25+
- -o negative_timeout=120
26+
- --log-level=LOG_WARNING # LOG_WARNING, LOG_INFO, LOG_DEBUG
27+
- --cache-size-mb=1000 # Default will be 80% of available memory, eviction will happen beyond that.

0 commit comments

Comments
 (0)