Skip to content

Commit 9fd13fe

Browse files
umagnusumagnus
committed
add mount storage doc
1 parent 79f696c commit 9fd13fe

File tree

3 files changed

+292
-0
lines changed

3 files changed

+292
-0
lines changed

‎deploy/example/blobfuse-mi/README.md

Lines changed: 238 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,238 @@
1+
# Mount an azure blob storage
2+
3+
In case you have the requirement, that your AKS cluster has to access a blob storage with kubelet identity or a dedicated user-assigned managed identity, the following solution will do this.
4+
5+
You can also use a different managed-identity for different persistent volumes (f.e. you have a pod, that should just have write access to some objects while having another pod, that should have write access everywhere.)
6+
7+
8+
## Before you begin
9+
10+
- The Azure CLI version 2.37.0 or later. Run `az --version` to find the version, and run `az upgrade` to upgrade the version. If you need to install or upgrade, see [Install Azure CLI][install-azure-cli].
11+
12+
- Install the aks-preview Azure CLI extension version 0.5.85 or later.
13+
14+
- Ensure, that you are authenticated or run `az login`
15+
16+
- Run `az account set --subscription "mysubscription"` to select the right subscription
17+
18+
- Create a storage account container(optional in dynamic provisioning), e.g.
19+
```bash
20+
resourcegroup="blobfuse-mi"
21+
storageaccountname="myaksblob"
22+
az storage account create -g "$resourcegroup" -n "$storageaccountname" --access-tier Hot --sku Standard_LRS
23+
az storage container create -n mycontainer --account-name "$storageaccountname" --public-access off
24+
```
25+
26+
- Get the clientID for `AzureStorageIdentityClientID`. If you use kubelet identity, the identity name is `blobfuse-mi-agentpool`, and the resourcegroup is node resourcegroup, or you can use a dedicated user-assigned managed identity
27+
```bash
28+
az identity list -g "$resourcegroup" --query "[?name == '$identityname'].clientId" -o tsv
29+
```
30+
31+
## dynamic provisioning in an existing resource group
32+
33+
1. Grant cluster system assigned identity(control plane identity) `Storage Account Contributor` role to resource group, if mount in an existing storage account, then should also grant identities to storage account
34+
35+
1. Grant kubelet identity `Storage Blob Data Owner` role to resource group to mount blob storage, if mount in an existing storage account, then should also grant identity to storage account
36+
37+
1. Create a storage class in an existing resource group
38+
- Option#1 create storage account by CSI driver, will create a new storage account when `storageAccount` and `containerName` are not provided.
39+
- Option#2 use your own storage account, set storage account name for `storageAccount`, you can also set an existing container name for `containerName` if you want to mount an existing container.
40+
```yml
41+
apiVersion: storage.k8s.io/v1
42+
kind: StorageClass
43+
metadata:
44+
name: blob-fuse
45+
provisioner: blob.csi.azure.com
46+
parameters:
47+
skuName: Premium_LRS
48+
protocol: fuse
49+
resourceGroup: EXISTING_RESOURCE_GROUP_NAME
50+
storageAccount: EXISTING_STORAGE_ACCOUNT_NAME # optional, if use existing storage account
51+
containerName: EXISTING_CONTAINER_NAME # optional, if use existing container
52+
AzureStorageAuthType: MSI
53+
AzureStorageIdentityClientID: "xxxxx-xxxx-xxx-xxx-xxxxxxx"
54+
reclaimPolicy: Delete
55+
volumeBindingMode: Immediate
56+
allowVolumeExpansion: true
57+
mountOptions:
58+
- -o allow_other
59+
- --file-cache-timeout-in-seconds=120
60+
- --use-attr-cache=true
61+
- --cancel-list-on-mount-seconds=10 # prevent billing charges on mounting
62+
- -o attr_timeout=120
63+
- -o entry_timeout=120
64+
- -o negative_timeout=120
65+
- --log-level=LOG_WARNING # LOG_WARNING, LOG_INFO, LOG_DEBUG
66+
- --cache-size-mb=1000 # Default will be 80% of available memory, eviction will happen beyond that.
67+
```
68+
69+
1. Create application
70+
- Create a statefulset with volume mount
71+
```console
72+
kubectl create -f https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/deploy/example/statefulset.yaml
73+
```
74+
75+
- Execute `df -h` command in the container
76+
```console
77+
kubectl exec -it statefulset-blob-0 -- df -h
78+
```
79+
<pre>
80+
Filesystem Size Used Avail Use% Mounted on
81+
...
82+
blobfuse 14G 41M 13G 1% /mnt/blob
83+
...
84+
</pre>
85+
86+
## static provisioning(use an existing storage account)
87+
### Option#1: grant kubelet identity access to storage account
88+
89+
1. Give kubelet identity access to storage account
90+
```bash
91+
aksnprg="$(az aks list -g "$resourcegroup" --query "[?name == '$aksname'].nodeResourceGroup" -o tsv)"
92+
kloid="$(az identity list -g "$aksnprg" --query "[?name == 'blobfuse-mi-agentpool'].principalId" -o tsv)"
93+
said="$(az storage account list -g "$resourcegroup" --query "[?name == '$storageaccountname'].id" -o tsv)"
94+
az role assignment create --assignee-object-id "$kloid" --role "Storage Blob Data Owner" --scope "$said"
95+
```
96+
97+
1. Get the clientID of kubelet identity
98+
```bash
99+
az identity list -g "$aksnprg" --query "[?name == 'blobfuse-mi-agentpool'].clientId" -o tsv
100+
```
101+
102+
### Option#2: grant a dedicated user-assigned managed identity access to storage account
103+
You can use a dedicated user-assigned managed identity to mount the storage.
104+
105+
1. Create user-assigned managed identity and give access to storage account
106+
```bash
107+
az identity create -n myaksblobmi -g "$resourcegroup"
108+
miioid="$(az identity list -g "$resourcegroup" --query "[?name == 'myaksblobmi'].principalId" -o tsv)"
109+
said="$(az storage account list -g "$resourcegroup" --query "[?name == '$storageaccountname'].id" -o tsv)"
110+
az role assignment create --assignee-object-id "$miioid" --role "Storage Blob Data Owner" --scope "$said"
111+
```
112+
113+
1. Assign the user-assigned managed identity to the AKS vm scale set (system nodepool)
114+
```bash
115+
aksnprg="$(az aks list -g "$resourcegroup" --query "[?name == '$aksname'].nodeResourceGroup" -o tsv)"
116+
aksnp="$(az vmss list -g "$aksnprg" --query "[?starts_with(name, 'aks-nodepool1-')].name" -o tsv)"
117+
miid="$(az identity list -g "$resourcegroup" --query "[?name == 'myaksblobmi'].id" -o tsv)"
118+
az vmss identity assign -g "$aksnprg" -n "$aksnp" --identities "$miid"
119+
```
120+
121+
1. Get the clientID of your user-assigned managed identity
122+
```bash
123+
az identity list -g "$resourcegroup" --query "[?name == 'myaksblobmi'].clientId" -o tsv
124+
```
125+
126+
### Mount the azure blob storage
127+
128+
1. Create storage class
129+
```console
130+
kubectl create -f https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/deploy/example/storageclass-blobfuse.yaml
131+
```
132+
133+
1. Create PV and set clientID for ``AzureStorageIdentityClientID``. Please also check ``resourceGroup`` and ``storageAccount``.
134+
```yml
135+
apiVersion: v1
136+
kind: PersistentVolume
137+
metadata:
138+
name: pv-blob
139+
spec:
140+
capacity:
141+
storage: 10Gi
142+
accessModes:
143+
- ReadWriteMany
144+
persistentVolumeReclaimPolicy: Retain # If set as "Delete" container would be removed after pvc deletion
145+
storageClassName: blob-fuse
146+
mountOptions:
147+
- -o allow_other
148+
- --file-cache-timeout-in-seconds=120
149+
csi:
150+
driver: blob.csi.azure.com
151+
readOnly: false
152+
# make sure this volumeid is unique in the cluster
153+
# `#` is not allowed in self defined volumeHandle
154+
volumeHandle: pv-blob
155+
volumeAttributes:
156+
protocol: fuse
157+
resourceGroup: blobfuse-mi
158+
storageAccount: myaksblob
159+
containerName: mycontainer
160+
AzureStorageAuthType: MSI
161+
AzureStorageIdentityClientID: "xxxxx-xxxx-xxx-xxx-xxxxxxx"
162+
```
163+
164+
1. Create PVC and a deployment with volume mount
165+
```console
166+
kubectl create -f https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/deploy/example/deployment.yaml
167+
# check pod
168+
kubectl get pods
169+
```
170+
171+
## how to add another pv with a dedicated user-assigned identity?
172+
173+
1. Create another user-assigned managed identity and give access to storage account
174+
```bash
175+
az identity create -n myaksblobmi2 -g "$resourcegroup"
176+
miioid="$(az identity list -g "$resourcegroup" --query "[?name == 'myaksblobmi2'].principalId" -o tsv)"
177+
said="$(az storage account list -g "$resourcegroup" --query "[?name == '$storageaccountname'].id" -o tsv)"
178+
az role assignment create --assignee-object-id "$miioid" --role "Storage Blob Data Reader" --scope "$said"
179+
```
180+
181+
1. Assign the user-assigned managed identity to the AKS vm scale set (system nodepool)
182+
```bash
183+
aksnprg="$(az aks list -g "$resourcegroup" --query "[?name == '$aksname'].nodeResourceGroup" -o tsv)"
184+
aksnp="$(az vmss list -g "$aksnprg" --query "[?starts_with(name, 'aks-nodepool1-')].name" -o tsv)"
185+
miid="$(az identity list -g "$resourcegroup" --query "[?name == 'myaksblobmi2'].id" -o tsv)"
186+
az vmss identity assign -g "$aksnprg" -n "$aksnp" --identities "$miid"
187+
```
188+
189+
1. Get the objectID of your user-assigned managed identity
190+
```bash
191+
az identity list -g -g "$resourcegroup" --query "[?name == 'myaksblobmi2'].principalId" -o tsv
192+
```
193+
194+
1. Create storage class
195+
```console
196+
kubectl create -f https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/deploy/example/storageclass-blobfuse.yaml
197+
```
198+
199+
1. Create PV and set objectID for ``AzureStorageIdentityClientID``. \
200+
Please also check ``resourceGroup`` and ``storageAccount``.
201+
```yml
202+
apiVersion: v1
203+
kind: PersistentVolume
204+
metadata:
205+
name: pv-blob
206+
spec:
207+
capacity:
208+
storage: 10Gi
209+
accessModes:
210+
- ReadWriteMany
211+
persistentVolumeReclaimPolicy: Retain # If set as "Delete" container would be removed after pvc deletion
212+
storageClassName: blob-fuse
213+
mountOptions:
214+
- -o allow_other
215+
- --file-cache-timeout-in-seconds=120
216+
csi:
217+
driver: blob.csi.azure.com
218+
readOnly: false
219+
# make sure this volumeid is unique in the cluster
220+
# `#` is not allowed in self defined volumeHandle
221+
volumeHandle: pv-blob
222+
volumeAttributes:
223+
protocol: fuse
224+
resourceGroup: blobfuse-mi
225+
storageAccount: myaksblob
226+
containerName: mycontainer
227+
AzureStorageAuthType: MSI
228+
AzureStorageIdentityClientID: "xxxxx-xxxx-xxx-xxx-xxxxxxx"
229+
```
230+
231+
1. Create PVC
232+
```console
233+
kubectl create -f https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/deploy/example/pvc-blob-csi-static.yaml
234+
# make sure pvc is created and in Bound status after a while
235+
kubectl describe pvc pvc-blob
236+
```
237+
238+
1. Now you can use the persistent volume claim ``pv-blob`` in another deployment.

‎deploy/example/blobfuse-mi/pv-blobfuse-mi.yaml

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
apiVersion: v1
2+
kind: PersistentVolume
3+
metadata:
4+
name: pv-blob
5+
spec:
6+
capacity:
7+
storage: 10Gi
8+
accessModes:
9+
- ReadWriteMany
10+
persistentVolumeReclaimPolicy: Retain # If set as "Delete" container would be removed after pvc deletion
11+
storageClassName: blob-fuse
12+
mountOptions:
13+
- -o allow_other
14+
- --file-cache-timeout-in-seconds=120
15+
csi:
16+
driver: blob.csi.azure.com
17+
readOnly: false
18+
# make sure this volumeid is unique in the cluster
19+
# `#` is not allowed in self defined volumeHandle
20+
volumeHandle: pv-blob
21+
volumeAttributes:
22+
protocol: fuse
23+
resourceGroup: aks-fuseblob-mi
24+
storageAccount: myaksblob
25+
containerName: mycontainer
26+
AzureStorageAuthType: MSI
27+
AzureStorageIdentityClientID: "xxxxxx-xxxx-xxxxxxxxxxx-xxxxxxx-xxxxx"

‎deploy/example/blobfuse-mi/storageclass-blobfuse-mi.yaml

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
---
2+
apiVersion: storage.k8s.io/v1
3+
kind: StorageClass
4+
metadata:
5+
name: blob-fuse
6+
provisioner: blob.csi.azure.com
7+
parameters:
8+
skuName: Premium_LRS
9+
protocol: fuse
10+
resourceGroup: EXISTING_RESOURCE_GROUP_NAME
11+
storageAccount: EXISTING_STORAGE_ACCOUNT_NAME # optional, if use existing storage account
12+
containerName: EXISTING_CONTAINER_NAME # optional, if use existing container
13+
AzureStorageAuthType: MSI
14+
AzureStorageIdentityClientID: "xxxxx-xxxx-xxx-xxx-xxxxxxx"
15+
reclaimPolicy: Delete
16+
volumeBindingMode: Immediate
17+
allowVolumeExpansion: true
18+
mountOptions:
19+
- -o allow_other
20+
- --file-cache-timeout-in-seconds=120
21+
- --use-attr-cache=true
22+
- --cancel-list-on-mount-seconds=10 # prevent billing charges on mounting
23+
- -o attr_timeout=120
24+
- -o entry_timeout=120
25+
- -o negative_timeout=120
26+
- --log-level=LOG_WARNING # LOG_WARNING, LOG_INFO, LOG_DEBUG
27+
- --cache-size-mb=1000 # Default will be 80% of available memory, eviction will happen beyond that.

0 commit comments

Comments
 (0)