Successfully Mounted Azure Blob to GKE COS Nodes with Workarounds, Seeking Clarification on enable-blobfuse-proxy=false Impact

[imranrazakhan]

@andyzhangx Thanks for providing hint of init.sh scripts I was able to successfully mount Azure Blob Storage to GKE nodes running Container-Optimized OS (COS), which are read-only at the root filesystem level.

I wanted to share the approach and ask a follow-up questions.

Context
My use case:
GKE nodes running COS (read-only root filesystem)
Need to mount Azure Blob containers to pods

1. First issue of installation of driver got fixed by the PR#2125 (fix: support blobfuse2 install for GKE cos node #2125)

2. After successfull installation, When using --enable-blobfuse-proxy=true in daemenset I encountered the following error:

E0813 11:50:50.359515  518492 nodeserver.go:459] rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exec: "blobfuse2": executable file not found in $PATH , output:

Although at node level i can see

/home/kubernetes/bin $ echo $PATH
/home/kubernetes/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin
/home/kubernetes/bin $ ls -ltrh |grep blob
-rwxr-xr-x 1 root root  30M Aug 10 15:57 blobfuse2
-rwxr-xr-x 1 root root  39M Aug 10 15:57 blobfuse-proxy

I set the --enable-blobfuse-proxy flag to false keeping in view discussion in issue #1830 (#1830), and it was able to mount.

Question
What functionality or benefits will I lose by setting --enable-blobfuse-proxy=false instead of keeping it true?

I want to understand the trade-offs so I can decide if I should:

Keep it false in production, or

Try to fix the blobfuse2 not found in $PATH issue and set it back to true.

[andyzhangx]

if you set --enable-blobfuse-proxy=false, then blobfuse2 mount would be invalid when blob csi driver is restarted.
the blobfuse-proxy just uses os.Environ() to get the $PATH value, so what's the following output on the node if you could build following go code and then run the binary on the node?

package main

import (
	"fmt"
	"os"
)

func main() {
	// Get a slice of key-value pairs representing the environment variables
	envVars := os.Environ()

	// Iterate over the slice and print each key-value pair
	for _, envVar := range envVars {
		fmt.Println(envVar)
	}
}

[imranrazakhan]

I will try to setup go on node and run it, but if i understood correctly server.go code will provide environment of pod not node, we have to change it to get node environment

blob-csi-driver/pkg/blobfuse-proxy/server/server.go

Line 90 in 1a68719

cmd.Env = append(os.Environ(), authEnv...)

[imranrazakhan]

@andyzhangx I run your provided go code on node and following is output

SHELL=/bin/bash
LESS=-R -M --shift 5
CONFIG_PROTECT_MASK=/etc/gentoo-release /etc/ca-certificates.conf
SSH_AUTH_SOCK=/tmp/ssh-XXXXTpyJrH/agent.1508142
EDITOR=/usr/bin/vim
PWD=/tmp
MANPATH=/usr/local/share/man:/usr/share/man
MOTD_SHOWN=pam
LANG=C.UTF-8
LS_COLORS=xxxxxx
SSH_CONNECTION=xxxxx
INFOPATH=/usr/share/info
TERM=xterm-256color
LESSOPEN=|lesspipe %s
SHLVL=1
PAGER=/usr/bin/less
SSH_CLIENT=xxxx
PATH=/home/kubernetes/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/home/imranraza_khan/go/bin

[andyzhangx]

I will try to setup go on node and run it, but if i understood correctly server.go code will provide environment of pod not node, we have to change it to get node environment

blob-csi-driver/pkg/blobfuse-proxy/server/server.go

Line 90 in 1a68719

cmd.Env = append(os.Environ(), authEnv...)

no, blobfuse-proxy is running as a service (process) on the node, so os.Environ() would get the node environment directly. What's the output of blobfuse2 --version if you run this command on the node directly? @imranrazakhan

that's a fix to inherit os env vars: #1607

[imranrazakhan]

@andyzhangx i ran directly on node

imran@gke-xxx $ blobfuse2 --version
blobfuse2 version 2.5.0~preview.2

[andyzhangx]

I have worked out a PR that would provide PATH value in blobfuse-proxy: #2130, and I built the blob csi driver image: andyzhangx/blob-csi:v1.27.0-log based on this PR.

[imranrazakhan]

@andyzhangx I change the image of blob container and tried to recreate pv / pvc and pod, But still same error without detail logging, Please find below detailed logs

I0816 04:41:00.650422  568598 utils.go:104] GRPC call: /csi.v1.Node/NodeStageVolume
I0816 04:41:00.650454  568598 utils.go:105] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/xxx/globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["-o allow_other","--file-cache-timeout-in-seconds=120","--use-adls=true"]}},"access_mode":{"mode":3}},"volume_context":{"AzureStorageAuthType":"spn","AzureStorageSPNClientID":"xxx","AzureStorageSPNTenantID":"xxx","containerName":"default","isHnsEnabled":"true","protocol":"fuse2","resourceGroup":"rg-xxx","storageAccount":"st-xxx"},"volume_id":"rg-xxx#st-xxx#default"}
I0816 04:41:00.651606  568598 blob.go:561] volumeID(rg-xxx#st-xxx#default) authEnv: [AZURE_STORAGE_AUTH_TYPE=spn]
I0816 04:41:00.651640  568598 blob.go:704] storageSPNClientSecret is not empty, use it to access storage account(st-xxx), container(default)
I0816 04:41:00.651658  568598 blob.go:709] storageSPNClientID(xxx) is not empty, use it to access storage account(st-xxx), container(default)
I0816 04:41:00.651677  568598 blob.go:714] storageSPNTenantID(xxx) is not empty, use it to access storage account(st-xxx), container(default)
I0816 04:41:00.651706  568598 nodeserver.go:361] CSI volume is read-only, mounting with extra option ro
I0816 04:41:00.651744  568598 nodeserver.go:436] target /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/xxx/globalmount protocol fuse2 volumeId rg-xxx#st-xxx#default
mountflags [--file-cache-timeout-in-seconds=120 --use-adls=true -o allow_other -o ro]
mountOptions [--file-cache-timeout-in-seconds=120 --use-adls=true -o allow_other -o ro --pre-mount-validate=true --use-https=true --cancel-list-on-mount-seconds=10 --empty-dir-check=false --tmp-path=/mnt/rg-xxx#st-xxx#default --container-name=default] volumeMountGroup
args /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/xxx/globalmount --file-cache-timeout-in-seconds=120 --use-adls=true -o allow_other -o ro --pre-mount-validate=true --use-https=true --cancel-list-on-mount-seconds=10 --empty-dir-check=false --tmp-path=/mnt/rg-xxx#st-xxx#default --container-name=default
serverAddress [st-xxx.blob.core.windows.net](http://st-xxx.blob.core.windows.net/)
I0816 04:41:00.651765  568598 nodeserver.go:166] start connecting to blobfuse proxy, protocol: fuse2, args: /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/xxx/globalmount --file-cache-timeout-in-seconds=120 --use-adls=true -o allow_other -o ro --pre-mount-validate=true --use-https=true --cancel-list-on-mount-seconds=10 --empty-dir-check=false --tmp-path=/mnt/rg-xxx#st-xxx#default --container-name=default
I0816 04:41:00.653732  568598 nodeserver.go:184] begin to mount with blobfuse proxy, protocol: fuse2, args: /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/xxx/globalmount --file-cache-timeout-in-seconds=120 --use-adls=true -o allow_other -o ro --pre-mount-validate=true --use-https=true --cancel-list-on-mount-seconds=10 --empty-dir-check=false --tmp-path=/mnt/rg-xxx#st-xxx#default --container-name=default
E0816 04:41:00.655308  568598 nodeserver.go:187] GRPC call returned with an error:rpc error: code = Unknown desc = exec: "blobfuse2": executable file not found in $PATH
E0816 04:41:00.655592  568598 nodeserver.go:466] rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exec: "blobfuse2": executable file not found in $PATH , output:
Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors.
I0816 04:41:00.655703  568598 azure_metrics.go:105] "Observed Request Latency" latency_seconds=0.00398057 request="blob_csi_driver_node_stage_volume" resource_group="rg-xxx" subscription_id="" source="[blob.csi.azure.com](http://blob.csi.azure.com/)" volumeid="rg-xxx#st-xxx#default" result_code="failed_csi_driver_node_stage_volume"
E0816 04:41:00.655718  568598 utils.go:109] GRPC error: rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exec: "blobfuse2": executable file not found in $PATH , output:
Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors.

Following is describe of DS to provide you options selected by me

$ k describe ds csi-blob-node -n kube-system
Name:           csi-blob-node
Namespace:      kube-system
Selector:       app=csi-blob-node
Node-Selector:  [kubernetes.io/os=linux](http://kubernetes.io/os=linux)
Labels:         <none>
Annotations:    deprecated.daemonset.template.generation: 9
Desired Number of Nodes Scheduled: 3
Current Number of Nodes Scheduled: 3
Number of Nodes Scheduled with Up-to-date Pods: 3
Number of Nodes Scheduled with Available Pods: 3
Number of Nodes Misscheduled: 0
Pods Status:  3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=csi-blob-node
  Annotations:      [kubectl.kubernetes.io/restartedAt](http://kubectl.kubernetes.io/restartedAt): 2025-08-13T13:48:59+02:00
  Service Account:  csi-blob-node-sa
  Init Containers:
   install-blobfuse-proxy:
    Image:      standard-mcr/oss/kubernetes-csi/blob-csi:v1.26.3-cos-fix
    Port:       <none>
    Host Port:  <none>
    Command:
      /blobfuse-proxy/init.sh
    Environment:
      DEBIAN_FRONTEND:         noninteractive
      INSTALL_BLOBFUSE_PROXY:  true
      INSTALL_BLOBFUSE:        false
      BLOBFUSE_VERSION:        1.4.5
      INSTALL_BLOBFUSE2:       true
      BLOBFUSE2_VERSION:       2.4.1
      SET_MAX_OPEN_FILE_NUM:   true
      MAX_FILE_NUM:            9000000
      DISABLE_UPDATEDB:        true
    Mounts:
      /host/etc from host-etc (rw)
      /host/home from host-usr (rw)
      /host/home/local from host-usr-local (rw)
  Containers:
   liveness-probe:
    Image:      /oss/v2/kubernetes-csi/livenessprobe:v2.15.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --csi-address=/csi/csi.sock
      --probe-timeout=10s
      --http-endpoint=localhost:29633
      --v=2
    Limits:
      memory:  100Mi
    Requests:
      cpu:        10m
      memory:     20Mi
    Environment:  <none>
    Mounts:
      /csi from socket-dir (rw)
   node-driver-registrar:
    Image:      oss/v2/kubernetes-csi/csi-node-driver-registrar:v2.13.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --csi-address=$(ADDRESS)
      --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
      --v=2
    Limits:
      memory:  100Mi
    Requests:
      cpu:     10m
      memory:  20Mi
    Environment:
      ADDRESS:               /csi/csi.sock
      DRIVER_REG_SOCK_PATH:  /var/lib/kubelet/plugins/blob.csi.azure.com/csi.sock
    Mounts:
      /csi from socket-dir (rw)
      /registration from registration-dir (rw)
   blob:
    Image:      [docker.io/andyzhangx/blob-csi:v1.27.0-log](http://docker.io/andyzhangx/blob-csi:v1.27.0-log)
    Port:       29635/TCP
    Host Port:  0/TCP
    Args:
      --v=5
      --endpoint=$(CSI_ENDPOINT)
      --enable-blobfuse-proxy=true
      --blobfuse-proxy-endpoint=$(BLOBFUSE_PROXY_ENDPOINT)
      --nodeid=$(KUBE_NODE_NAME)
      --user-agent-suffix=OSS-kubectl
      --metrics-address=0.0.0.0:29635
      --enable-aznfs-mount=false
    Limits:
      memory:  2100Mi
    Requests:
      cpu:     10m
      memory:  20Mi
    Liveness:  http-get http://localhost:29633/healthz delay=30s timeout=30s period=30s #success=1 #failure=5
    Environment:
      AZURE_CREDENTIAL_FILE:    <set to the key 'path' of config map 'azure-cred-file'>  Optional: true
      CSI_ENDPOINT:             unix:///csi/csi.sock
      BLOBFUSE_PROXY_ENDPOINT:  unix:///csi/blobfuse-proxy.sock
      KUBE_NODE_NAME:            (v1:spec.nodeName)
    Mounts:
      /csi from socket-dir (rw)
      /etc/kubernetes/ from azure-cred (rw)
      /lib/modules from lib-modules (ro)
      /mnt from blob-cache (rw)
      /opt/microsoft/aznfs/data from aznfs-data (rw)
      /var/lib/kubelet/ from mountpoint-dir (rw)
   aznfswatchdog:
    Image:      /mcr-csi-provisioner/oss/kubernetes-csi/blob-csi:v1.26.6
    Port:       <none>
    Host Port:  <none>
    Command:
      aznfswatchdog
    Limits:
      memory:  100Mi
    Requests:
      cpu:        10m
      memory:     20Mi
    Environment:  <none>
    Mounts:
      /opt/microsoft/aznfs/data from aznfs-data (rw)
      /var/lib/kubelet/ from mountpoint-dir (rw)
  Volumes:
   host-usr:
    Type:          HostPath (bare host directory volume)
    Path:          /home
    HostPathType:
   host-usr-local:
    Type:          HostPath (bare host directory volume)
    Path:          /home/local
    HostPathType:
   host-etc:
    Type:          HostPath (bare host directory volume)
    Path:          /etc
    HostPathType:
   socket-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/blob.csi.azure.com
    HostPathType:  DirectoryOrCreate
   mountpoint-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/
    HostPathType:  DirectoryOrCreate
   registration-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins_registry/
    HostPathType:  DirectoryOrCreate
   azure-cred:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/
    HostPathType:  DirectoryOrCreate
   blob-cache:
    Type:          HostPath (bare host directory volume)
    Path:          /home/mnt
    HostPathType:  DirectoryOrCreate
   aznfs-data:
    Type:          HostPath (bare host directory volume)
    Path:          /var/microsoft/aznfs/data
    HostPathType:  DirectoryOrCreate
   lib-modules:
    Type:               HostPath (bare host directory volume)
    Path:               /lib/modules
    HostPathType:       DirectoryOrCreate
  Priority Class Name:  system-node-critical
  Node-Selectors:       [kubernetes.io/os=linux](http://kubernetes.io/os=linux)
  Tolerations:          op=Exists
Events:
  Type    Reason            Age   From                  Message
  ----    ------            ----  ----                  -------
  Normal  SuccessfulDelete  38m   daemonset-controller  Deleted pod: csi-blob-node-wqsxb
  Normal  SuccessfulCreate  38m   daemonset-controller  Created pod: csi-blob-node-8d75b
  Normal  SuccessfulDelete  37m   daemonset-controller  Deleted pod: csi-blob-node-fvxbh
  Normal  SuccessfulCreate  37m   daemonset-controller  Created pod: csi-blob-node-hwlfw
  Normal  SuccessfulDelete  37m   daemonset-controller  Deleted pod: csi-blob-node-xbm65
  Normal  SuccessfulCreate  37m   daemonset-controller  Created pod: csi-blob-node-w9szz

[andyzhangx]

pls also change the Image: standard-mcr/oss/kubernetes-csi/blob-csi:v1.26.3-cos-fix in install-blobfuse-proxy dasmonset

[imranrazakhan]

@andyzhangx I have prepared (blob-csi:v1.26.3-cos-fix) custom image on base of following PR, it look like your image doesn't contain this fix, as i start getting error that

#2125

$ k logs -f csi-blob-node-w4pnc -n kube-system -c install-blobfuse-proxy
+ INSTALL_BLOBFUSE_PROXY=true
+ DISABLE_UPDATEDB=true
+ SET_MAX_OPEN_FILE_NUM=true
+ SET_READ_AHEAD_SIZE=true
+ MIGRATE_K8S_REPO=false
+ READ_AHEAD_KB=15380
+ KUBELET_PATH=/var/lib/kubelet
+ ALLOW_PACKAGE_INSTALL_DOWNGRADE=true
+ [ /var/lib/kubelet != /var/lib/kubelet ]
+ HOST_CMD=nsenter --mount=/proc/1/ns/mnt
+ nsenter --mount=/proc/1/ns/mnt cat /etc/os-release
+ grep ^ID=
+ cut -d= -f2
+ tr -d "
+ DISTRIBUTION=cos
+ nsenter --mount=/proc/1/ns/mnt uname -m
+ ARCH=x86_64
+ echo Linux distribution: cos, Arch: x86_64
+ . ./blobfuse-proxy/install-proxy.sh
Linux distribution: cos, Arch: x86_64
+ set -xe
skip install blobfuse/blobfuse2 for cos....
+ [ cos != ubuntu ]
+ echo skip install blobfuse/blobfuse2 for cos....
+ [ true = true ]
+ updateBlobfuseProxy=true
+ [ -f /host/usr/bin/blobfuse-proxy ]
+ [ true = true ]
copy blobfuse-proxy....
+ echo copy blobfuse-proxy....
+ rm -rf /host//var/lib/kubelet/plugins/blob.csi.azure.com/blobfuse-proxy.sock
+ cp /blobfuse-proxy/blobfuse-proxy /host/usr/bin/blobfuse-proxy --force
cp: cannot create regular file '/host/usr/bin/blobfuse-proxy': No such file or directory

[andyzhangx]

try this image: andyzhangx/blob-csi:v1.27.0-log2

[imranrazakhan]

@andyzhangx I tried with new image, it look like go is not able to pick host path correctly, where we can see bash in init-container picked path correctly, or we may have to mount / or if could use nsenter

I0816 08:45:50.049850 1381896 nodeserver.go:166] start connecting to blobfuse proxy, protocol: fuse2, args: /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/xxx/globalmount --file-cache-timeout-in-seconds=120 --use-adls=true -o allow_other -o ro --cancel-list-on-mount-seconds=10 --empty-dir-check=false --tmp-path=/mnt/rg-xx#stxxx#default --container-name=default --pre-mount-validate=true --use-https=true
I0816 08:45:50.050466 1381896 nodeserver.go:184] begin to mount with blobfuse proxy, protocol: fuse2, args: /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/xxx/globalmount --file-cache-timeout-in-seconds=120 --use-adls=true -o allow_other -o ro --cancel-list-on-mount-seconds=10 --empty-dir-check=false --tmp-path=/mnt/rg-xx#stxxx#default --container-name=default --pre-mount-validate=true --use-https=true
E0816 08:45:50.051249 1381896 nodeserver.go:187] GRPC call returned with an error:rpc error: code = Unknown desc = exec: "blobfuse2": executable file not found in $PATH, PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin,
E0816 08:45:50.051407 1381896 nodeserver.go:466] rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exec: "blobfuse2": executable file not found in $PATH, PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin, , output:
Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors.

systemctl status blobfuse-proxy.service

imran@gke-XXX $ systemctl status blobfuse-proxy.service
● blobfuse-proxy.service - Blobfuse proxy service
     Loaded: loaded (/etc/systemd/system/blobfuse-proxy.service; enabled; preset: disabled)
     Active: active (running) since Sat 2025-08-16 08:26:52 UTC; 11h ago
   Main PID: 2729963 (blobfuse-proxy)
      Tasks: 8
     Memory: 7.3M
        CPU: 1.035s
     CGroup: /system.slice/blobfuse-proxy.service
             └─2729963 /home/kubernetes/bin/blobfuse-proxy --v=5 --blobfuse-proxy-endpoint=unix://var/lib/kubelet/plugins/blob.csi.azure.com/blobfuse-proxy.sock

[andyzhangx]

so the blobfuse-proxy.service cannot use $PATH on the node in GKE node while it works on Ubuntu node

[andyzhangx]

what about add Environment=PATH=/etc/kubernetes/bin:$PATH in /etc/systemd/system/blobfuse-proxy.service and then run

systemctl daemon-reload
systemctl restart blobfuse-proxy.service

[Service]
...
Delegate=yes
Environment="PATH=/etc/kubernetes/bin:$PATH"
KillMode=process
Restart=always
OOMScoreAdjust=-999

[imranrazakhan]

@andyzhangx It fixed the PATH issue, but now getting below error, I already mounted /home in volumes, but couldnt pinpoint how to override it

E0817 11:37:14.652781 2499221 nodeserver.go:187] GRPC call returned with an error:rpc error: code = Unknown desc = exit status 1 Error: failed to create default work dir [mkdir /.blobfuse2: read-only file system]
E0817 11:37:14.652939 2499221 nodeserver.go:466] rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 Error: failed to create default work dir [mkdir /.blobfuse2: read-only file system]
, output:

I tried to create pv with extra mountOptions - --tmp-path=/home/mnt/blobfuse2 but still its pointing to /.blobfuse2

apiVersion: v1
kind: PersistentVolume
metadata:
  finalizers:
  - kubernetes.io/pv-protection
  name: pv-blob-fuse
spec:
  accessModes:
  - ReadOnlyMany
  capacity:
    storage: 10Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: pvc-blob
    namespace: adls2gke
  csi:
    driver: blob.csi.azure.com
    nodeStageSecretRef:
      name: azure-secret
      namespace: adls2gke
    volumeAttributes:
      AzureStorageAuthType: spn
      AzureStorageSPNClientID: xx
      AzureStorageSPNTenantID: xx
      containerName: default
      isHnsEnabled: "true"
      protocol: fuse2
      resourceGroup: rg-xx
      storageAccount: stxx
    volumeHandle: rg-xx
  mountOptions:
    - -o allow_other
    - --file-cache-timeout-in-seconds=120
    - --use-adls=true
    - --tmp-path=/var/lib/kubelet/blobfuse2
  persistentVolumeReclaimPolicy: Retain
  storageClassName: blob-fuse
  volumeMode: Filesystem

[imranrazakhan]

Finally i am able to mount with proxy

I0817 16:56:03.825648 3406408 nodeserver.go:139] NodePublishVolume: volume rg-xxx mounting /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/xxx/globalmount at /var/lib/kubelet/pods/xxx/volumes/kubernetes.io~csi/pv-blob-fuse/mount with mountOptions: [bind]
I0817 16:56:03.825675 3406408 mount_linux.go:260] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/xxx/globalmount /var/lib/kubelet/pods/xxx/volumes/kubernetes.io~csi/pv-blob-fuse/mount)
I0817 16:56:03.830356 3406408 mount_linux.go:260] Mounting cmd (mount) with arguments ( -o bind,remount,ro,nodev,nosuid /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/xxx/globalmount /var/lib/kubelet/pods/xxx/volumes/kubernetes.io~csi/pv-blob-fuse/mount)
I0817 16:56:03.832288 3406408 nodeserver.go:155] NodePublishVolume: volume rg-xxx mount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/xxx/globalmount at /var/lib/kubelet/pods/xxx/volumes/kubernetes.io~csi/pv-blob-fuse/mount successfully
I0817 16:56:03.832328 3406408 utils.go:111] GRPC response: {}

Following changes were require in mountoptions of PV

    - --default-working-dir=/home/mnt
    - --tmp-path=/home/mnt/tmp

Working PV is below

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-blob-fuse
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadOnlyMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: blob-fuse
  mountOptions:
    - -o allow_other
    - --file-cache-timeout-in-seconds=120
    - --use-adls=true
    - --default-working-dir=/home/mnt
    - --tmp-path=/home/mnt
  csi:
    driver: [blob.csi.azure.com](http://blob.csi.azure.com/)
    readOnly: false
    volumeHandle: "rg-xxx#default"
    volumeAttributes:
      resourceGroup: rg-xxx
      storageAccount: stxx
      containerName: default
      AzureStorageAuthType: spn
      AzureStorageSPNClientID: xxx
      AzureStorageSPNTenantID: xxx
      isHnsEnabled: "true"
      protocol: fuse2
    nodeStageSecretRef:
      name: azure-secret

@andyzhangx I already saw your PR with Environment fix, Can we merge it to master? I want to use it to avoid lot of patches on my side.