Lost access to cross-tenant blob container via Entra Application workload identity after 2 hours

[JoeyC-Dev]

What happened:
When the Pod lifetime reaches around 2 hours, access to blob container lost like below:

Reproducible in the 2nd attempt:

If we delete the Pod and re-create it, it can access the blob container again:

What you expected to happen:
The connection keeps.

How to reproduce it:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: ${namespace}
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ${svc}
  namespace: ${namespace}
EOF

volUniqId=${sa}#${container}#$(tr -dc a-zA-Z0-9 < /dev/urandom | head -c 4)

cat <<EOF | kubectl apply -f -
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azureblob-fuse
provisioner: blob.csi.azure.com
parameters:
  skuName: Standard_LRS
reclaimPolicy: Delete
mountOptions:
  - '-o allow_other'
  - '--file-cache-timeout-in-seconds=120'
  - '--use-attr-cache=true'
  - '--cancel-list-on-mount-seconds=10'
  - '-o attr_timeout=120'
  - '-o entry_timeout=120'
  - '-o negative_timeout=120'
  - '--log-level=LOG_WARNING'
  - '--cache-size-mb=1000'
allowVolumeExpansion: true
volumeBindingMode: Immediate
---
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: blob.csi.azure.com
  name: pv-blob-wi
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: azureblob-fuse
  mountOptions:
    - -o allow_other
    - --file-cache-timeout-in-seconds=120
  csi:
    driver: blob.csi.azure.com
    volumeHandle: ${volUniqId}
    volumeAttributes:
      storageaccount: ${sa}
      containerName: ${container}
      clientID: ${appClientId}
      resourcegroup: ${rG2}
      tenantID: ${tenant2}
      subscriptionid: ${subscription2}
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-blob-wi
  namespace: ${namespace}
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  volumeName: pv-blob-wi
  storageClassName: azureblob-fuse
EOF

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: blobfuse-mount-1
  namespace: ${namespace}
spec:
  serviceAccountName: ${svc}
  containers:
  - name: demo
    image: alpine
    command: ["/bin/sh"]
    args: ["-c", "while true; do cat /mnt/azure/text; sleep 5; done"]
    volumeMounts:
      - mountPath: /mnt/azure
        name: volume
        readOnly: false
  volumes:
   - name: volume
     persistentVolumeClaim:
       claimName: pvc-blob-wi
EOF

Anything else we need to know?:
I am using App registration instead of managed identity here for one-to-multi tenant scenario.
Considered that I do can access data initially, the OIDC/FIDC part should be configured correctly.

Blob CSI Driver won't give any useful log:

kubectl logs csi-blob-node-gk65f -n kube-system -c blob

Environment:

• CSI Driver version: blob-csi:v1.25.5
• Kubernetes version (use kubectl version): 1.31.7
• OS (e.g. from /etc/os-release): AKSUbuntu-2204containerd-202504.16.0

[andyzhangx]

@JoeyC-Dev if you are using sp, why not use AzureStorageAuthType: spn directly in pv config?

# azurestoragespnclientid, azurestoragespntenantid field setting in secret is only supported from v1.21.3
kubectl create secret generic azure-secret --from-literal azurestoragespnclientsecret="xxx" azurestoragespnclientid="xxx" azurestoragespntenantid="xxx" --type=Opaque

https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/deploy/example/pv-blobfuse-auth.yaml

 csi:
    driver: blob.csi.azure.com
    # make sure volumeHandle is unique for every storage blob container in the cluster
    volumeHandle: "{resource-group-name}#{account-name}#{container-name}"
    volumeAttributes:
      resourceGroup: EXISTING_RESOURCE_GROUP_NAME
      storageAccount: EXISTING_STORAGE_ACCOUNT_NAME
      containerName: EXISTING_CONTAINER_NAME
      # refer to https://github.com/Azure/azure-storage-fuse#environment-variables
      AzureStorageAuthType: spn
      AzureStorageSPNClientID:
      AzureStorageSPNTenantID:
    nodeStageSecretRef:  # secret should be stored here, not needed for msi auth type
      name: azure-secret
      namespace: default

[JoeyC-Dev]

@andyzhangx If I am not wrong, SPN is relying on credential set in nodeStageSecretRef as it requires azurestoragespnclientsecret, not same as workload identity.

This is workload identity implantation. I want to avoid static credential.
The reason not using "managed identity" is that I will have to set FIDC 10 times if there are 10 tenants. (And if I need to move AKS to another cluster, it is disaster) Should not be a problem here to use Application?

[andyzhangx]

@JoeyC-Dev current workload identity support does not require account key auth, while it only works for managed identity now, I think we may keep the original behavior if user still wants to use workload identity to retrieve account key and mount with account key.

[JoeyC-Dev]

@andyzhangx What I meant is: using spn is not acceptable as it will use static credential. So I try to apply workload identity onto Application: for multi-tenant set-up.

I am suspecting if the access to blob container will also expire in 2 hours for user assigned managed identity. If using msal for workload identity, there should not be difference when using SP or managed identity. If I was not wrong, AAD team implement FID on SP earlier than on managed identity.

Let me do another test and see if it will reproduce on managed identity - let's see the result in 2 hours.

[JoeyC-Dev]

Okay, the issue does not occur when using managed identity:

So I try to check if this is msal issue:
First, I use msal-go image to pull secrets from Key Vault, by using SP (workload identity) for 2 hours:
https://azure.github.io/azure-workload-identity/docs/quick-start.html
(Note that in workload identity official doc, it provides option to use Entra Application.)
And it looks fine:

So, this expiry issue only occurs when using Entra Application. (And for me, I cannot use account token as we should not use static password. I believe in many organizations there are relative Azure policy to ban that feature.)

I am out of idea what is wrong here. For now, I don't think this is msal dependency issue.
Do you think this is bug to be fixed or bug-as-intended? @andyzhangx

(At least for managed identity, it works fine so no need to revert to gain account token)

[andyzhangx]

@JoeyC-Dev I will add a new parameter(useAccountKeyAuth) for workload identity generated by sp. have you seen the token would be invalid after 24 hours? from our testing, it's not, just make sure that mount is still valid in your env.

[andyzhangx]

fixed by #1988 with revert change in v1.25.6 and v1.26.3