Trident is choking on client rate limiter and context cancel errors

[sergeykuperman]

Describe the bug
our trial cluster is experiencing a wave of "context cancelled" and "client rate limiter Wait returned an error: context canceled" type of errors, volume provisioning and attachment are very slow, with many retries. Please advise on how to troubleshoot this issue.

example logs:

controller.log

I have tried to increase k8sAPIQPS parameter to 300 in tridentorchestrator (we use operator for trident installation), but this does not seem to help.
Cluster currently holds 2600 tridentvolumes.

Environment
We are using trident 25.06, aws fsx ONTAP , "ontap-san-economy" driver
trident controller runs in a gardener cluster.

• Trident version: 25.06
• Trident installation flags used: -n trident
• Container runtime: containerd 2.0
• Kubernetes version: v1.31.13
• Kubernetes orchestrator: Gardener
• Kubernetes enabled feature gates:
• OS: Garden Linux 1877.6
• NetApp backend types: AWS Ontap (linux)
• Other:

To Reproduce
This happens sporadically, and i do not know what exactly is the root cause, but it renders the system almost unusable

Expected behavior
provisioning, attaching and managing happens relatively quickly

Additional context
This happens to one of our productive clusters, (SAP)
We would be grateful for any tips or support on this case.
i have tried to create account here
https://mysupport.netapp.com/site/user/registration
To create support ticket,
but this site seems bugged as well, it does not let me past the email verification step, keeps asking for more codes, and sending them to email, entering the codes does not advance the registration process.

[jdm85nl]

Regarding the context deadline errors, we had the same thing.
We upgraded to version 25.10 and the Context deadline errors where solved:

https://github.com/NetApp/trident/releases/tag/v25.10.0

• Fixed issue where slow storage operations were leading to ContextDeadline errors.

[sergeykuperman]

After upgrading to v25.10.0 the issues with slowness and context cancelled still persist:
events when creating workload with pvc+pv:

Cseastranger@W-5CG3503TSZ:~/repos/esm/trident-chart$ k get events -n ws-ns-workspaces-ws-2sjj4 -w
LAST SEEN   TYPE     REASON              OBJECT                                                 MESSAGE
46s         Normal   Scheduled           pod/workspaces-ws-2sjj4-deployment-5c95c97744-9ml2h    Successfully assigned ws-ns-workspaces-ws-2sjj4/workspaces-ws-2sjj4-deployment-5c95c97744-9ml2h to ip-10-250-151-86.eu-central-1.compute.internal
46s         Normal   SuccessfulCreate    replicaset/workspaces-ws-2sjj4-deployment-5c95c97744   Created pod: workspaces-ws-2sjj4-deployment-5c95c97744-9ml2h
46s         Normal   ScalingReplicaSet   deployment/workspaces-ws-2sjj4-deployment              Scaled up replica set workspaces-ws-2sjj4-deployment-5c95c97744 to 1
48s         Normal   NoPods              poddisruptionbudget/ws-2sjj4-pdb                       No matching pods found
0s          Warning   FailedAttachVolume   pod/workspaces-ws-2sjj4-deployment-5c95c97744-9ml2h    AttachVolume.Attach failed for volume "pp-consume-9bebc0e108f5" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
0s          Warning   FailedAttachVolume   pod/workspaces-ws-2sjj4-deployment-5c95c97744-9ml2h    AttachVolume.Attach failed for volume "pp-consume-9bebc0e108f5" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
0s          Warning   FailedAttachVolume   pod/workspaces-ws-2sjj4-deployment-5c95c97744-9ml2h    AttachVolume.Attach failed for volume "pp-consume-9bebc0e108f5" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
1s          Warning   FailedAttachVolume   pod/workspaces-ws-2sjj4-deployment-5c95c97744-9ml2h    AttachVolume.Attach failed for volume "pp-consume-9bebc0e108f5" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
0s          Warning   FailedAttachVolume   pod/workspaces-ws-2sjj4-deployment-5c95c97744-9ml2h    AttachVolume.Attach failed for volume "pp-consume-9bebc0e108f5" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
0s          Warning   FailedAttachVolume   pod/workspaces-ws-2sjj4-deployment-5c95c97744-9ml2h    AttachVolume.Attach failed for volume "pp-consume-9bebc0e108f5" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
0s          Warning   FailedAttachVolume   pod/workspaces-ws-2sjj4-deployment-5c95c97744-9ml2h    AttachVolume.Attach failed for volume "pp-consume-9bebc0e108f5" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
0s          Warning   FailedAttachVolume   pod/workspaces-ws-2sjj4-deployment-5c95c97744-9ml2h    AttachVolume.Attach failed for volume "pp-consume-9bebc0e108f5" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
0s          Normal    SuccessfulAttachVolume   pod/workspaces-ws-2sjj4-deployment-5c95c97744-9ml2h    AttachVolume.Attach succeeded for volume "pp-consume-9bebc0e108f5"
0s          Warning   FailedMount              pod/workspaces-ws-2sjj4-deployment-5c95c97744-9ml2h    MountVolume.MountDevice failed for volume "pp-consume-9bebc0e108f5" : rpc error: code = Internal desc = could not set LUKS volume passphrase
0s          Warning   FailedToRetrieveImagePullSecret   pod/workspaces-ws-2sjj4-deployment-5c95c97744-9ml2h    Unable to retrieve some image pull secrets (managed-image-pull); attempting to pull the image may not succeed.
0s          Normal    Pulled                            pod/workspaces-ws-2sjj4-deployment-5c95c97744-9ml2h    Container image "build-releases-external.common.cdn.repositories.cloud.sap/istio/proxyv2:1.24.0@sha256:ee6565e57319e01b5e45b929335eb9dc3d4b30d531b4652467e6939ae81b41f7" already present on machine
1s          Normal    Created                           pod/workspaces-ws-2sjj4-deployment-5c95c97744-9ml2h    Created container: istio-port-security
successful events from here on, took more than 8 minutes to provision volume and attach it to pod.

it takes about 3 mins to return response to command tridentctl get volumes -n trident
trident controller logs:

trident.log

i see alot of "context canceled" errors there still.
logs of the trident-node-linux daemonset on the node where the volume was mounted:

trident-node.log

image versions of trident after the upgrade:

seastranger@W-5CG3503TSZ:~/repos/esm/trident-chart$ k get pods trident-controller-79fdfb5d77-gnldp -n trident -o yaml | grep image
    image: build-releases-external.common.cdn.repositories.cloud.sap/netapp/trident:25.10.0@sha256:6d9c7979e3cc9ac62380688cc3190e1166fb8dbd0062ea16fa6c70e2cefaae6e
    imagePullPolicy: IfNotPresent
    image: build-releases-external.common.cdn.repositories.cloud.sap/netapp/trident-autosupport:25.10.0@sha256:b273f7c1922ab2273c9b99f22383be88b6511fcd77f34299e9ecb1f85081fb2b
    imagePullPolicy: IfNotPresent
    image: build-releases-external.common.cdn.repositories.cloud.sap/sig-storage/csi-provisioner:v5.3.0
    imagePullPolicy: IfNotPresent
    image: build-releases-external.common.cdn.repositories.cloud.sap/sig-storage/csi-attacher:v4.10.0
    imagePullPolicy: IfNotPresent
    image: build-releases-external.common.cdn.repositories.cloud.sap/sig-storage/csi-resizer:v1.14.0
    imagePullPolicy: IfNotPresent
    image: build-releases-external.common.cdn.repositories.cloud.sap/sig-storage/csi-snapshotter:v8.3.0

container arguments for the controller:

seastranger@W-5CG3503TSZ:~/repos/esm/trident-chart$ kubectl get pod trident-controller-79fdfb5d77-gnldp -n trident -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}Args: {.args}{"\n\n"}{end}'
trident-main
Args: ["--crd_persistence","--k8s_pod","--https_rest","--https_port=8443","--csi_node_name=$(KUBE_NODE_NAME)","--csi_endpoint=$(CSI_ENDPOINT)","--csi_role=controller","--log_format=text","--log_level=debug","--log_workflows=","--log_layers=","--disable_audit_log=true","--address=127.0.0.1","--http_request_timeout=90s","--enable_force_detach=false","--enable_concurrency=false","--metrics","-debug","--k8s_api_qps=300","--k8s_api_burst=600"]

trident-autosupport
Args: ["--k8s-pod","--log-format=text","--trident-silence-collector=false","-debug"]

csi-provisioner
Args: ["--v=8","--timeout=600s","--csi-address=$(ADDRESS)","--retry-interval-start=8s","--retry-interval-max=30s","--worker-threads=100","--kube-api-qps=300","--kube-api-burst=600"]

csi-attacher
Args: ["--v=8","--timeout=60s","--retry-interval-start=10s","--worker-threads=100","--csi-address=$(ADDRESS)","--kube-api-qps=300","--kube-api-burst=600"]

csi-resizer
Args: ["--v=8","--timeout=300s","--workers=100","--csi-address=$(ADDRESS)","--kube-api-qps=300","--kube-api-burst=600"]

csi-snapshotter
Args: ["--v=8","--timeout=300s","--worker-threads=100","--csi-address=$(ADDRESS)","--kube-api-qps=300","--kube-api-burst=600"]

I notice that the attacher has hardcoded t/o of 1 minute, and it is not configurable via tridentorchestrator, is this by design ?
Would be grateful for further assistance

[sergeykuperman]

the log above was from our staging cluster, in our prod cluster the issue is even worse, provisioning and attaching fail, i see thousands of logs that look as follow:

time="2025-12-21T20:08:19Z" level=debug msg="Volume event." eventType=Normal logLayer=csi_frontend message="could not find PVC with UID 1724508d-6e38-4449-a4b5-e0e8dbfee3bb: PVC 1724508d-6e38-4449-a4b5-e0e8dbfee3bb was not in cache after 30.00 seconds" name=pvc-1724508d-6e38-4449-a4b5-e0e8dbfee3bb reason=ProvisioningFailed requestID=e18b36bb-e34b-4fef-bfea-b388eb563a15 requestSource=CSI workflow="volume=create"

attaching example volume provisioning failure:

vol_log2.txt
and the full log:

trident.log

[wonderland]

Just trying to get a better understanding of the setup and scale. Could you provide some details on

• number of K8s nodes
• number of PVCs k get pvc -A --no-headers | wc -l
• number of PVs (might not be equal to # of PVCs, depending on policy): k get pv --no-headers | wc -l
• number of volumes managed by Trident (might not be identical to # of PVCs) k get tridentvolume -A --no-headers | wc -l
• number of volumesnapshots: k get volumesnapshot -A --no-headers | wc -l

Just in case you can quickly try, maybe run Trident with the new concurrency feature enabled (Techpreview for now)?
https://docs.netapp.com/us-en/trident/trident-get-started/kubernetes-customize-deploy.html#configuration-options
Parameter enableConcurrency

[sergeykuperman]

seastranger@W-5CG3503TSZ:~/repos/esm/landscapes/landscape--trial-aws--us10-trial$ knodes | wc -l
38
seastranger@W-5CG3503TSZ:~/repos/esm/landscapes/landscape--trial-aws--us10-trial$ k get pvc -A  --no-headers | grep netapp | wc -l
2067
seastranger@W-5CG3503TSZ:~/repos/esm/landscapes/landscape--trial-aws--us10-trial$ k get pv -A | grep netapp |wc -l
2074
seastranger@W-5CG3503TSZ:~/repos/esm/landscapes/landscape--trial-aws--us10-trial$ k get tridentvolume -n trident  | wc -l
4105
seastranger@W-5CG3503TSZ:~/repos/esm/landscapes/landscape--trial-aws--us10-trial$ k get volumesnapshot -A | grep trident | wc -l
12

cannot enable concurrency, we use ontap-san-economy driver, which does not support this yet
number of tridentvolumes is greater than the pv count because we have archivation feature in the cluster, where we move workload with it's pvc+pv out of the cluster (patch retain policy to Retain to avoid backend volume deletion) and into s3 bucket.
if needed this pvc+pv can be restored back into cluster (unarchive flow)
If we could organize a zoom meeting or something similar i could present our setup, and answer any further questions,
i have also opened a support ticket in aws regarding this issue.

Now that i think about it, could this mismatch:
alot of tridentvolumes in the cluster, which do not have corresponding pv+pvc pair, and thus not cached by reflector in trident-controller be the thing that causes reconcile cycle to take a lot of time and cause api server spikes and all the "Not found in cache" messages in controller ?

[clintonk]

could this mismatch: a lot of tridentvolumes in the cluster...

Maybe. That shouldn't happen under normal usage. You can use tridentctl to delete the orphaned CRs. Please let us know if that makes a difference.

[sergeykuperman]

could this mismatch: a lot of tridentvolumes in the cluster...

Maybe. That shouldn't happen under normal usage. You can use tridentctl to delete the orphaned CRs. Please let us know if that makes a difference.

they are not actually orphaned, they are needed for the PVC+PV pair to function if they are unarchived
(restored to cluster from s3).
Maybe i could use tridentctl import volume command to recreate the tridentvolume+pvc+pv resources in cluster on unarchive, but this currently does not work with LUKS encryption + ontap-san-economy driver:
#1092
also wont deleting tridentvolume resource delete the underlying LUN in the backend ?
we have to keep our archived volumes recoverable on demand.

[wonderland]

Can you describe your process a bit more? Especially the part that leads to the high number of volumes still kept by Trident but unknown to K8s? In general, if you intend to re-use them, the policy should be retain and then you will have the PV around as well to quickly re-use as a PVC if needed.

I suspected the FSxN volume recovery queue. If you frequently delete larger amounts of volumes, FSxN will keep them for another 12h by default and that could cause such situations as well. But it sounds like you are aware why this large number of volumes is still around and it has a different reason?

[sergeykuperman]

our cluster has large amounts of workloads (we call them workspaces) which are basically a deployment with connected volume (managed by rook-ceph in-cluster system in the old implementation, or by fsx ontap + trident in our new solution we are trying to introduce)
each such workload has a decent amount of resources associated (crds, secrets configmaps etc) and the crds have associated controllers.

largest cluster has around 170,000 of such workloads, 150,000 of them are not used recently.
such big amounts of metadata in a single cluster creates heavy load on the control plane and the ETCD, which can crash, as well as slows down controllers, and watchers in the cluster, if we do not offload (archive) unused metadata for workloads that are not recently used.

we created archive/unarchive mechanism, which allows us to temporarily place all resources associated with a workspace (including pvc+pv pair) in dedicated place in s3 bucket, with ability to restore them back in the cluster on-demand.

The storageclass retain policy is Delete, so when we archive (move to s3) we patch the pv to be Retain instead, then move the pv+pvc to dedicated spot in s3.

When the customer comes back , and wishes to use his workspace (workload) un-archive mechanism is triggered which recreates all needed resources back into the k8s cluster, and the deployment with the associated volume can be quickly spinned up, and used.

This creates state, where tridentvolume is in k8s cluster (known to trident) LUN is still retained in the backend, but pv+pvc are not present in the cluster(stored in s3).

when pv+pvc are restored from s3 into the cluster, the pod can immediately use it. (this is what should happen in the un-archive flow)

ideally we would like to move the tridentvolume away from the cluster as well, (to further decrease cluster amount of resources) when workspace is archived, but afaik there is no supported mechanism in trident to do it, and retain the LUN in the backend (correct me if i am wrong)

[sergeykuperman]

I have several questions of my own, of you don't mind me asking them here:

• does deletion of tridentvolume trigger deletion of the underlying LUN in the backend?
• can i invoke underlying API used by tridentctl delete volume programmatically from go code ? is there an example i could use ?
• do i have to delete all snapshots before attempting to delete volume ? (will deletion be blocked by existing snaps)
• what is the recommended way of deleting all snapshots taken from specific volume? is there tridentctl command/rest api i could use to clean up all volume's snapshots (including in the backend) ?
• if the answer to previous is no, could i perhaps use label selectors to find corresponding VolumeSnapshot objects , patch the corresponding VolumeSnapshotContent to Delete policy, and delete this way ?

Thanks in advance