Velero fails to backup PVC for StatefulSet with error message about DaemonSet

[jheiselman]

Kubernetes Distribution: VMware Tanzu
Kubernetes Version: 1.24
velero version:
client: v1.13.0
server: v1.13.0

Description:
We are running the same version of Kubernetes and velero in all of our environments. In one cluster, I am able to backup PVCs that are bound to a StatefulSet just fine without any errors. In another cluster, velero fails to backup the PVCs and emits an error message indicating that the pod is not in a running state, but it seems to misidentify the pod as being controlled by a DaemonSet when it is actually a StatefulSet.

All PVCs are using the same vmware storage type. The StatefulSets themselves are different with the successful one being Postgresql and the failed one being MariaDB, though, I'm not sure why that would matter.

velero debug bundle:
bundle-2024-07-03-15-46-14.tar.gz

Since this is working in one cluster and not working in another, I don't think it's a bug, but I cannot figure out what is going wrong.

[blackpiglet]

daemonset pod not found in running state in node webteam-webadmin-cl-worker-md-b2jbg-755d4d978f-dvlb9" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:104

This error means the Velero node-agent pod is not running on the worker where the file-system backup is happening.
Please check whether there is any taint on the node that prevents the node-agent running on it.

[jheiselman]

Sorry, just finally getting back to this issue. The node agent pod is running on all worker nodes without issues.

I think the problem may be related to the S3 bucket. I see messages from one of the node-agent pods stating that the repository hasn't been initialized even though it was working previously. So, not a velero problem. I probably just need to recreate the repo and then get velero to check it again.

[hmuendel]

This has been a while, but I came here while trying to solve this problem, so maybe this is helpful for others:
vmware-tanzu/helm-charts#659