Velero backup partially failed while backing up pv using kopia on AWS EKS

[Latish-chaitanya]

Hi everyone,

I’ve installed Velero in my cluster and set up the Kopia node agent. The backup location is configured and accessible.

However, when I attempt to create a backup, I encounter an error stating:
"Backup completed with status: Partially Failed."

When I describe the backup, I see the following error message:
"Velero: name: /postgres-db-0 message: /Error backing up item error: /daemonset pod not found in running state in node."

I’ve verified that the daemonset is indeed running using the following command:

kubectl get daemonset -n velero

The output shows:

NAME               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kopia-node-agent   1         1         1       1            1           <none>          6d19h

Despite the daemonset running, I’m still facing the issue.

Has anyone encountered this before, or can you point me in the right direction to resolve it?

Thanks in advance for your help!

[blackpiglet]

The node agent DaemonSet may not be able to run pod on every node in the cluster.
The reason could be that the node has some taints that the node agent cannot tolerate.
Please check whether the PVC postgres-db-0 mounting node has the node agent pod running.

[Latish-chaitanya]

I only have a single node, and the kopia-node-agent DaemonSet is running on that node.

[blackpiglet]

Did you make some changes to the Velero node-agent DaemonSet?
The name should be node-agent instead of kopia-node-agent.

This is the code to find the DaemonSet pod in Velero v1.14.1.
It tried to filter pods by the label role=node-agent. Please check whether your kopia-node-agent pod has the needed label.

velero/pkg/nodeagent/node_agent.go

Lines 85 to 113 in 8afe3ce

	// IsRunningInNode checks if the node agent pod is running properly in a specified node. If not, return the error found
	func IsRunningInNode(ctx context.Context, namespace string, nodeName string, crClient ctrlclient.Client) error {
	if nodeName == "" {
	return errors.New("node name is empty")
	}
	pods := new(v1.PodList)
	parsedSelector, err := labels.Parse(fmt.Sprintf("name=%s", daemonSet))
	if err != nil {
	return errors.Wrap(err, "fail to parse selector")
	}
	err = crClient.List(ctx, pods, &ctrlclient.ListOptions{LabelSelector: parsedSelector})
	if err != nil {
	return errors.Wrap(err, "failed to list daemonset pods")
	}
	for i := range pods.Items {
	if kube.IsPodRunning(&pods.Items[i]) != nil {
	continue
	}
	if pods.Items[i].Spec.NodeName == nodeName {
	return nil
	}
	}
	return errors.Errorf("daemonset pod not found in running state in node %s", nodeName)
	}