Does kopia maintenance block backups?

[yIKX5]

Hello,

I would like to know if kopia full maintenance could block backups from starting.
Recently I observed a kopia full maintenance runnning for more than 2 days. During that maintenance, it seems like the schedule controller still creates backups, but these backups are ignored. Even those simple k8s object backups, that don't need kopia, are not executed.
As a result, the backup count increased during that 2 days, and once the maintenance finished, velero finally start to pick up the backups, and the backup count decrease dramatically.

Is this behavior expected? FYI the velero version is 1.15.2

Thanks!

[blackpiglet]

No. By saying the backups were ignored, do you mean the backups stay in the New phase?

[yIKX5]

why did Velero crash?

It was OOM killed, so the backup failure is expected. It would be another issue, and I will try to increase memory limits.

The main issue that I would like to figure out is why backups are not picked up by velero during kopia full maintenance period. I think if kopia maintenance blocks kopia backups then it makes sense to me, but I am not sure if that is the case.

[yIKX5]

I checked the last full maintenance, and it was a similar situation.
A kopia backup is started after the full maintenance. This kopia backup then hangs for a long time, and the object backups are all ignored, meaning they are in New phase:

velero-backup-obj-20250401154559             New               0        0          <nil>
                 13d                          <none>
velero-backup-obj-20250401153059             New               0        0          <nil>
                 13d                          <none>
velero-backup-obj-20250401151559             New               0        0          <nil>
                 13d                          <none>

After kopia maintenance finishes, the kopia backup soon quited, and the object backups are unblocked.

I remember there was some lock mechanism on backup repo when using restic. Would it be like the kopia maintenance acquired the lock so the kopia backup hang?

[blackpiglet]

No. There isn't such a lock mechanism in Kopia. That's one of the reasons why Velero chose Kopia to replace Restic.
Please check whether there was any backup in InProgress state, or the Velero server was running correctly when the backups were not handled.

[yIKX5]

The kopia backup was in InProgress state. I will try to debug why that backup hangs.

[blackpiglet]

Velero can only handle one backup at the same time.
If there is already one backup in the InProgress state, the other backups will be in New state, until the in-progress one completes or enter the finalizing state.

[yIKX5]

Thanks for your support! It is very helpful for me to understand how velero works.
It is probably kopia is very slow, and then the maintenance and the kopia backup are both slow. And when a kopia backup is InProgress for long time, all the other backups are ignored.
I will try to check why kopia is slow.

[yIKX5]

Hi @blackpiglet ,
I stumbled upon the repoLocker codes, seems like it

• locks the repo when prune repo:

  velero/pkg/repository/manager/manager.go

  Line 150 in d2c6b6b

  m.repoLocker.LockExclusive(repo.Name)

• locks the repo when backup pod volumes:

  velero/pkg/podvolume/backupper.go

  Line 296 in d2c6b6b

  b.repoLocker.Lock(repo.Name)

Will this be the reason that the kopia backup is blocked by the maintenance? How does this repoLocker work?
Thanks!

[kaovilai]

Yes, Kopia maintenance does block backups in Velero

Great investigation! You've correctly identified the key mechanism. Let me explain how the
repoLocker works and why maintenance blocks backups:

How the repoLocker mechanism works:

The repoLocker implements a reader-writer lock pattern to ensure repository consistency:

1. Maintenance operations (like pruning) acquire an exclusive lock on the repository through
   manager.PruneRepo(). This blocks ALL other repository operations.

2. Backup operations need to acquire a non-exclusive lock through
   PrepareRepo() or ConnectToRepo() to access the repository. These operations
   will block and wait when an exclusive lock is held.

3. The locking hierarchy ensures:

   • Multiple backups can run concurrently (they all get read locks)
   • Maintenance blocks everything (requires write lock)
   • When maintenance is running, backups queue up waiting for the lock

Why your backups were blocked:

When Kopia full maintenance was running for 2+ days, it held an exclusive lock on the repository. Any scheduled backups trying to start would:

1. Attempt to acquire a non-exclusive lock via PrepareRepo()
2. Block waiting for the maintenance to complete
3. Remain in "New" state until they could acquire the lock

The exclusive locks are also used for:

• InitRepo: Repository initialization
• Forget: Removing snapshots
• BatchForget: Removing multiple snapshots

This is by design to prevent repository corruption. However, a 2-day maintenance run is unusually long and suggests either:

• Very large repository requiring extensive maintenance
• Performance issues with the storage backend
• Potential bug causing maintenance to hang

Recommendations:

1. Check Velero server logs for maintenance progress/errors
2. Investigate storage backend performance (S3/Azure/GCS)

The blocking behavior is intentional for data safety, but the extended duration indicates an underlying issue that should be investigated.

[Lyndon-Li]

Will this be the reason that the kopia backup is blocked by the maintenance? How does this repoLocker work?

This depends on which Velero version you are using.
From 1.14 on, the repo maintenance moves to a maintenance job, which means the repo maintenance runs in a separate process. And the repoLocker is an inner-process lock, so it actually doesn't blocks backups/restores/snapshot deletion running in Velero server/node-agent/backupPods/restorePods.
Prior to 1.14, the lock blocks start of backups/restores/snapshot deletion from Velero server, but it doesn't block node-agent activities.
repoLocker is actually a legacy thing which helps Restic repo to do the maintenance more safely (but not a must have). After Restic path is fully deprecated in 1.19, this lock will be removed/refactored.

[blackpiglet]

@yIKX5
As Lyndon pointed out, there is a lock mechanism, but after the repository maintenance runs in a separate Job, the lock mechanism doesn't block the backup when the maintenance Job is running.

[yIKX5]

Thank you all for your help! @kaovilai @Lyndon-Li @blackpiglet
Now it makes sense to me that this lock is a legacy thing which cannot lock other jobs. And because there are no multi-threading stuff happening in this scenario, it does nothing.
Now I lean on that it is the object storage blocked somewhere, but it is difficult for me to debug that. I did experience that the object storage (we're using openstack swift) chocked on deleting lots of objects, and kopia maintenance was indeed deleting many objects.