Bootstrap from the backup and set backup in same time

[adns44]

Hi!

I started using the CNPG Barman Cloud Plugin and I faced an issue related to bootstrap a cluster from backup and immediately set the backup to same place.

I have a working CNPG database that is extended with back up to S3.

---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
  name: garage-store
  namespace: harbor
spec:
  configuration:
    destinationPath: s3://cluster-services-cnpg/harbor/pg-harbor
    endpointURL: http://172.31.255.255:3900
    s3Credentials:
      accessKeyId:
        name: cnpg-s3-credentials
        key: key_id
      secretAccessKey:
        name: cnpg-s3-credentials
        key: secret_key
      region:
        name: cnpg-s3-credentials
        key: region
    data:
      compression: "bzip2"
      jobs: 5
    wal:
      compression: bzip2
  retentionPolicy: "2d"
---
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: pg-harbor
  namespace: harbor
spec:
  instances: 3
  bootstrap:
    initdb:
      database: harbor
      owner: harbor
      secret:
        name: pg-harbor-creds
  storage:
    storageClass: database
    size: 5Gi
  plugins:
  - name: barman-cloud.cloudnative-pg.io
    isWALArchiver: true
    parameters:
      barmanObjectName: garage-store

Works well. After bootstrap DB starts and making the backups when shceduled

---
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: backup-pg
  namespace: harbor
spec:
  schedule: "0 0 */3 * * *"
  backupOwnerReference: self
  cluster:
    name: pg-harbor
  method: plugin
  pluginConfiguration:
    name: barman-cloud.cloudnative-pg.io

My problem occurs when I bootstrap the cluster from the backup. As I have previous backup, I'd like to bootstram the cluster from this and continue operation, continue backups to the same location. My previous manifest modified a little.

---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
  name: garage-store
  namespace: harbor
spec:
  configuration:
    destinationPath: s3://cluster-services-cnpg/harbor/pg-harbor
    endpointURL: http://172.31.255.255:3900
    s3Credentials:
      accessKeyId:
        name: cnpg-s3-credentials
        key: key_id
      secretAccessKey:
        name: cnpg-s3-credentials
        key: secret_key
      region:
        name: cnpg-s3-credentials
        key: region
    data:
      compression: "bzip2"
      jobs: 5
    wal:
      compression: bzip2
  retentionPolicy: "2d"
---
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: pg-harbor
  namespace: harbor
spec:
  instances: 3
  bootstrap:
    recovery:
      source: source
  externalClusters:
  - name: source
    plugin:
      name: barman-cloud.cloudnative-pg.io
      parameters:
        barmanObjectName: garage-store
        serverName: pg-harbor
  storage:
    storageClass: database
    size: 5Gi
  plugins:
  - name: barman-cloud.cloudnative-pg.io
    isWALArchiver: true
    parameters:
      barmanObjectName: garage-store

When I check the pod status I got this:

☸ admin@services (harbor) k get pod --watch pg-harbor-1-full-recovery-jkhd7
NAME                              READY   STATUS     RESTARTS   AGE
pg-harbor-1-full-recovery-jkhd7   0/2     Init:0/2   0          4s
pg-harbor-1-full-recovery-jkhd7   0/2     Init:1/2   0          14s
pg-harbor-1-full-recovery-jkhd7   0/2     Init:1/2   0          15s
pg-harbor-1-full-recovery-jkhd7   0/2     PodInitializing   0          24s
pg-harbor-1-full-recovery-jkhd7   1/2     PodInitializing   0          24s
pg-harbor-1-full-recovery-jkhd7   2/2     Running           0          25s
pg-harbor-1-full-recovery-jkhd7   1/2     Error             0          26s
pg-harbor-1-full-recovery-jkhd7   0/2     Error             0          27s
pg-harbor-1-full-recovery-jkhd7   0/2     Error             0          29s
pg-harbor-1-full-recovery-jkhd7   0/2     Error             0          30s
^C
➜ took 29s ☸ admin@services (harbor)

And when I inspect the logs

☸ admin@services (harbor) k logs pg-harbor-1-full-recovery-jkhd7
Defaulted container "full-recovery" out of: full-recovery, bootstrap-controller (init), plugin-barman-cloud (init)
{"level":"info","ts":"2025-12-21T13:58:47.851283563Z","msg":"Starting webserver","logging_pod":"pg-harbor-1-full-recovery","address":"localhost:8010","hasTLS":false}
{"level":"info","ts":"2025-12-21T13:58:47.952621901Z","msg":"Restore through plugin detected, proceeding...","logging_pod":"pg-harbor-1-full-recovery"}
{"level":"error","ts":"2025-12-21T13:58:49.333358679Z","msg":"Error while restoring a backup","logging_pod":"pg-harbor-1-full-recovery","error":"rpc error: code = Unknown desc = unexpected failure invoking barman-cloud-wal-archive: exit status 1","stacktrace":"github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.3.1/pkg/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/instance/restore.restoreSubCommand\n\tinternal/cmd/manager/instance/restore/restore.go:79\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/instance/restore.(*restoreRunnable).Start\n\tinternal/cmd/manager/instance/restore/restore.go:62\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.22.3/pkg/manager/runnable_group.go:260"}
{"level":"info","ts":"2025-12-21T13:58:49.333465791Z","msg":"Stopping and waiting for non leader election runnables"}
{"level":"info","ts":"2025-12-21T13:58:49.333486512Z","msg":"Stopping and waiting for leader election runnables"}
{"level":"info","ts":"2025-12-21T13:58:49.333500334Z","msg":"Stopping and waiting for warmup runnables"}
{"level":"info","ts":"2025-12-21T13:58:49.333550259Z","msg":"Webserver exited","logging_pod":"pg-harbor-1-full-recovery","address":"localhost:8010"}
{"level":"info","ts":"2025-12-21T13:58:49.333564299Z","msg":"Stopping and waiting for caches"}
{"level":"info","ts":"2025-12-21T13:58:49.333623698Z","msg":"Stopping and waiting for webhooks"}
{"level":"info","ts":"2025-12-21T13:58:49.333658061Z","msg":"Stopping and waiting for HTTP servers"}
{"level":"info","ts":"2025-12-21T13:58:49.333687895Z","msg":"Wait completed, proceeding to shutdown the manager"}
{"level":"error","ts":"2025-12-21T13:58:49.333710044Z","msg":"restore error","logging_pod":"pg-harbor-1-full-recovery","error":"while restoring cluster: rpc error: code = Unknown desc = unexpected failure invoking barman-cloud-wal-archive: exit status 1","stacktrace":"github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.3.1/pkg/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/instance/restore.NewCmd.func1\n\tinternal/cmd/manager/instance/restore/cmd.go:101\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.1/command.go:1015\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v1.10.1/command.go:1148\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v1.10.1/command.go:1071\nmain.main\n\tcmd/manager/main.go:71\nruntime.main\n\t/opt/hostedtoolcache/go/1.25.3/x64/src/runtime/proc.go:285"}

 ☸ admin@services (harbor)

If I remove the plugin from the spec field and bootstrap only from the backup and edit it later it works well.

I'd like to ask is this possible to bootstrap from the backup and immediately continue the work? I need this to use with FluxCD GitOps so if it work manual work will not be needed in case of a reinstall or any issue.

[dosubot]

Hi @adns44! I'm Dosu and I’m helping the plugin-barman-cloud team.

You can bootstrap a CNPG cluster from a backup and immediately continue backups to the same S3 location, but there are important requirements and limitations to make this work in a fully automated (GitOps) way.

The error you’re seeing (unexpected failure invoking barman-cloud-wal-archive: exit status 1) happens because CloudNativePG enforces a safety check: the WAL archive location (for the given serverName) must be empty before a new cluster starts archiving to it after restore. This prevents accidental data loss or mixing WALs from different clusters. If you try to restore and then archive to the same location with the same serverName, and the archive isn’t empty, the restore fails by design [docs].

To achieve seamless GitOps automation:

• The archive path must be empty before restore, or you must use a unique serverName for each cluster/restore. If you reuse the same serverName and archive location, you’ll hit this safety check and need manual cleanup.
• If you want to automate everything (no manual steps), consider generating a unique serverName for each restore, or automate cleanup of the WAL archive path before restore.
• There is an annotation (cnpg.io/skipEmptyWalArchiveCheck: "enabled") to bypass the check, but this is risky and not recommended unless you are very confident in your recovery process, as it can lead to data loss [docs].

Other things to check:

• Make sure your ObjectStore and Cluster manifests reference the correct barmanObjectName and serverName.
• For S3/MinIO, set AWS_REQUEST_CHECKSUM_CALCULATION and AWS_RESPONSE_CHECKSUM_VALIDATION to when_required in your ObjectStore’s instanceSidecarConfiguration.env for compatibility.
• If you use parallel WAL restore (wal.maxParallel > 1), be aware of known bugs—set it to 1 for reliable restores [issue].

Summary:
Automated restore and backup to the same location is possible, but only if you ensure the WAL archive path is empty or use a unique serverName for each restore. Otherwise, you’ll need manual intervention, which breaks full GitOps automation [docs].

If you want more details on how to structure your manifests for this workflow, let me know!

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}