Skip to content

Commit a0b3125

Browse files
author
github-actions
committed
docs: import CloudNativePG
1 parent 60c9e24 commit a0b3125

File tree

189 files changed

+36584
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

189 files changed

+36584
-0
lines changed

‎versioned_docs/version-1.28/appendixes/backup_barmanobjectstore.md‎

Lines changed: 353 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,353 @@
1+
---
2+
id: backup_barmanobjectstore
3+
title: Appendix B - Backup on object stores
4+
---
5+
6+
# Appendix B - Backup on object stores
7+
8+
<!-- SPDX-License-Identifier: CC-BY-4.0 -->
9+
10+
!!! Warning
11+
As of CloudNativePG 1.26, **native Barman Cloud support is deprecated** in
12+
favor of the **Barman Cloud Plugin**. This page has been moved to the appendix
13+
for reference purposes. While the native integration remains functional for
14+
now, we strongly recommend beginning a gradual migration to the plugin-based
15+
interface after appropriate testing. For guidance, see
16+
[Migrating from Built-in CloudNativePG Backup](https://cloudnative-pg.io/plugin-barman-cloud/docs/migration/).
17+
18+
CloudNativePG natively supports **online/hot backup** of PostgreSQL
19+
clusters through continuous physical backup and WAL archiving on an object
20+
store. This means that the database is always up (no downtime required)
21+
and that Point In Time Recovery is available.
22+
23+
The operator can orchestrate a continuous backup infrastructure
24+
that is based on the [Barman Cloud](https://pgbarman.org) tool. Instead
25+
of using the classical architecture with a Barman server, which
26+
backs up many PostgreSQL instances, the operator relies on the
27+
`barman-cloud-wal-archive`, `barman-cloud-check-wal-archive`,
28+
`barman-cloud-backup`, `barman-cloud-backup-list`, and
29+
`barman-cloud-backup-delete` tools. As a result, base backups will
30+
be *tarballs*. Both base backups and WAL files can be compressed
31+
and encrypted.
32+
33+
For this, it is required to use an image with `barman-cli-cloud` included.
34+
You can use the image `ghcr.io/cloudnative-pg/postgresql` for this scope,
35+
as it is composed of a community PostgreSQL image and the latest
36+
`barman-cli-cloud` package.
37+
38+
!!! Important
39+
Always ensure that you are running the latest version of the operands
40+
in your system to take advantage of the improvements introduced in
41+
Barman cloud (as well as improve the security aspects of your cluster).
42+
43+
!!! Warning "Changes in Barman Cloud 3.16+ and Bucket Creation"
44+
Starting with Barman Cloud 3.16, most Barman Cloud commands no longer
45+
automatically create the target bucket, assuming it already exists. Only the
46+
`barman-cloud-check-wal-archive` command creates the bucket now. Whenever this
47+
is not the first operation run on an empty bucket, CloudNativePG will throw an
48+
error. As a result, to ensure reliable, future-proof operations and avoid
49+
potential issues, we strongly recommend that you create and configure your
50+
object store bucket *before* creating a `Cluster` resource that references it.
51+
52+
A backup is performed from a primary or a designated primary instance in a
53+
`Cluster` (please refer to
54+
[replica clusters](../replica_cluster.md)
55+
for more information about designated primary instances), or alternatively
56+
on a [standby](../backup.md#backup-from-a-standby).
57+
58+
## Common object stores
59+
60+
If you are looking for a specific object store such as
61+
[AWS S3](object_stores.md#aws-s3),
62+
[Microsoft Azure Blob Storage](object_stores.md#azure-blob-storage),
63+
[Google Cloud Storage](object_stores.md#google-cloud-storage), or a compatible
64+
provider, please refer to [Appendix C - Common object stores for backups](object_stores.md).
65+
66+
## WAL archiving
67+
68+
WAL archiving is the process that feeds a [WAL archive](../backup.md#wal-archive)
69+
in CloudNativePG.
70+
71+
The WAL archive is defined in the `.spec.backup.barmanObjectStore` stanza of
72+
a `Cluster` resource.
73+
74+
!!! Info
75+
Please refer to [`BarmanObjectStoreConfiguration`](https://pkg.go.dev/github.com/cloudnative-pg/barman-cloud/pkg/api#BarmanObjectStoreConfiguration)
76+
in the barman-cloud API for a full list of options.
77+
78+
If required, you can choose to compress WAL files as soon as they
79+
are uploaded and/or encrypt them:
80+
81+
```yaml
82+
apiVersion: postgresql.cnpg.io/v1
83+
kind: Cluster
84+
[...]
85+
spec:
86+
backup:
87+
barmanObjectStore:
88+
[...]
89+
wal:
90+
compression: gzip
91+
encryption: AES256
92+
```
93+
94+
You can configure the encryption directly in your bucket, and the operator
95+
will use it unless you override it in the cluster configuration.
96+
97+
PostgreSQL implements a sequential archiving scheme, where the
98+
`archive_command` will be executed sequentially for every WAL
99+
segment to be archived.
100+
101+
!!! Important
102+
By default, CloudNativePG sets `archive_timeout` to `5min`, ensuring
103+
that WAL files, even in case of low workloads, are closed and archived
104+
at least every 5 minutes, providing a deterministic time-based value for
105+
your Recovery Point Objective ([RPO](../before_you_start.md#rpo)). Even though you change the value
106+
of the [`archive_timeout` setting in the PostgreSQL configuration](https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-ARCHIVE-TIMEOUT),
107+
our experience suggests that the default value set by the operator is
108+
suitable for most use cases.
109+
110+
When the bandwidth between the PostgreSQL instance and the object
111+
store allows archiving more than one WAL file in parallel, you
112+
can use the parallel WAL archiving feature of the instance manager
113+
like in the following example:
114+
115+
```yaml
116+
apiVersion: postgresql.cnpg.io/v1
117+
kind: Cluster
118+
[...]
119+
spec:
120+
backup:
121+
barmanObjectStore:
122+
[...]
123+
wal:
124+
compression: gzip
125+
maxParallel: 8
126+
encryption: AES256
127+
```
128+
129+
In the previous example, the instance manager optimizes the WAL
130+
archiving process by archiving in parallel at most eight ready
131+
WALs, including the one requested by PostgreSQL.
132+
133+
When PostgreSQL will request the archiving of a WAL that has
134+
already been archived by the instance manager as an optimization,
135+
that archival request will be just dismissed with a positive status.
136+
137+
## Retention policies
138+
139+
CloudNativePG can manage the automated deletion of backup files from
140+
the backup object store, using **retention policies** based on the recovery
141+
window.
142+
143+
Internally, the retention policy feature uses `barman-cloud-backup-delete`
144+
with `--retention-policy “RECOVERY WINDOW OF {{ retention policy value }} {{ retention policy unit }}”`.
145+
146+
For example, you can define your backups with a retention policy of 30 days as
147+
follows:
148+
149+
```yaml
150+
apiVersion: postgresql.cnpg.io/v1
151+
kind: Cluster
152+
[...]
153+
spec:
154+
backup:
155+
barmanObjectStore:
156+
destinationPath: "<destination path here>"
157+
s3Credentials:
158+
accessKeyId:
159+
name: aws-creds
160+
key: ACCESS_KEY_ID
161+
secretAccessKey:
162+
name: aws-creds
163+
key: ACCESS_SECRET_KEY
164+
retentionPolicy: "30d"
165+
```
166+
167+
!!! Note "There's more ..."
168+
The **recovery window retention policy** is focused on the concept of
169+
*Point of Recoverability* (`PoR`), a moving point in time determined by
170+
`current time - recovery window`. The *first valid backup* is the first
171+
available backup before `PoR` (in reverse chronological order).
172+
CloudNativePG must ensure that we can recover the cluster at
173+
any point in time between `PoR` and the latest successfully archived WAL
174+
file, starting from the first valid backup. Base backups that are older
175+
than the first valid backup will be marked as *obsolete* and permanently
176+
removed after the next backup is completed.
177+
178+
## Compression algorithms
179+
180+
CloudNativePG by default archives backups and WAL files in an
181+
uncompressed fashion. However, it also supports the following compression
182+
algorithms via `barman-cloud-backup` (for backups) and
183+
`barman-cloud-wal-archive` (for WAL files):
184+
185+
* bzip2
186+
* gzip
187+
* lz4
188+
* snappy
189+
* xz
190+
* zstd
191+
192+
The compression settings for backups and WALs are independent. See the
193+
[DataBackupConfiguration](https://pkg.go.dev/github.com/cloudnative-pg/barman-cloud/pkg/api#DataBackupConfiguration) and
194+
[WALBackupConfiguration](https://pkg.go.dev/github.com/cloudnative-pg/barman-cloud/pkg/api#WalBackupConfiguration) sections in
195+
the barman-cloud API reference.
196+
197+
It is important to note that archival time, restore time, and size change
198+
between the algorithms, so the compression algorithm should be chosen according
199+
to your use case.
200+
201+
The Barman team has performed an evaluation of the performance of the supported
202+
algorithms for Barman Cloud. The following table summarizes a scenario where a
203+
backup is taken on a local MinIO deployment. The Barman GitHub project includes
204+
a [deeper analysis](https://github.com/EnterpriseDB/barman/issues/344#issuecomment-992547396).
205+
206+
| Compression | Backup Time (ms) | Restore Time (ms) | Uncompressed size (MB) | Compressed size (MB) | Approx ratio |
207+
|-------------|------------------|-------------------|------------------------|-----------------------|--------------|
208+
| None | 10927 | 7553 | 395 | 395 | 1:1 |
209+
| bzip2 | 25404 | 13886 | 395 | 67 | 5.9:1 |
210+
| gzip | 116281 | 3077 | 395 | 91 | 4.3:1 |
211+
| snappy | 8134 | 8341 | 395 | 166 | 2.4:1 |
212+
213+
## Tagging of backup objects
214+
215+
Barman 2.18 introduces support for tagging backup resources when saving them in
216+
object stores via `barman-cloud-backup` and `barman-cloud-wal-archive`. As a
217+
result, if your PostgreSQL container image includes Barman with version 2.18 or
218+
higher, CloudNativePG enables you to specify tags as key-value pairs
219+
for backup objects, namely base backups, WAL files and history files.
220+
221+
You can use two properties in the `.spec.backup.barmanObjectStore` definition:
222+
223+
* `tags`: key-value pair tags to be added to backup objects and archived WAL
224+
file in the backup object store
225+
* `historyTags`: key-value pair tags to be added to archived history files in
226+
the backup object store
227+
228+
The excerpt of a YAML manifest below provides an example of usage of this
229+
feature:
230+
231+
```yaml
232+
apiVersion: postgresql.cnpg.io/v1
233+
kind: Cluster
234+
[...]
235+
spec:
236+
backup:
237+
barmanObjectStore:
238+
[...]
239+
tags:
240+
backupRetentionPolicy: "expire"
241+
historyTags:
242+
backupRetentionPolicy: "keep"
243+
```
244+
245+
## Extra options for the backup and WAL commands
246+
247+
You can append additional options to the `barman-cloud-backup` and `barman-cloud-wal-archive` commands by using
248+
the `additionalCommandArgs` property in the
249+
`.spec.backup.barmanObjectStore.data` and `.spec.backup.barmanObjectStore.wal` sections respectively.
250+
These properties are lists of strings that will be appended to the
251+
`barman-cloud-backup` and `barman-cloud-wal-archive` commands.
252+
253+
For example, you can use the `--read-timeout=60` to customize the connection
254+
reading timeout.
255+
256+
For additional options supported by `barman-cloud-backup` and `barman-cloud-wal-archive` commands you can refer to the
257+
official barman documentation [here](https://www.pgbarman.org/documentation/).
258+
259+
If an option provided in `additionalCommandArgs` is already present in the
260+
declared options in its section (`.spec.backup.barmanObjectStore.data` or `.spec.backup.barmanObjectStore.wal`), the extra option will be
261+
ignored.
262+
263+
The following is an example of how to use this property:
264+
265+
For backups:
266+
267+
```yaml
268+
apiVersion: postgresql.cnpg.io/v1
269+
kind: Cluster
270+
[...]
271+
spec:
272+
backup:
273+
barmanObjectStore:
274+
[...]
275+
data:
276+
additionalCommandArgs:
277+
- "--min-chunk-size=5MB"
278+
- "--read-timeout=60"
279+
```
280+
281+
For WAL files:
282+
283+
```yaml
284+
apiVersion: postgresql.cnpg.io/v1
285+
kind: Cluster
286+
[...]
287+
spec:
288+
backup:
289+
barmanObjectStore:
290+
[...]
291+
wal:
292+
additionalCommandArgs:
293+
- "--max-concurrency=1"
294+
- "--read-timeout=60"
295+
```
296+
297+
## Recovery from an object store
298+
299+
You can recover from a backup created by Barman Cloud and stored on a supported
300+
object store. After you define the external cluster, including all the required
301+
configuration in the `barmanObjectStore` section, you need to reference it in
302+
the `.spec.recovery.source` option.
303+
304+
This example defines a recovery object store in a blob container in Azure:
305+
306+
```yaml
307+
apiVersion: postgresql.cnpg.io/v1
308+
kind: Cluster
309+
metadata:
310+
name: cluster-restore
311+
spec:
312+
[...]
313+
314+
superuserSecret:
315+
name: superuser-secret
316+
317+
bootstrap:
318+
recovery:
319+
source: clusterBackup
320+
321+
externalClusters:
322+
- name: clusterBackup
323+
barmanObjectStore:
324+
destinationPath: https://STORAGEACCOUNTNAME.blob.core.windows.net/CONTAINERNAME/
325+
azureCredentials:
326+
storageAccount:
327+
name: recovery-object-store-secret
328+
key: storage_account_name
329+
storageKey:
330+
name: recovery-object-store-secret
331+
key: storage_account_key
332+
wal:
333+
maxParallel: 8
334+
```
335+
336+
The previous example assumes that the application database and its owning user
337+
are named `app` by default. If the PostgreSQL cluster being restored uses
338+
different names, you must specify these names before exiting the recovery phase,
339+
as documented in ["Configure the application database"](../recovery.md#configure-the-application-database).
340+
341+
!!! Important
342+
By default, the `recovery` method strictly uses the `name` of the
343+
cluster in the `externalClusters` section as the name of the main folder
344+
of the backup data within the object store. This name is normally reserved
345+
for the name of the server. You can specify a different folder name
346+
using the `barmanObjectStore.serverName` property.
347+
348+
!!! Note
349+
This example takes advantage of the parallel WAL restore feature,
350+
dedicating up to 8 jobs to concurrently fetch the required WAL files from the
351+
archive. This feature can appreciably reduce the recovery time. Make sure that
352+
you plan ahead for this scenario and correctly tune the value of this parameter
353+
for your environment. It will make a difference when you need it, and you will.

0 commit comments

Comments
 (0)