Skip to content

Commit d865d60

Browse files
author
Jonathan S. Katz
committed
Continue to add to Tutorial
Added the following sections: - Setup TLS - pgbouncer - High Availability - Disaster Recovery - pg_dump
1 parent d15b89c commit d865d60

File tree

6 files changed

+705
-0
lines changed

6 files changed

+705
-0
lines changed

‎docs/content/tutorial/delete-cluster.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,3 +50,5 @@ We've covered the fundamental lifecycle elements of the PostgreSQL Operator, but
5050

5151
- [Architecture]({{< relref "architecture/_index.md" >}})
5252
- [Common `pgo` Client Tasks]({{< relref "pgo-client/common-tasks.md" >}})
53+
54+
The tutorial will now go into some more advanced topics. Up next, learn how to [secure connections to your PostgreSQL clusters with TLS]({{< relref "tutorial/tls.md" >}}).

‎docs/content/tutorial/disaster-recovery.md‎

Lines changed: 188 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,188 @@
1+
---
2+
title: "Disaster Recovery"
3+
draft: false
4+
weight: 190
5+
---
6+
7+
When using the PostgreSQL Operator, the answer to the question "do you take backups of your database" is automatically "yes!"
8+
9+
The PostgreSQL Operator leverages a pgBackRest repository to facilitate the usage of the pgBackRest features in a PostgreSQL cluster. When a new PostgreSQL cluster is created, it simultaneously creates a pgBackRest repository as described in [creating a PostgreSQL cluster]({{< relref "tutorial/create-cluster.md" >}}) section.
10+
11+
For more information on how disaster recovery in the PostgreSQL Operator works, please see the [disaster recovery architecture]({{< relref "architecture/disaster-recovery.md">}}) section.
12+
13+
## Creating a Backup
14+
15+
The PostgreSQL Operator uses the open source [pgBackRest](https://www.pgbackrest.org) backup and recovery utility for managing backups and PostgreSQL archives. pgBackRest has several types of backups that you can take:
16+
17+
- Full: Back up the entire database
18+
- Differential: Create a backup of everything since the last full back up was taken
19+
- Incremental: Back up everything since the last backup was taken, whether it was full, differential, or incremental
20+
21+
When a new PostgreSQL cluster is provisioned by the PostgreSQL Operator, a full pgBackRest backup is taken by default.
22+
23+
To create a backup, you can run the following command:
24+
25+
```
26+
pgo backup hippo
27+
```
28+
29+
which by default, will create an incremental pgBackRest backup. The reason for this is that the PostgreSQL Operator initially creates a pgBackRest full backup when the cluster is initial provisioned, and pgBackRest will take incremental backups for each subsequent backup until a different backup type is specified.
30+
31+
Most [pgBackRest options](https://pgbackrest.org/command.html#command-backup) are supported and can be passed in by the PostgreSQL Operator via the `--backup-opts` flag.
32+
33+
### Creating a Full Backup
34+
35+
You can create a full backup using the following command:
36+
37+
```
38+
pgo backup hippo --backup-opts="--type=full"
39+
```
40+
41+
### Creating a Differential Backup
42+
43+
You can create a differential backup using the following command:
44+
45+
```
46+
pgo backup hippo --backup-opts="--type=diff"
47+
```
48+
49+
### Creating an Incremental Backup
50+
51+
You can create a differential backup using the following command:
52+
53+
```
54+
pgo backup hippo --backup-opts="--type=incr"
55+
```
56+
57+
An incremental backup is created without specifying any options after a full or differential backup is taken.
58+
59+
### Creating Backups in S3
60+
61+
The PostgreSQL Operator supports creating backups in S3 or any object storage system that uses the S3 protocol. For more information, please read the section on [PostgreSQL Operator Backups with S3]({{< relref "architecture/disaster-recovery.md">}}#using-s3) in the architecture section.
62+
63+
## Set Backup Retention
64+
65+
By default, pgBackRest will allow you to keep on creating backups until you run out of disk space. As such, it may be helpful to manage how many backups are retained.
66+
67+
pgBackRest comes with several flags for managing how backups can be retained:
68+
69+
- `--repo1-retention-full`: how many full backups to retain
70+
- `--repo1-retention-diff`: how many differential backups to retain
71+
- `--repo1-retention-archive`: how many sets of WAL archives to retain alongside the full and differential backups that are retained
72+
73+
For example, to create a full backup and retain the previous 7 full backups, you would execute the following command:
74+
75+
```
76+
pgo backup hippo --backup-opts="--type=full --repo1-retention-full=7"
77+
```
78+
79+
pgBackRest also supports time-based retention. Please [review the pgBackRest documentation for more information](https://pgbackrest.org/command.html#command-backup).
80+
81+
## Schedule Backups
82+
83+
It is good practice to take backups regularly. The PostgreSQL Operator allows you to schedule backups to occur automatically.
84+
85+
The PostgreSQL Operator comes with a scheduler is essentially a [cron](https://en.wikipedia.org/wiki/Cron) server that will run jobs that it is specified. Schedule commands use the cron syntax to set up scheduled tasks.
86+
87+
![PostgreSQL Operator Schedule Backups](/images/postgresql-cluster-dr-schedule.png)
88+
89+
For example, to schedule a full backup once a day at 1am, the following command can be used:
90+
91+
```
92+
pgo create schedule hippo --schedule="0 1 * * *" \
93+
--schedule-type=pgbackrest --pgbackrest-backup-type=full
94+
```
95+
96+
To schedule an incremental backup once every 3 hours:
97+
98+
```
99+
pgo create schedule hippo --schedule="0 */3 * * *" \
100+
--schedule-type=pgbackrest --pgbackrest-backup-type=incr
101+
```
102+
103+
You can also add the backup retention settings to these commands.
104+
105+
## View Backups
106+
107+
You can view all of the available backups in your pgBackRest repository with the `pgo show backup` command:
108+
109+
```
110+
pgo show backup hippo
111+
```
112+
113+
## Restores
114+
115+
The PostgreSQL Operator supports the ability to perform a full restore on a PostgreSQL cluster (i.e. a "clone" or "copy") as well as a point-in-time-recovery. There are two types of ways to restore a cluster:
116+
117+
- Restore to a new cluster using the `--restore-from` flag in the [`pgo create cluster`]({{< relref "/pgo-client/reference/pgo_create_cluster.md" >}}) command. This is effectively a [clone](#clone-a-postgresql-cluster) or a copy.
118+
- Restore in-place using the [`pgo restore`]({{< relref "/pgo-client/reference/pgo_restore.md" >}}) command. Note that this is **destructive**.
119+
120+
It is typically better to perform a restore to a new cluster, particularly when performing a point-in-time-recovery, as it can allow you to more effectively manage your downtime and avoid making undesired changes to your production data.
121+
122+
Additionally, the "restore to a new cluster" technique works so long as you have a pgBackRest repository available: the pgBackRest repository does not need to be attached to an active cluster! For example, if a cluster named `hippo` was deleted as such:
123+
124+
```
125+
pgo delete cluster hippo --keep-backups
126+
```
127+
128+
you can create a new cluster from the backups like so:
129+
130+
```
131+
pgo create cluster datalake --restore-from=hippo
132+
```
133+
134+
Below provides guidance on how to perform a restore to a new PostgreSQL cluster both as a full copy and to a specific point in time. Additionally, it also shows how to restore in place to a specific point in time.
135+
136+
### Restore to a New Cluster (aka "copy" or "clone")
137+
138+
Restoring to a new PostgreSQL cluster allows one to take a backup and create a new PostgreSQL cluster that can run alongside an existing PostgreSQL cluster. There are several scenarios where using this technique is helpful:
139+
140+
- Creating a copy of a PostgreSQL cluster that can be used for other purposes. Another way of putting this is "creating a clone."
141+
- Restore to a point-in-time and inspect the state of the data without affecting the current cluster
142+
143+
and more.
144+
145+
#### Restore Everything
146+
147+
To create a new PostgreSQL cluster from a backup and restore it fully, you can
148+
execute the following command:
149+
150+
```
151+
pgo create cluster datalake --restore-from=hippo
152+
```
153+
154+
#### Partial Restore / Point-in-time-Recovery (PITR)
155+
156+
To create a new PostgreSQL cluster and restore it to specific point-in-time (e.g. before a key table was dropped), you can use the following command, substituting the time that you wish to restore to:
157+
158+
```
159+
pgo create cluster datalake \
160+
--restore-from hippo \
161+
--restore-opts "--type=time --target='2019-12-31 11:59:59.999999+00'"
162+
```
163+
164+
When the restore is complete, the cluster is immediately available for reads and writes. To inspect the data before allowing connections, add pgBackRest's `--target-action=pause` option to the `--restore-opts` parameter.
165+
166+
The PostgreSQL Operator supports the full set of pgBackRest restore options, which can be passed into the `--backup-opts` parameter. For more information, please review the [pgBackRest restore options](https://pgbackrest.org/command.html#command-restore)
167+
168+
### Restore in-place
169+
170+
Restoring a PostgreSQL cluster in-place is a **destructive** action that will perform a recovery on your existing data directory. This is accomplished using the [`pgo restore`]({{< relref "/pgo-client/reference/pgo_restore.md" >}})
171+
command. The most common scenario is to restore the database to a specific point in time.
172+
173+
#### Point-in-time-Recovery (PITR)
174+
175+
The more likely scenario when performing a PostgreSQL cluster restore is to recover to a particular point-in-time (e.g. before a key table was dropped). For example, to restore a cluster to December 31, 2019 at 11:59pm:
176+
177+
```
178+
pgo restore hippo --pitr-target="2019-12-31 11:59:59.999999+00" \
179+
--backup-opts="--type=time"
180+
```
181+
182+
When the restore is complete, the cluster is immediately available for reads and writes. To inspect the data before allowing connections, add pgBackRest's `--target-action=pause` option to the `--backup-opts` parameter.
183+
184+
The PostgreSQL Operator supports the full set of pgBackRest restore options, which can be passed into the `--backup-opts` parameter. For more information, please review the [pgBackRest restore options](https://pgbackrest.org/command.html#command-restore)
185+
186+
## Next Steps
187+
188+
There are cases where you may want to take [logical backups]({{< relref "tutorial/pgdump.md" >}}), aka `pg_dump` / `pg_dumpall`. Let's learn how to do that with the PostgreSQL Operator!

‎docs/content/tutorial/high-availability.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
---
2+
title: "High Availability"
3+
draft: false
4+
weight: 180
5+
---
6+
7+
One of the great things about PostgreSQL is its reliability: it is very stable and typically "just works." However, there are certain things that can happen in the environment that PostgreSQL is deployed in that can affect its uptime, including:
8+
9+
- The database storage disk fails or some other hardware failure occurs
10+
- The network on which the database resides becomes unreachable
11+
- The host operating system becomes unstable and crashes
12+
- A key database file becomes corrupted
13+
- A data center is lost
14+
15+
There may also be downtime events that are due to the normal case of operations, such as performing a minor upgrade, security patching of operating system, hardware upgrade, or other maintenance.
16+
17+
Fortunately, the Crunchy PostgreSQL Operator is prepared for this.
18+
19+
![PostgreSQL Operator High-Availability Overview](/images/postgresql-ha-overview.png)
20+
21+
The Crunchy PostgreSQL Operator supports a distributed-consensus based high-availability (HA) system that keeps its managed PostgreSQL clusters up and running, even if the PostgreSQL Operator disappears. Additionally, it leverages Kubernetes specific features such as [Pod Anti-Affinity](#how-the-crunchy-postgresql-operator-uses-pod-anti-affinity) to limit the surface area that could lead to a PostgreSQL cluster becoming unavailable. The PostgreSQL Operator also supports automatic healing of failed primaries and leverages the efficient pgBackRest "delta restore" method, which eliminates the need to fully reprovision a failed cluster!
22+
23+
This tutorial will cover the "howtos" of high availbility. For more information on the topic, please review the detailed [high availability architecture]({{< relref "architecture/high-availability/_index.md" >}}) section.
24+
25+
## Create a HA PostgreSQL Cluster
26+
27+
High availability is enabled in the PostgreSQL Operator by default so long as you have more than one replica. To create a high availability PostgreSQL cluster, you can execute the following command:
28+
29+
```
30+
pgo create cluster hippo --replica-count=1
31+
```
32+
33+
## Scale a PostgreSQL Cluster
34+
35+
You can scale an existing PostgreSQL cluster to add HA to it by using the [`pgo scale`]({{< relref "pgo-client/reference/pgo_scale.md">}}) command:
36+
37+
```
38+
pgo scale hippo
39+
```
40+
41+
## Scale Down a PostgreSQL Cluster
42+
43+
To scale down a PostgreSQL cluster, you will have to provide a target of which instance you want to scale down. You can do this with the [`pgo scaledown`]({{< relref "pgo-client/reference/pgo_scaledown.md">}}) command:
44+
45+
```
46+
pgo scaledown hippo --query
47+
```
48+
49+
which will yield something similar to:
50+
51+
```
52+
Cluster: hippo
53+
REPLICA STATUS NODE REPLICATION LAG PENDING RESTART
54+
hippo-ojnd running node01 0 MB false
55+
```
56+
57+
Once you have determined which instance you want to scale down, you can run the following command:
58+
59+
```
60+
pgo scaledown hippo --target=hippo-ojnd
61+
```
62+
63+
## Manual Failover
64+
65+
Each PostgreSQL cluster will manage its own availability. If you wish to manually fail over, you will need to use the [`pgo failover`]({{< relref "pgo-client/reference/pgo_failover.md">}}) command. First, determine which instance you want to fail over to:
66+
67+
```
68+
pgo failover hippo --query
69+
```
70+
71+
which will yield something similar to:
72+
73+
```
74+
Cluster: hippo
75+
REPLICA STATUS NODE REPLICATION LAG PENDING RESTART
76+
hippo-ojnd running node01 0 MB false
77+
```
78+
79+
Once you have determine your failover target, you can run the following command:
80+
81+
```
82+
pgo failover hippo --target==hippo-ojnd
83+
```
84+
85+
## Synchronous Replication
86+
87+
If you have a [write sensitive workload and wish to use synchronous replication]({{< relref "architecture/high-availability/_index.md" >}}#synchronous-replication-guarding-against-transactions-loss), you can create your PostgreSQL cluster with synchronous replication turned on:
88+
89+
```
90+
pgo create cluster hippo --sync-replication
91+
```
92+
93+
Please understand the tradeoffs of synchronous replication before using it.
94+
95+
## Pod Anti-Affinity and Node Affinity
96+
97+
To leran how to use pod anti-affinity and node affinity, please refer to the [high availability architecture documentation]({{< relref "architecture/high-availability/_index.md" >}})
98+
99+
## Next Steps
100+
101+
Backups, restores, point-in-time-recoveries: [disaster recovery]({{< relref "architecture/disaster-recovery.md" >}}) is a big topic! We'll learn about you can [perform disaster recovery]({{< relref "tutorial/disaster-recovery.md" >}}) and more in the PostgreSQL Operator.

0 commit comments

Comments
 (0)