Skip to content

Commit b8a3ef3

Browse files
author
avandras
committed
Extend multisite doc with howtos
1 parent 574df4c commit b8a3ef3

File tree

1 file changed

+176
-16
lines changed

1 file changed

+176
-16
lines changed

‎docs/multisite.rst‎

Lines changed: 176 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -3,42 +3,202 @@
33
Using Patroni in multisite mode
44
===============================
55

6-
In some circumstances a tiered failover process is desirable where there are distinct sites, and failover across sites needs to be more resilient against temporary outages. For this purpose a special multisite mode is available. In multisite mode each site runs a separate Patroni cluster with its own DCS. To coordinate which site is the primary and which is the standby there is a global DCS for leader site election. In each site the local leader instance is responsible for global leader site election. The site that acquires the leader lock runs Patroni normally, other sites configure themselves as standby clusters.
6+
Introduction
7+
++++++++++++
8+
9+
The multisite mode has been developed to increase resilience of Patroni setups spanning multiple sites against temporary outages. In multisite mode each site runs a separate Patroni cluster with its own DCS, being able to perform leader switches (switchovers and failovers) as usual Patroni clusters. On top of this, in multisite mode here is a global DCS for leader site election, which coordinates which site is the primary and which is the standby. In each site the local leader instance is responsible for global leader site election. The site that acquires the leader lock runs Patroni normally, other sites configure themselves as standby clusters.
710

811
When to use multisite mode
912
--------------------------
1013

11-
If network reliability and bandwidth between sites is good and latency low (<10ms), then most likely multisite mode is not useful. Instead a simple Patroni cluster that spans the two sites will be a simpler and more robust solution.
14+
If network reliability and bandwidth between sites is good and latency low (<10ms), multisite mode is most likely not useful. Instead, a simple Patroni cluster that spans the two sites will be a simpler and more robust solution.
1215

1316
Multisite mode is useful when automatic cross site failover is needed, but the cross site failover needs to be much more resilient against temporary outages. It is also useful when cluster member IP addresses are not globally routable and cross site communication needs to pass through an externally visible proxy address.
1417

15-
Global DCS considerations
16-
-------------------------
18+
DCS considerations
19+
------------------
20+
21+
There are multiple possible ways of setting up DCS for multisite mode, but in every case there are two separate concerns covered. One is the local DCS, which is backing the site-local actions of Patroni. In addition, there is the global DCS, being responsible for keeping track of site state.
22+
23+
Global DCS
24+
~~~~~~~~~~
1725

18-
The multisite deployment will only be as resilient as the global DCS cluster. In case of a typical 3 node etcd cluster, this means that if any 2 nodes share a potential failure point, then that failure will bring the whole multisite cluster into read only mode within the multisite TTL timeout.
26+
The multisite deployment will only be as resilient as the global DCS cluster. DCS has to maintain quorum (more than half of all nodes connected to each other, being able to write the same changes). In case of a typical 3 node DCS cluster, this means quorum is 2, and if any 2 nodes share a potential failure point (e.g. being attached to the same network component), then that failure will bring the whole multisite cluster into read only mode within the multisite TTL timeout (see Configuration below).
1927

20-
For example if there are 2 datacenters, and two of three etcd nodes are in datacenter A, then if the whole datacenter goes offline (e.g. power outage, fire, network connection to datacenter severed) then the other site will not be able to promote. If the other site happened to be leader at that point, it will demote itself to retain safety.
28+
Let's consider an example where there are 2 datacenters, and two of the three DCS nodes are in datacenter A. If the whole datacenter goes offline (e.g. power outage, fire, network connection to datacenter severed) then the other site in datacenter B will not be able to promote. If that site happened to be leader at the pont of the DCS failure, it will demote itself to avoid a split brain situation, thus retaining safety.
2129

22-
In short, this means that to survive a full site outage the system needs to have at least 3 sites. To simplify things, one of the 3 sites is only required to have a single etcd node. If only 2 sites are available, then hosting this third quorum node on public cloud infrastructure is a viable option.
30+
In short, this means that to survive a full site outage the system needs to have at least 3 sites. To simplify things, one of the 3 sites is only required to have a single DCS node. If only 2 sites are available, then hosting this third quorum node on public cloud infrastructure is a viable option.
2331

2432
Here is a typical deployment architecture for using multisite mode:
2533

2634
.. image:: _static/multisite-architecture.png
2735

28-
If the network latencies between sites are very high, then etcd might require special tuning. By default etcd uses a heartbeat interval of 100ms and election timeout of 1s. If round trip time between sites is more than 100ms then those values should be increased.
36+
Cross-site latency
37+
##################
38+
39+
If the network latencies between sites are very high, then DCS might require special tuning. For example, etcd uses a heartbeat interval of 100 ms and election timeout of 1 s by default. If round trip time between sites is more than 100 ms, these values should be increased.
40+
41+
Local DCS
42+
~~~~~~~~~
43+
44+
This is not different from a usual Patroni setup.
45+
46+
47+
48+
Operational how-tos
49+
+++++++++++++++++++
50+
51+
Installation
52+
------------
53+
54+
Linux
55+
~~~~~
56+
57+
Prerequisites
58+
#############
59+
60+
Before starting the installation, Python3 and the matching pip binary have to be installed on the system.
61+
62+
Patroni stores its state and some of its config in a distributed configuration store (DCS). You have to install one of the possible solutions, e.g. etcd 3.5 (https://etcd.io/docs/v3.5/install/).
63+
64+
Installation steps
65+
##################
66+
67+
As systemd is is now the de-facto init system across Linux distributions, we use it in the below steps.
68+
69+
#. Download and unpack source from ...
70+
#. `cd` to the resulting `patroni` directory
71+
#. `pip install -r requirements.txt`
72+
#. `pip install psycopg`
73+
#. create Patroni config (see ...)
74+
#. to run Patroni as a systemd service, create a systemd unit config based on the linked example: https://github.com/patroni/patroni/blob/master/extras/startup-scripts/patroni.service
75+
#. start Patroni with `[sudo] systemctl start patroni`
76+
77+
Windows
78+
~~~~~~~
79+
80+
You can use Cybertec's packaged versions from https://github.com/cybertec-postgresql/patroni-windows-packaging/releases - go for the versions marked 'multisite'. Follow the [Setup Guide](https://github.com/cybertec-postgresql/patroni-windows-packaging/blob/master/doc/setup.md).
81+
82+
If you need, for example, a different PostgreSQL version from what's provided, open a Github issue there, and a new release will soon be prepared.
83+
2984

3085
Configuration
3186
-------------
3287

3388
Configuring multisite mode is done using a top level ``multisite`` section in Patroni configuration file.
3489

35-
- **multisite**:
90+
The configuration is very similar to the usual Patroni config. In fact, the keys and their respective values under `multisite` obey the same rules as those in a conventional configuration.
91+
92+
An example configuration for two Patroni sites:
93+
94+
```yaml
95+
multisite:
96+
name: dc1
97+
namespace: /multisite/
98+
etcd3: # <DCS>
99+
hosts:
100+
# dc1
101+
- 10.0.1.1:2379
102+
- 10.0.1.2:2379
103+
- 10.0.1.3:2379
104+
# dc2
105+
- 10.0.2.1:2379
106+
- 10.0.2.2:2379
107+
- 10.0.2.3:2379
108+
# dc 3
109+
- 10.0.0.1:2379
110+
host: 10.0.1.1,10.0.1.2,10.0.1.3 # How the leader of the other site(s) can connect to the primary on this site
111+
port: 5432
112+
# Multisite failover timeouts
113+
ttl: 90
114+
retry_timeout: 40
115+
```
116+
117+
Details of the configuration parameters
118+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
119+
120+
`name`
121+
: The name of the site. All nodes that share the same value are considered to be a part of the same site, thus it must be different for each site.
122+
`namespace`
123+
: Optional path within DCS where Patroni stores the multisite state. If used, it should be different from the namespace used by the base config, but the same on all sites.
124+
`<DCS>` (in the example `etcd3`)
125+
: The DCS implementation in use. Possible values are `etcd`, `etcd3`, `zookeeper`, `consul`, `exhibitor`, `kubernetes`, or `raft` (the latter is deprecated).
126+
`<DCS>.hosts`
127+
: a list of IP addresses of nodes forming the global DCS cluster, including the extra (tiebreaking) node(s)
128+
`host`
129+
: Comma-separated list of IPs of the Patroni nodes that can become a primary on the present site
130+
`port`
131+
: Postgres port, through which other sites' members can connect to this site. It can be specified once if all nodes use the same port, or as a comma-separated list matching the different port numbers, in the order used in the `host` key.
132+
`ttl`
133+
: Time to live of site leader lock. If the site is unable to elect a functioning leader within this timeout, a different site can take over the leader role. Must be a few times longer than the usual `ttl` value in order to prevent unnecessary site failovers.
134+
`retry_timeout`
135+
: How long the global etcd cluster can be inaccessible before the cluster is demoted. Must be a few times longer than the usual `retry_timeout` value in order to prevent unnecessary site failovers.
136+
137+
Passwords in the YAML configuration
138+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
139+
140+
As all standby sites replicate from the leader, users and their passwords are the same on each Postgres node. Therefore the YAML configuration should specify the same password for each user under `postgresql.authentication`.
141+
142+
143+
Site failover
144+
-------------
145+
146+
In case the multisite leader lock is not updated for at least the time specified by multisite TTL, the standby leader(s) of the other site(s) will try to update the lock. If successful, the standby leader will be promoted to a proper leader. As a result, the Postgres primary instance will be now found in a new site.
147+
148+
Restoring the old leader site after site failover
149+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
150+
151+
Once the problems leading to the site failover are resolved, the old leader site will be able to join the multisite cluster as a standby leader. There is no automatic attempt made for restoring the original order - that is, if desired, switching back to the old leader site must be done manually, via a site switchover.
152+
153+
Connections to the primary
154+
~~~~~~~~~~~~~~~~~~~~~~~~~~
155+
156+
Applications should be ready to try to connect to the new primary. See 'Connecting to a multisite cluster' for more details.
157+
158+
159+
Site switchover
160+
---------------
161+
162+
When circumstances arise that makes it necessary to switch the location of the Postgres primary from one site to another, one could do it by performing a site switchover. Just like a normal switchover, a site switchover can be initiated using `patronictl` (or, alternatively, and API call to the Rest API). The CTL command is as simple as
163+
164+
```
165+
patronictl multisite-switchover
166+
```
167+
168+
The API call could look like the following (replace 'dc2' with the desired site name):
169+
170+
```
171+
curl --data-binary '{ "target_site": "dc2"}' http://127.0.0.1:8008/multisite_switchover
172+
```
173+
174+
Once the site switchover is done, the old leader site will become a standby site automatically.
175+
176+
Connections to the primary
177+
~~~~~~~~~~~~~~~~~~~~~~~~~~
178+
179+
Applications should be ready to try to connect to the new primary. See 'Connecting to a multisite cluster' for more details.
180+
36181

37-
- **name**: Name for the site. All nodes that share the same value are considered to be a part of the same site.
38-
- **namespace**: Optional. Path within DCS where to store multisite state. Helpful to override in testing if the same etcd cluster is used for local and global DCS.
39-
- **etcd**: Same settings as when configuring main DCS.
40-
- **host**: Address that cluster members from the other sites can use to access this node when it is the leader.
41-
- **port**: Port used by other site members when connecting to this site.
42-
- **ttl**: Time to live of site leader lock. If the site is unable to elect a functioning leader within this timeout then a different site can take over the leader role.
43-
- **retry_timeout**: How long can the global etcd cluster be inaccessible before the cluster is demoted.
182+
Glossary
183+
++++++++
44184

185+
DCS
186+
: distributed configuration store
187+
site
188+
: a Patroni cluster with any number of nodes, and the respective DCS - usually corresponding to a data centre
189+
primary
190+
: the writable PostgreSQL node, from which the other nodes replicate their data (either directly or in a cascading fashion)
191+
leader
192+
: the node which other nodes inside the same site replicate from - the leader can be a replica itself, in which case it's called a _standby leader_
193+
site switchover
194+
: a (manual) leader site switch performed when both sites are functioning fine
195+
site failover
196+
: when the main site goes down (meaning there is no Patroni leader and none of the remaining nodes (if any left) can become a leader), the standby leader will be promoted, becoming a leader proper, and the Postgres instance running there becoming the primary
197+
leader site
198+
: the site where the PostgreSQL primary instance is
199+
standby site
200+
: a site replicating from the leader site, and a potential target for site switchover/failover
201+
DCS quorum
202+
: more than half of the DCS nodes are available (and can take part in a leader race)
203+
multisite leader lock
204+
: just like under normal Patroni operation, the leader puts/updates an entry in DCS, thus notifying other sites that there is a functioning Postgres primary running. The entry mentioned is the multisite leader lock.

0 commit comments

Comments
 (0)