Skip to content

Commit 10ee3ce

Browse files
v-albemiv-albemi
authored
Merge pull request #109763 from rachel-msft/edits0331
Logical decoding preview
2 parents 013e1c8 + f5c7d23 commit 10ee3ce

File tree

2 files changed

+149
-0
lines changed

2 files changed

+149
-0
lines changed

‎articles/postgresql/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,8 @@
149149
items:
150150
- name: Read replicas
151151
href: concepts-read-replicas.md
152+
- name: Logical decoding
153+
href: concepts-logical.md
152154
- name: Hyperscale (Citus)
153155
items:
154156
- name: Nodes

‎articles/postgresql/concepts-logical.md

Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
---
2+
title: Logical decoding - Azure Database for PostgreSQL - Single Server
3+
description: Describes logical decoding and wal2json for change data capture in Azure Database for PostgreSQL - Single Server
4+
author: rachel-msft
5+
ms.author: raagyema
6+
ms.service: postgresql
7+
ms.topic: conceptual
8+
ms.date: 03/31/2020
9+
---
10+
11+
# Logical decoding
12+
13+
[Logical decoding in PostgreSQL](https://www.postgresql.org/docs/current/logicaldecoding.html) allows you to stream data changes to external consumers. Logical decoding is popularly used for event streaming and change data capture scenarios.
14+
15+
Logical decoding uses an output plugin to convert Postgres’s write ahead log (WAL) into a readable format. Azure Database for PostgreSQL provides two output plugins: [test_decoding](https://www.postgresql.org/docs/current/test-decoding.html) and [wal2json](https://github.com/eulerto/wal2json).
16+
17+
18+
> [!NOTE]
19+
> Logical decoding is in public preview on Azure Database for PostgreSQL - Single Server.
20+
21+
22+
## Set up your server
23+
To start using logical decoding, enable your server to save and stream the WAL.
24+
25+
1. Set azure.replication_support to `logical` using the Azure CLI.
26+
```
27+
az postgres server configuration set --resource-group mygroup --server-name myserver --name azure.replication_support --value logical
28+
```
29+
30+
> [!NOTE]
31+
> If you use read replicas, azure.replication_support set to `logical` also allows replicas to run. If you stop using logical decoding, change the setting back to `replica`.
32+
33+
34+
2. Restart the server to apply the changes.
35+
```
36+
az postgres server restart --resource-group mygroup --name myserver
37+
```
38+
39+
## Start logical decoding
40+
41+
Logical decoding can be consumed via streaming protocol or SQL interface. Both methods use [replication slots](https://www.postgresql.org/docs/current/logicaldecoding-explanation.html#LOGICALDECODING-REPLICATION-SLOTS). A slot represents a stream of changes from a single database.
42+
43+
Using a replication slot requires Postgres's replication privileges. At this time, the replication privilege is only available for the server's admin user.
44+
45+
### Streaming protocol
46+
Consuming changes using the streaming protocol is often preferable. You can create your own consumer / connector, or use a tool like [Debezium](https://debezium.io/).
47+
48+
Visit the wal2json documentation for [an example using the streaming protocol with pg_recvlogical](https://github.com/eulerto/wal2json#pg_recvlogical).
49+
50+
51+
### SQL interface
52+
In the example below, we use the SQL interface with the wal2json plugin.
53+
54+
1. Create a slot.
55+
```SQL
56+
SELECT * FROM pg_create_logical_replication_slot('test_slot', 'wal2json');
57+
```
58+
59+
2. Issue SQL commands. For example:
60+
```SQL
61+
CREATE TABLE a_table (
62+
id varchar(40) NOT NULL,
63+
item varchar(40),
64+
PRIMARY KEY (id)
65+
);
66+
67+
INSERT INTO a_table (id, item) VALUES ('id1', 'item1');
68+
DELETE FROM a_table WHERE id='id1';
69+
```
70+
71+
3. Consume the changes.
72+
```SQL
73+
SELECT data FROM pg_logical_slot_get_changes('test_slot', NULL, NULL, 'pretty-print', '1');
74+
```
75+
76+
The output will look like:
77+
```
78+
{
79+
"change": [
80+
]
81+
}
82+
{
83+
"change": [
84+
{
85+
"kind": "insert",
86+
"schema": "public",
87+
"table": "a_table",
88+
"columnnames": ["id", "item"],
89+
"columntypes": ["character varying(40)", "character varying(40)"],
90+
"columnvalues": ["id1", "item1"]
91+
}
92+
]
93+
}
94+
{
95+
"change": [
96+
{
97+
"kind": "delete",
98+
"schema": "public",
99+
"table": "a_table",
100+
"oldkeys": {
101+
"keynames": ["id"],
102+
"keytypes": ["character varying(40)"],
103+
"keyvalues": ["id1"]
104+
}
105+
}
106+
]
107+
}
108+
```
109+
110+
4. Drop the slot once you are done using it.
111+
```SQL
112+
SELECT pg_drop_replication_slot('test_slot');
113+
```
114+
115+
116+
## Monitoring slots
117+
118+
You must monitor logical decoding. Any unused replication slot must be dropped. Slots hold on to Postgres WAL logs and relevant system catalogs until changes have been read by a consumer. If your consumer fails or has not been properly configured, the unconsumed logs will pile up and fill your storage. Also, unconsumed logs increase the risk of transaction ID wraparound. Both situations can cause the server to become unavailable. Therefore, it is critical that logical replication slots are consumed continuously. If a logical replication slot is no longer used, drop it immediately.
119+
120+
The 'active' column in the pg_replication_slots view will indicate whether there is a consumer connected to a slot.
121+
```SQL
122+
SELECT * FROM pg_replication_slots;
123+
```
124+
125+
[Set alerts](howto-alert-on-metric.md) on *Storage used* and *Max lag across replicas* metrics to notify you when the values increase past normal thresholds.
126+
127+
> [!IMPORTANT]
128+
> You must drop unused replication slots. Failing to do so can lead to server unavailability.
129+
130+
## How to drop a slot
131+
If you are not actively consuming a replication slot you should drop it.
132+
133+
To drop a replication slot called `test_slot` using SQL:
134+
```SQL
135+
SELECT pg_drop_replication_slot('test_slot');
136+
```
137+
138+
> [!IMPORTANT]
139+
> If you stop using logical decoding, change azure.replication_support back to `replica` or `off`. The WAL details retained by `logical` are more verbose, and should be disabled when logical decoding is not in use.
140+
141+
142+
## Next steps
143+
144+
* Visit the Postgres documentation to [learn more about logical decoding](https://www.postgresql.org/docs/current/logicaldecoding-explanation.html).
145+
* Reach out to [our team](mailto:[email protected]) if you have questions about logical decoding.
146+
* Learn more about [read replicas](concepts-read-replicas.md).
147+

0 commit comments

Comments
 (0)