You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When tables in the database get large, it's hard to manage how often they're vacuumed, how much space they take up, and how to keep their indexes efficient. This can make queries slower and affect performance. Partitioning of large tables is a solution for these situations. In this article, you find out how to use pg_partman extension to create range-based partitions of tables in your Azure Database for PostgreSQL Flexible Server.
17
+
In this article you learn how to optimize Azure Database for PostgreSQL Flexible Server by using pg_partman. When tables in the database get large, it's hard to manage how often they're vacuumed, how much space they take up, and how to keep their indexes efficient. This can make queries slower and affect performance. Partitioning of large tables is a solution for these situations. In this article, you find out how to use pg_partman extension to create range-based partitions of tables in your Azure Database for PostgreSQL Flexible Server.
18
18
19
19
## Prerequisites
20
20
21
21
To enable pg_partman extension, follow these steps.
22
22
23
-
- Add pg_partman extension under azure extensions as shown from server parameters on the portal.
23
+
- Add pg_partman extension under Azure extensions as shown from server parameters on the portal.
24
24
25
-
:::image type="content" source="media/how-to-use-pg-partman/pg-partman-prerequisites.png" alt-text="Screenshot of prerequisites.":::
25
+
:::image type="content" source="media/how-to-use-pg-partman/pg-partman-prerequisites.png" alt-text="Screenshot of prerequisites.":::
26
26
27
-
```sql
28
-
CREATE EXTENSION PG_PARTMAN;
29
-
```
30
-
31
-
## Overview
32
-
33
-
When an identity feature uses sequences, the data that comes from the parent table gets new sequence value. It doesn't generate new sequence values when the data is directly added to the child table.
34
-
35
-
PG_partman uses a template to control whether the table is UNLOGGED or not. This means that the Alter table command can't change this status for a partition set. By changing the status on the template, you can apply it to all future partitions. But for existing child tables, you must use the Alter command manually. [Here](https://www.postgresql.org/message-id/flat/15954-b61523bed4b110c4%40postgresql.org) is a bug that shows why.
36
-
37
-
There's another extension related to PG_partman called pg_partman_bgw, which must be included in Shared_Preload_Libraries. It offers a scheduled function run_maintenance(). It takes care of the partition sets that have automatic_maintenance turned ON in `part_config`.
38
-
39
-
:::image type="content" source="media/how-to-use-pg-partman/pg-partman-prerequisites-outlined.png" alt-text="Screenshot of prerequisites highlighted.":::
40
-
41
-
You can use server parameters in the Azure portal to change the following configuration options that affect the BGW process:
27
+
```sql
28
+
CREATE EXTENSION pg_partman;
29
+
```
42
30
43
-
`pg_partman_bgw.dbname` - Required. This parameter should contain one or more databases that run_maintenance() needs to be run on. If more than one, use a comma separated list. If nothing is set, BGW doesn't run the procedure.
31
+
- There's another extension related to `pg_partman` called `pg_partman_bgw`, which must be included in Shared_Preload_Libraries. It offers a scheduled function run_maintenance(). It takes care of the partition sets that have automatic_maintenance turned ON in `part_config`.
44
32
45
-
`pg_partman_bgw.interval` - Number of seconds between calls to run_maintenance() procedure. Default is 3600 (1 hour). This can be updated based on the requirement of the project.
33
+
:::image type="content" source="media/how-to-use-pg-partman/pg-partman-prerequisites-outlined.png" alt-text="Screenshot of prerequisites highlighted.":::
46
34
47
-
`pg_partman_bgw.role` - The role that run_maintenance() procedure runs as. Default is postgres. Only a single role name is allowed.
35
+
You can use server parameters in the Azure portal to change the following configuration options that affect the BGW process:
48
36
49
-
`pg_partman_bgw.analyze` - By default, it's set to OFF. Same purpose as the p_analyze argument to run_maintenance().
37
+
`pg_partman_bgw.dbname` - Required. This parameter should contain one or more databases that run_maintenance() needs to be run on. If more than one, use a comma separated list. If nothing is set, BGW doesn't run the procedure.
38
+
39
+
`pg_partman_bgw.interval`-Number of seconds between calls to run_maintenance() procedure. Default is 3600 (1 hour). This can be updated based on the requirement of the project.
40
+
41
+
`pg_partman_bgw.role`- The role that run_maintenance() procedure runs as. Default is postgres. Only a single role name is allowed.
42
+
43
+
`pg_partman_bgw.analyze`- By default, it's set to OFF. Same purpose as the p_analyze argument to run_maintenance().
44
+
45
+
`pg_partman_bgw.jobmon` - Same purpose as the p_jobmon argument to run_maintenance(). By default, it's set to ON.
50
46
51
-
`pg_partman_bgw.jobmon` - Same purpose as the p_jobmon argument to run_maintenance(). By default, it's set to ON.
47
+
> [!NOTE]
48
+
>1. When an identity feature uses sequences, the data that comes from the parent table gets new sequence value. It doesn't generate new sequence values when the data is directly added to the child table.
49
+
> 2. `pg_partman` uses a template to control whether the table is UNLOGGED or not. This means that the Alter table command can't change this status for a partition set. By changing the status on the template, you can apply it to all future partitions. But for existing child tables, you must use the Alter command manually. [Here](https://www.postgresql.org/message-id/flat/15954-b61523bed4b110c4%40postgresql.org) is a bug that shows why.
52
50
53
51
## Permissions
54
52
55
-
Pg_partman doesn't require a super user role to run. The only requirement is that the role that runs pg_partman functions has ownership over all the partition sets/schema where new objects will be created. It's recommended to create a separate role for pg_partman and give it ownership over the schema/all the objects that pg_partman will operate on.
53
+
Super user role is not required with `pg_partman`. The only requirement is that the role that runs `pg_partman` functions has ownership over all the partition sets/schema where new objects will be created. It's recommended to create a separate role for `pg_partman` and give it ownership over the schema/all the objects that `pg_partman` will operate on.
56
54
57
55
```sql
58
56
CREATE ROLE partman_role WITH LOGIN;
@@ -66,7 +64,7 @@ GRANT TEMPORARY ON DATABASE <databasename> to partman_role; -- this allows cre
66
64
```
67
65
## Creating partitions
68
66
69
-
Pg_partman relies on range type partitions and not on trigger-based partitions. This shows how pg_partman assists with the partitioning of a table.
67
+
Range type partitions are supported in `pg_partman`, not trigger-based partitions. The below code shows how `pg_partman` assists with the partitioning of a table.
70
68
71
69
```sql
72
70
CREATE SCHEMA partman;
@@ -76,9 +74,9 @@ PARTITION BY RANGE(d_date);
76
74
CREATE INDEX idx_partition_date ON partman.partition_test(d_date);
77
75
```
78
76
79
-
:::image type="content" source="media/how-to-use-pg-partman/pg-partman-table-output.png" alt-text="Screenshot of table output.":::
77
+
:::image type="content" source="media/how-to-use-pg-partman/pg-partman-table-output.png" alt-text="Screenshot of the table output for pg_partman.":::
80
78
81
-
Using the create_parent function, you can set up the number of partitions you want on the partition table.
79
+
Using the `create_parent` function, you can set up the number of partitions you want on the partition table.
82
80
83
81
```sql
84
82
SELECT public.create_parent(
@@ -97,96 +95,101 @@ SET infinite_time_partitions = true,
97
95
WHERE parent_table = 'partman.partition_test';
98
96
```
99
97
100
-
This command divides the p_parent_table into smaller parts based on the p_control column, using native partitioning (the other option is trigger-based partitioning, but pg_partman doesn't support it yet). The partitions are created at a daily interval. We'll create 20 future partitions in advance, instead of the default value of 4. We'll also specify the p_start_partition, where we mention the past date from which the partitions should start.
98
+
This command divides the `p_parent_table` into smaller parts based on the `p_control` column, using native partitioning (the other option is trigger-based partitioning, but `pg_partman` doesn't support it yet). The partitions are created at a daily interval. We'll create 20 future partitions in advance, instead of the default value of 4. We'll also specify the `p_start_partition`, where we mention the past datefrom which the partitions should start.
101
99
102
-
The `create_parent()` function populates two tables `part_config` and `part_config_sub`. There's a maintenance function `run_maintenance()`. You can schedule a cron job for this procedure to run on a periodic basis. This function checks all parent tables in *part_config* table and creates new partitions for them or runs the tables set retention policy. To know more about the functions and tables in pg_partman go through [here.](https://github.com/pgpartman/pg_partman/blob/master/doc/pg_partman.md)
100
+
The `create_parent()` function populates two tables `part_config`and`part_config_sub`. There's a maintenance function `run_maintenance()`. You can schedule a cron job for this procedure to run on a periodic basis. This function checks all parent tables in `part_config` table and creates new partitions for them or runs the tables set retention policy. To know more about the functions and tables in `pg_partman` go through the [PostgreSQL Partition Manager Extension](https://github.com/pgpartman/pg_partman/blob/master/doc/pg_partman.md) article.
103
101
104
-
To create new partitions every time the `run_maintenance()` is run in the background using `bgw` extension, run the below update statement.
102
+
To create new partitions every time the `run_maintenance()` is run in the background using `pg_partman_bgw` extension, run the below `update` statement.
UPDATE partman.part_config SET premake = premake+1 WHERE parent_table = 'partman.partition_test';
108
106
```
109
107
110
-
If the premake is the same and your run_maintenance() procedure is run, there wont be any new partitions created for that day. For the next day as premake defines from the current day a new partition for a day is created with the execution of you run_maintenance() function.
108
+
If the premake is the same and your `run_maintenance()` procedure is run, there wont be any new partitions created for that day. For the next day as premake defines from the current day a new partition for a day is created with the execution of you `run_maintenance()` function.
111
109
112
-
Using the insert command below, insert 100k rows for each month.
110
+
Using the insert command below, insert 100k rows for each month.
Run the command below to see the partitions created.
137
135
138
136
```sql
139
-
Postgres=>\d+partman.partition_test;
137
+
\d+ partman.partition_test;
140
138
```
141
139
142
-
:::image type="content" source="media/how-to-use-pg-partman/pg-partman-table-output-partitions.png" alt-text="Screenshot of table out with partitions." lightbox="media/how-to-use-pg-partman/pg-partman-table-output-partitions.png":::
140
+
:::image type="content" source="media/how-to-use-pg-partman/pg-partman-table-output-partitions.png" alt-text="Screenshot of table output with partitions." lightbox="media/how-to-use-pg-partman/pg-partman-table-output-partitions.png":::
143
141
144
142
Here's the output of the select statement executed.
145
143
146
-
:::image type="content" source="media/how-to-use-pg-partman/pg-partman-explain-plan-output.png" alt-text="Screenshot of explain plan output." lightbox="media/how-to-use-pg-partman/pg-partman-explain-plan-output.png":::
144
+
:::image type="content" source="media/how-to-use-pg-partman/pg-partman-explain-plan-output.png" alt-text="Screenshot of an explanation plan output." lightbox="media/how-to-use-pg-partman/pg-partman-explain-plan-output.png":::
147
145
148
146
## How to manually run the run_maintenance procedure
149
147
148
+
`partman.run_maintenance()` command could be run manually rather than the `pg_partman_bgw`. Use the below command to run the maintenance procedure manually.
> If you insert data before creating partitions, the data goes to the default partition. If the default partition has data that belongs to a new partition that you want to be created later, then you get a default partition violation error and the procedure won't work. Therefore, change the premake value as recommended above and then run the procedure.
156
156
157
157
## How to schedule maintenance procedure using pg_cron
158
158
159
-
Run the maintenance procedure using pg_cron. To enable `pg_cron` on your server follow the below steps.
160
-
1. Add PG_CRON to `azure.extensions`, `Shared_preload_libraries` and `cron.database_name` server parameter from Azure portal.
159
+
Run the maintenance procedure using `pg_cron`. To enable `pg_cron` on your server follow the below steps.
160
+
1. Add pg_cron to `azure.extensions`, `Shared_preload_libraries` and `cron.database_name` server parameter from Azure portal.
161
161
162
-
:::image type="content" source="media/how-to-use-pg-partman/pg-partman-pgcron-prerequisites.png" alt-text="Screenshot of pgcron prerequisites.":::
162
+
:::image type="content" source="media/how-to-use-pg-partman/pg-partman-pgcron-prerequisites.png" alt-text="Screenshot of pgcron extension prerequisites.":::
163
163
164
-
:::image type="content" source="media/how-to-use-pg-partman/pg-partman-pgcron-prerequisites-2.png" alt-text="Screenshot of pgcron prerequisites2.":::
164
+
:::image type="content" source="media/how-to-use-pg-partman/pg-partman-pgcron-prerequisites-2.png" alt-text="Screenshot of pgcron extension prerequisites2.":::
165
165
166
-
:::image type="content" source="media/how-to-use-pg-partman/pg-partman-pgcron-database-name.png" alt-text="Screenshot of pgcron databasename.":::
166
+
:::image type="content" source="media/how-to-use-pg-partman/pg-partman-pgcron-database-name.png" alt-text="Screenshot of pgcron extension databasename.":::
167
167
168
168
2. Hit Save button and let the deployment complete.
169
169
170
170
3. Once done the pg_cron is automatically created. If you still, try to install then you get the below message.
171
171
172
172
```sql
173
-
postgres=>CREATE EXTENSION pg_cron;
174
-
ERROR: extension "pg_cron" already exists
173
+
CREATE EXTENSION pg_cron;
174
+
```
175
175
176
-
postgres=>
176
+
```output
177
+
ERROR: extension "pg_cron" already exists
177
178
```
178
179
179
180
4. To schedule the cron job, use the below command.
5. You can view all the cron job using the command below.
186
187
187
188
```sql
188
-
postgres=> select * from cron.job;
189
+
SELECT * FROM cron.job;
190
+
```
189
191
192
+
```output
190
193
-[ RECORD 1 ]-----------------------------------------------------------------------
191
194
192
195
jobid | 1
@@ -203,26 +206,24 @@ Run the maintenance procedure using pg_cron. To enable `pg_cron` on your server
203
206
6. Run history of the job can be checked using the command below.
204
207
205
208
```sql
206
-
postgres=> select * from cron.job_run_details;
207
-
208
-
(0 rows)
209
+
SELECT * FROM cron.job_run_details;
209
210
```
210
211
211
212
Currently the results show 0 records as the job has not run yet.
212
213
213
214
7. To unschedule the cron job, use the command below.
214
215
215
216
```sql
216
-
postgres=> select cron.unschedule(1);
217
+
SELECT cron.unschedule(1);
217
218
```
218
219
219
-
## Limitations and considerations
220
+
## Frequently Asked Questions
220
221
221
-
- Why is my `bgw` not running the maintenance proc based on the interval provided.
222
+
- Why is my `pg_partman_bgw` not running the maintenance proc based on the interval provided.
222
223
223
224
Check the server parameter `pg_partman_bgw.dbname` and update it with the proper databasename. Also, check the server parameter `pg_partman_bgw.role` and provide the appropriate role with the role. You should also make sure you connecting to server using the same user to create the extension instead of postgres.
224
225
225
-
- I'm encountering an error when my bgw is running the maintenance proc. What could be the reasons?
226
+
- I'm encountering an error when my `pg_partman_bgw` is running the maintenance proc. What could be the reasons?
0 commit comments