You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# Best practices for bulk data upload for Azure Database for PostgreSQL - Flexible Server
15
15
16
16
There are two types of bulk loads:
17
17
- Initial data load of an empty database
@@ -23,7 +23,7 @@ This article discusses various loading techniques along with best practices when
23
23
24
24
Performance-wise, the data loading methods arranged in the order of most time consuming to least time consuming is as follows:
25
25
- Single record Insert
26
-
- Batch into 100-1000 rows per commit. One can use transaction block to wrap multiple records per commit [Batch Inserts]
26
+
- Batch into 100-1000 rows per commit. One can use transaction block to wrap multiple records per commit
27
27
- INSERT with multi row values
28
28
- COPY command
29
29
@@ -33,27 +33,27 @@ The preferred method to load the data into the database is by copy command. If t
33
33
34
34
#### Drop indexes
35
35
36
-
Before an initial data load, it is advised to drop all the indexes in the tables. It is always more efficient to create the indexes after the data load.
36
+
Before an initial data load, it's advised to drop all the indexes in the tables. It's always more efficient to create the indexes after the data load.
37
37
38
38
#### Drop constraints
39
39
40
40
##### Unique key constraints
41
41
42
-
To achieve strong performance, it's advised to drop unique key constraints before a initial data load and recreate it once the data load is completed. However, be aware that dropping unique key constraints cancels the safeguards against duplicated data.
42
+
To achieve strong performance, it's advised to drop unique key constraints before an initial data load, and recreate it once the data load is completed. However, dropping unique key constraints cancels the safeguards against duplicated data.
43
43
44
44
##### Foreign key constraints
45
45
46
46
It's advised to drop foreign key constraints before initial data load and recreate once data load is completed.
47
47
48
-
Changing the `session_replication_role` parameter to replica also disables all foreign key checks.However, be aware making the change can leave data in an inconsistent state if not properly used.
48
+
Changing the `session_replication_role` parameter to replica also disables all foreign key checks.However, be aware making the change can leave data in an inconsistent state if not properly used.
49
49
50
50
#### Unlogged tables
51
51
52
-
Use of unlogged tables will make data load faster. Data written to unlogged tables is not written to the write-ahead log.
52
+
Use of unlogged tables will make data load faster. Data written to unlogged tables isn't written to the write-ahead log.
53
53
54
54
The disadvantages of using unlogged tables are
55
-
- They are not crash-safe. An unlogged table is automatically truncated after a crash or unclean shutdown.
56
-
- Data from unlogged tables cannot be replicated to standby servers.
55
+
- They aren't crash-safe. An unlogged table is automatically truncated after a crash or unclean shutdown.
56
+
- Data from unlogged tables can't be replicated to standby servers.
57
57
58
58
The pros and cons of using unlogged tables should be considered before using in initial data loads.
59
59
@@ -84,7 +84,7 @@ The maintenance_work_mem can be set to a maximum of 2 GB on a flexible server. `
84
84
85
85
`checkpoint_timeout`
86
86
87
-
On the flexible server, the checkpoint_timeout can be increased to maximum 24h from default 5 minutes. It is advised to increase the value to 1 hour before initial data loads on Flexible server.
87
+
On the flexible server, the checkpoint_timeout can be increased to maximum 24 h from default 5 minutes. It's advised to increase the value to 1 hour before initial data loads on Flexible server.
88
88
89
89
`checkpoint_completion_target`
90
90
@@ -96,12 +96,12 @@ The max_wal_size can be set to the maximum allowed value on the Flexible server,
96
96
97
97
`wal_compression`
98
98
99
-
wal_compression can be turned on. Enabling the parameter can reduce the WAL volume without increasing the risk of unrecoverable data corruption, but at the cost of some extra CPU spent on the compression during WAL logging and on the decompression during WAL replay.
99
+
wal_compression can be turned on. Enabling the parameter can have some extra CPU cost spent on the compression during WAL logging and on the decompression during WAL replay.
100
100
101
101
102
102
#### Flexible server recommendations
103
103
104
-
Before the start of initial data load on a Flexible server, it is recommended to
104
+
Before the start of initial data load on a Flexible server, it's recommended to
105
105
106
106
- Disable high availability [HA] on the server. You can enable HA once initial load is completed on master/primary.
107
107
- Create read replicas after initial data load is completed.
@@ -118,7 +118,7 @@ Sets the maximum number of workers that the system can support for parallel quer
118
118
119
119
`max_parallel_maintenance_workers`
120
120
121
-
Controls the maximum number of worker process, which can be used to CREATE INDEX.
121
+
Controls the maximum number of worker processes, which can be used to CREATE INDEX.
122
122
123
123
One could also create the indexes by making recommended settings at the session level. An example of how it can be done at the session level is shown below:
124
124
@@ -133,7 +133,7 @@ CREATE INDEX test_index ON test_table (test_column);
133
133
134
134
#### Table partitioning
135
135
136
-
It is always recommended to partition large tables. Some advantages of partitioning, especially during incremental loads:
136
+
It's always recommended to partition large tables. Some advantages of partitioning, especially during incremental loads:
137
137
- Creation of new partitions based on the new deltas makes it efficient to add new data to the table.
138
138
- Maintenance of tables becomes easier. One can drop a partition during incremental data loads avoiding time-consuming deletes on large tables.
139
139
- Autovacuum would be triggered only on partitions that were changed or added during incremental loads, which make maintaining statistics on the table easier.
@@ -145,7 +145,7 @@ Monitoring and maintaining table statistics is important for query performance o
145
145
#### Index creation on foreign key constraints
146
146
147
147
Creating indexes on foreign keys in the child tables would be beneficial in the following scenarios:
148
-
- Data updates or deletions in the parent table. When data is updated or deleted in the parent table lookups would be performed on the child table.To make lookups faster you could index foreign keys on the child table.
148
+
- Data updates or deletions in the parent table. When data is updated or deleted in the parent table lookups would be performed on the child table.To make lookups faster, you could index foreign keys on the child table.
149
149
- Queries, where we see join between parent and child tables on key columns.
150
150
151
151
#### Unused indexes
@@ -235,7 +235,7 @@ SELECT round (pg_wal_lsn_diff('LSN value when run second time','LSN value when r
235
235
236
236
`wal_compression`
237
237
238
-
wal_compression can be turned on. Enabling the parameter can reduce the WAL volume without increasing the risk of unrecoverable data corruption, but at the cost of some extra CPU spent on the compression during WAL logging and on the decompression during WAL replay.
238
+
wal_compression can be turned on. Enabling the parameter can have some extra CPU cost spent on the compression during WAL logging and on the decompression during WAL replay.
Copy file name to clipboardExpand all lines: articles/postgresql/flexible-server/how-to-high-io-utilization.md
+24-24Lines changed: 24 additions & 24 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,5 +1,5 @@
1
1
---
2
-
title: High IOPS Utilization
2
+
title: High IOPS Utilization for Azure Database for PostgreSQL - Flexible Server
3
3
description: Troubleshooting guide for high IOPS utilization in Azure Database for PostgreSQL - Flexible Server
4
4
author: sarat0681
5
5
ms.author: sbalijepalli
@@ -9,7 +9,7 @@ ms.date: 08/16/2022
9
9
ms.custom: template-how-to #Required; leave this attribute/value as-is.
10
10
---
11
11
12
-
# Troubleshoot high IOPS utilization in Azure Database for PostgreSQL - Flexible Server
12
+
# Troubleshoot high IOPS utilization for Azure Database for PostgreSQL - Flexible Server
13
13
14
14
This article shows you how to quickly identify the root cause of high IOPS utilization and possible remedial actions to control IOPS utilization when using [Azure Database for PostgreSQL - Flexible Server](overview.md).
15
15
@@ -19,15 +19,15 @@ In this article, you learn:
19
19
- How to identify root causes, such as long-running queries, checkpoint timings, disruptive autovacuum daemon process, and high storage utilization.
20
20
- How to resolve high IO utilization using Explain Analyze, tune checkpoint-related server parameters, and tune autovacuum daemon.
21
21
22
-
###Tools to identify high IO utilization
22
+
## Tools to identify high IO utilization
23
23
24
24
Consider these tools to identify high IO utilization.
25
25
26
-
####Azure metrics
26
+
### Azure metrics
27
27
28
28
Azure Metrics is a good starting point to check the IO utilization for the definite date and period. Metrics give information about the time duration the IO utilization is high. Compare the graphs of Write IOPs, Read IOPs, Read Throughput, and Write Throughput to find out times when the workload caused high IO utilization. For proactive monitoring, you can configure alerts on the metrics. For step-by-step guidance, see [Azure Metrics](./howto-alert-on-metrics.md).
29
29
30
-
####Query store
30
+
### Query store
31
31
32
32
Query Store automatically captures the history of queries and runtime statistics and retains them for your review. It slices the data by time to see temporal usage patterns. Data for all users, databases, and queries is stored in a database named azure_sys in the Azure Database for PostgreSQL instance. For step-by-step guidance, see [Query Store](./concepts-query-store.md).
33
33
@@ -38,7 +38,7 @@ select * from query_store.qs_view qv where is_system_query is FALSE
38
38
order by blk_read_time + blk_write_time desclimit5;
39
39
```
40
40
41
-
####pg_stat_statements
41
+
### pg_stat_statements
42
42
43
43
The pg_stat_statements extension helps identify queries that consume IO on the server.
44
44
@@ -54,11 +54,11 @@ LIMIT 5;
54
54
> [!NOTE]
55
55
> When using query store or pg_stat_statements for columns blk_read_time and blk_write_time to be populated enable server parameter `track_io_timing`.For more information about the **track_io_timing** parameter, review [Server Parameters](https://www.postgresql.org/docs/current/runtime-config-statistics.html).
56
56
57
-
###Identify root causes
57
+
## Identify root causes
58
58
59
59
If IO consumption levels are high in general, the following could be possible root causes:
60
60
61
-
####Long-running transactions
61
+
### Long-running transactions
62
62
63
63
Long-running transactions can consume IO, that can lead to high IO utilization.
64
64
@@ -71,13 +71,13 @@ WHERE pid <> pg_backend_pid() and state IN ('idle in transaction', 'active')
71
71
ORDER BY duration DESC;
72
72
```
73
73
74
-
####Checkpoint timings
74
+
### Checkpoint timings
75
75
76
76
High IO can also be seen in scenarios where a checkpoint is happening too frequently. One way to identify this is by checking the Postgres log file for the following log text "LOG: checkpoints are occurring too frequently."
77
77
78
78
You could also investigate using an approach where periodic snapshots of `pg_stat_bgwriter` with a timestamp is saved. Using the snapshots saved the average checkpoint interval, number of checkpoints requested and number of checkpoints timed can be calculated.
79
79
80
-
####Disruptive autovacuum daemon process
80
+
### Disruptive autovacuum daemon process
81
81
82
82
Execute the below query to monitor autovacuum:
83
83
@@ -90,15 +90,15 @@ The query is used to check how frequently the tables in the database are being v
90
90
**autovacuum_count** : provides number of times the table was vacuumed.
91
91
**autoanalyze_count**: provides number of times the table was analyzed.
92
92
93
-
###Resolve high IO utilization
93
+
## Resolve high IO utilization
94
94
95
95
To resolve high IO utilization, there are three methods you could employ - using Explain Analyze, terminating long-running transactions, or tuning server parameters.
96
96
97
-
####Explain Analyze
97
+
### Explain Analyze
98
98
99
99
Once you identify the query that's consuming high IO, use **EXPLAIN ANALYZE** to further investigate the query and tune it. For more information about the **EXPLAIN ANALYZE** command, review [Explain Plan](https://www.postgresql.org/docs/current/sql-explain.html).
100
100
101
-
####Terminating long running transactions
101
+
### Terminating long running transactions
102
102
103
103
You could consider killing a long running transaction as an option.
104
104
@@ -113,19 +113,19 @@ ORDER BY duration DESC;
113
113
114
114
You can also filter by other properties like `usename` (username), `datname` (database name) etc.
115
115
116
-
Once you have the session's PID you can terminate using the following query:
116
+
Once you have the session's PID, you can terminate using the following query:
117
117
118
118
```sql
119
119
SELECT pg_terminate_backend(pid);
120
120
```
121
121
122
-
####Server parameter tuning
122
+
### Server parameter tuning
123
123
124
124
If it's observed that the checkpoint is happening too frequently, increase `max_wal_size` server parameter until most checkpoints are time driven, instead of requested. Eventually, 90% or more should be time based, and the interval between two checkpoints is close to the `checkpoint_timeout` set on the server.
125
125
126
-
##### `max_wal_size`
126
+
`max_wal_size`
127
127
128
-
Peak business hours is a good time to arrive at `max_wal_size` value. Follow the below listed steps to arrive at a value.
128
+
Peak business hours are a good time to arrive at `max_wal_size` value. Follow the below listed steps to arrive at a value.
129
129
130
130
Execute the below query to get current WAL LSN, note down the result:
131
131
@@ -145,23 +145,23 @@ Execute below query that uses the two results to check the difference in GB:
145
145
select round (pg_wal_lsn_diff ('LSN value when run second time', 'LSN value when run first time')/1024/1024/1024,2) WAL_CHANGE_GB;
146
146
```
147
147
148
-
##### `checkpoint_completion_target`
148
+
`checkpoint_completion_target`
149
149
150
150
A good practice would be to set it to 0.9. As an example, a value of 0.9 for a `checkpoint_timeout` of 5 minutes indicates the target to complete a checkpoint is 270 sec [0.9*300 sec]. A value of 0.9 provides fairly consistent I/O load. An aggressive value of `check_point_completion_target` may result in increased IO load on the server.
151
151
152
-
##### `checkpoint_timeout`
152
+
`checkpoint_timeout`
153
153
154
-
The `checkpoint_timeout` value can be increased from default value set on the server. Please note while increasing the `checkpoint_timeout` take into consideration that increasing the value would also increase the time for crash recovery.
154
+
The `checkpoint_timeout` value can be increased from default value set on the server. Note while increasing the `checkpoint_timeout` take into consideration that increasing the value would also increase the time for crash recovery.
155
155
156
-
####Autovacuum tuning to decrease disruptions
156
+
### Autovacuum tuning to decrease disruptions
157
157
158
-
For more details on monitoring and tuning in scenarios where autovacuum is too disruptive please review [Autovacuum Tuning](./how-to-autovacuum-tuning.md).
158
+
For more details on monitoring and tuning in scenarios where autovacuum is too disruptive review [Autovacuum Tuning](./how-to-autovacuum-tuning.md).
159
159
160
-
####Increase storage
160
+
### Increase storage
161
161
162
162
Increasing storage will also help in addition of more IOPS to the server. For more details on storage and associated IOPS review [Compute and Storage Options](./concepts-compute-storage.md).
163
163
164
-
###Next steps
164
+
## Next steps
165
165
166
166
- Troubleshoot and tune Autovacuum [Autovacuum Tuning](./how-to-autovacuum-tuning.md)
167
167
- Compute and Storage Options [Compute and Storage Options](./concepts-compute-storage.md)
0 commit comments