-
Notifications
You must be signed in to change notification settings - Fork 259
feat(rdb): pg16 MTA-5642a #4476
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 3 commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
ca9867d
feat(rdb): pg16 final touches
ldecarvalho-doc c5be409
fix(rdb): review walter
ldecarvalho-doc 485162b
fix(rdb): review neda
ldecarvalho-doc e22feac
fix(rdb): review hugo
ldecarvalho-doc 9ea9c9d
fix(rdb): review hugo 2
ldecarvalho-doc 1fc15dd
fix(rdb): review extensions
ldecarvalho-doc edd3dbd
fix(rdb): review bene
ldecarvalho-doc File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
105 changes: 105 additions & 0 deletions
105
...atabases-for-postgresql-and-mysql/api-cli/logical-replication-as-subscriber.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,105 @@ | ||
| --- | ||
| meta: | ||
| title: Setting up logical replication as a subscriber in PostgreSQL | ||
| description: Learn how to set up and use logical replication as a subscriber in PostgreSQL to migrate your database with zero downtime | ||
| content: | ||
| h1: Setting up logical replication as a subscriber in PostgreSQL | ||
| paragraph: This page explains how to set up and use logical replication as a subscriber in PostgreSQL to migrate your database to Scaleway, with zero downtime and real-time data synchronization | ||
| tags: postgresql logical replication subscriber migration database | ||
| dates: | ||
| validation: 2025-02-20 | ||
| posted: 2025-02-20 | ||
| --- | ||
|
|
||
| The logical replication of databases as a subscriber is available with [PostgreSQL 16](/managed-databases-for-postgresql-and-mysql/reference-content/pg-version-updates#postgresql-16), which is now supported in Scaleway's Managed Databases for PostgreSQL. | ||
|
|
||
| The feature allows you to replicate data from a non-managed database to a Managed Database for PostgreSQL in real-time, without having to replicate the entire database and keep them in sync, without having to take the original database offline or lock it for an extended period. | ||
|
|
||
| <Message type="important"> | ||
| At Scaleway we use a public interface to connect to an external database, even when you use a private endpoint. The connection, however, is completely secure as all ports are closed and outgoing data is encrypted. Learn more in the [Limitations section](#limitations) of this page. | ||
| </Message> | ||
|
|
||
| Logical replication is especially useful when migrating your databases to Scaleway's Managed Databases. | ||
|
|
||
| By setting up the new database as a subscriber to the original database you can ensure: | ||
| - **Zero-downtime** - Migrate your data without taking the original database offline. Your applications can continue to write data on the original database while the new database is being populated. | ||
| - **Real-time data synchronization** - The subscriber database receives updates in real-time, ensuring that data is consistent across the two databases. | ||
| - **Flexible migration windows** - You can choose when to switch your application to the new database, without having to worry about data consistency. Logical replication ensures that the new database is always up-to-date with the latest changes in the original database. | ||
| - **Easy rollback** - If you encounter issues during migration, such as data corruption and application issues, you can point your application back to the original database. Since the subscriber database constantly keeps a copy of the data in sync with the publisher, you can avoid data loss or inconsistencies. | ||
|
|
||
|
|
||
| ## How to set up the subscription | ||
|
|
||
| To be able to create a subscription, you must have the privileges of the `pg_create_subscription` role, as well as CREATE privileges on the current database. | ||
|
|
||
| 1. Grant `pg_create_subscription` and CREATE privileges to your role of choice. In this example we use `my_replication_user`. Make sure you replace all variables with the information of your databases. | ||
|
|
||
| ```sql | ||
| GRANT pg_create_subscription TO my_replication_user; | ||
| GRANT CREATE ON DATABASE my_database TO my_replication_user; | ||
| ``` | ||
| 2. Create a publication in the original database to define which tables and schemas will be replicated. In the example below we include all tables in the database. | ||
| <Message type="tip"> | ||
| Refer to the official [Create Publication](https://www.postgresql.org/docs/16/sql-createpublication.html) PostgreSQL documentation to learn how to detail the specific tables or schemas. | ||
| </Message> | ||
|
|
||
| ```sql | ||
| CREATE PUBLICATION my_publication FOR ALL TABLES; | ||
| ``` | ||
| 3. Subscribe the subscriber database to the publication created in the previous step. This will create a subscription that connects both databases. | ||
| <Message type="important"> | ||
| Subscriptions in the same database cannot have the same name. | ||
| </Message> | ||
|
|
||
| ```sql | ||
| CREATE SUBSCRIPTION my_subscription | ||
| CONNECTION 'host=publisher_host port=5432 user=my_replication_user password=my_password dbname=my_database' | ||
| PUBLICATION my_publication; | ||
| ``` | ||
| 4. Run the commands below to respectively grant the user read access to the subscription, and to check its statistics. This helps you make sure the subscription was successful. You can check the number of pending changes and last error message in the output. | ||
| ```sql | ||
| GRANT pg_read_all_stats TO my_replication_user; | ||
| SELECT * FROM pg_stat_subscription; | ||
| ``` | ||
| 5. Run the first command to grant the user read access to all data. Run the second to get a table containing data about all subscriptions defined in the database. This data includes the subscription ID, name, owner, whether the subscription is enabled or not, and connection information. | ||
ldecarvalho-doc marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| ```sql | ||
| GRANT pg_read_all_data TO myuser; | ||
ldecarvalho-doc marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| SELECT * FROM pg_subscription; | ||
ldecarvalho-doc marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| ``` | ||
| <Message type="tip"> | ||
| Refer to the official [PostgreSQL documentation](https://www.postgresql.org/docs/current/catalog-pg-subscription.html) for more information on the `pg_subscription` catalog. | ||
| </Message> | ||
|
|
||
|
|
||
| ## Limitations | ||
|
|
||
| When creating a subscription from a Scaleway Database Instance to a public IP, the public interface of the Database Instance node is used to connect to the publisher. This might incur in the following consequences: | ||
|
|
||
| - If you only have a private endpoint, your Database Instance will still have a public management IP address. The ports are all closed, stopping all connections except the one with the publisher database. All outgoing data is securely encrypted. | ||
| - If your Database Instance set-up is composed of more than one node, your IP address is subject to change. In case of failover, for example, the IP address connecting to the publisher database will change. We cannot predict the new IP your nodes might take. However, you can use [Scaleway's Autonomous System](https://ipinfo.io/AS12876) to check the number (ASN) of your primary node's IP address, and look up the IP address allocation for the ASN. This can help you predict the IP address of your failover nodes. | ||
| - There is a limit to the number of subscriptions that can be created per Database Instance offer. You can find the table of maximum number of subscriptions per node type below. | ||
|
|
||
| | Node Type | Maximum subscriptions | | ||
| | :--- | :--- | | ||
| | db-dev-s | 2 | | ||
| | db-dev-m | 4 | | ||
| | db-dev-l | 8 | | ||
| | db-dev-xl | 12 | | ||
| | db-gp-xs | 16 | | ||
| | db-gp-s | 16 | | ||
| | db-gp-m | 16 | | ||
| | db-gp-l | 16 | | ||
| | db-gp-xl | 16 | | ||
| | db-play2-pico | 2 | | ||
| | db-play2-nano | 4 | | ||
| | db-pro2-xxs | 8 | | ||
| | db-pro2-xs | 16 | | ||
| | db-pro2-s | 16 | | ||
| | db-pro2-m | 16 | | ||
| | db-pro2-l | 16 | | ||
| | db-pop2-2c-8g | 8 | | ||
| | db-pop2-4c-16g | 16 | | ||
| | db-pop2-8c-32g | 16 | | ||
| | db-pop2-16c-64g | 16 | | ||
| | db-pop2-32c-128g | 16 | | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
95 changes: 95 additions & 0 deletions
95
...ged-databases-for-postgresql-and-mysql/reference-content/pg-version-updates.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,95 @@ | ||
| --- | ||
| meta: | ||
| title: PostgreSQL version updates | ||
| description: Learn more about the latest updates and features available in each PostgreSQL version at Scaleway | ||
| content: | ||
| h1: PostgreSQL version updates | ||
| paragraph: Stay up-to-date with the latest features and improvements in each PostgreSQL version, and learn how to take advantage of them in your Scaleway Managed Databases | ||
| tags: databases postgresql versions updates features | ||
| dates: | ||
| validation: 2024-02-24 | ||
| categories: | ||
| - managed-databases | ||
| - postgresql-and-mysql | ||
| --- | ||
|
|
||
| This page lists updates for PostreSQL versions and their corresponding features that are supported at Scaleway. | ||
|
|
||
| ## PostgreSQL 16 | ||
|
|
||
| Find below the new features and updates available with PostgreSQL 16. | ||
|
|
||
| ## Roles | ||
|
|
||
| The `RDB` admin role can now assign the following predefined roles: | ||
|
|
||
| | Role | Description | | ||
| | :--- | :--- | | ||
| | `pg_read_all_data` | Has the right to read all data as if they have SELECT rights on objects and USAGE rights on schemas, without explicitly having said rights. | | ||
| | `pg_write_all_data`| Has the right to read all data as if they have INSERT, UPDATE and DELETE rights on objects and USAGE rights on schemas, without explicitly having said rights. | | ||
| | `pg_read_all_settings`| Has the right to read all configuration variables. | | ||
| | `pg_read_all_stats` | Has the right to read all `pg_stat_*` views and use statistics related extensions. | | ||
| | `pg_stat_scan_tables` | Has the right to run monitoring functions that may take ACCESS SHARE locks on tables, potentially for a long time. | | ||
| | `pg_monitor`| Has the right to read and execute various monitoring views and functions. | | ||
| | `pg_signal_backend` | Has the right to signal another backend to cancel a query or terminate its session. | | ||
| | `pg_checkpoint` | Has the right to execute the CHECKPOINT command. | | ||
| | `pg_create_subscription` | Has the right to issue CREATE SUBSCRIPTION, if they have CREATE permissions on the database. | | ||
|
|
||
| <Message type="tip"> | ||
| Refer to the official [Predefined Roles](https://www.postgresql.org/docs/16/predefined-roles.html) PostgreSQL documentation for a more detailed description of the roles above. | ||
| </Message> | ||
|
|
||
| ## Features | ||
|
|
||
| New features are available with PostgreSQL 16: | ||
|
|
||
| - The logical replication of databases as a subscriber is now supported. Refer to the [Setting up logical replication as a subscriber in PostgreSQL](/managed-databases-for-postgresql-and-mysql/api-cli/logical-replication-as-subscriber) documentation page for more information. | ||
| - Passwords are now encrypted using the `SCRAM-SHA-256` setting. If you upgrade your engine and are currently using MD5, you also have to migrate to SCRAM-SHA-256. Refer to the official [Password Authentication](https://www.postgresql.org/docs/current/auth-password.html#AUTH-PASSWORD) PostgreSQL documentation to learn how to do so. | ||
| - Support of the Timescale [pre-restore](https://docs.timescale.com/api/latest/administration/#timescaledb_pre_restore) and [post-restore](https://docs.timescale.com/api/latest/administration/#timescaledb_post_restore) features. They allow you to restore the database using `pg_restore`. | ||
|
|
||
| ## Extensions | ||
|
|
||
| The following extensions were also upgraded. | ||
|
|
||
| - **PostGIS** - [3.5](https://postgis.net/tags/3.5/) | ||
| - **PG GEOS** - 3.13 native with PostGIS 3.5 | ||
| - **Timescale** - [2.17](https://github.com/timescale/timescaledb/releases/tag/2.17.0) | ||
| - **pgRouting** - [3.6.3](https://github.com/pgRouting/pgrouting/releases/tag/v3.6.3) | ||
ldecarvalho-doc marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| - **pgvector** - [0.7.4](https://github.com/pgvector/pgvector/releases/tag/v0.7.4) | ||
ldecarvalho-doc marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| - **H3 PG** - [4.1.3](https://github.com/zachasme/h3-pg/releases/tag/v4.1.3) | ||
ldecarvalho-doc marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ## Advanced settings | ||
|
|
||
| These are the new Advanced Settings available with PostgreSQL 16: | ||
ldecarvalho-doc marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
ldecarvalho-doc marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ### Autovaccum | ||
|
|
||
| - `autovacuum_vacuum_insert_scale_factor` | ||
| - `autovacuum_vacuum_insert_threshold` | ||
| - `autovacuum_vacuum_scale_factor` | ||
| - `autovacuum_vacuum_threshold` | ||
|
|
||
| Refer to the official [Autovaccum](https://www.postgresql.org/docs/16/routine-vacuuming.html#AUTOVACUUM) PostgreSQL documentation for more information. | ||
|
|
||
| ### Error reporting and logging | ||
|
|
||
| - `cron.timezone` | ||
| - `log_checkpoints` | ||
| - `log_lock_waits` | ||
| - `log_min_duration_statement` | ||
| - `log_replication_commands` | ||
| - `log_temp_files` | ||
|
|
||
| Refer to the official [Error reporting and logging](https://www.postgresql.org/docs/16/runtime-config-logging.html) PostgreSQL documentation for more information. | ||
|
|
||
| ### PG stats parameters | ||
|
|
||
| - `pg_stat_statements.max` | ||
| - `pg_stat_statements.track` | ||
| - `pg_stat_statements.track_utility` | ||
|
|
||
| Refer to the official [PG stats parameters](ttps://www.postgresql.org/docs/16/pgstatstatements.html#PGSTATSTATEMENTS-CONFIG-PARAMS) PostgreSQL documentation for more information. | ||
|
|
||
|
|
||
|
|
||
|
|
||
ldecarvalho-doc marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.