Merge pull request #278234 from markingmyname/iaas

[PostgreSQL] [Migration Service] IaaS Offline/Online Articles and additional AWS changes

Author: v-shils

articles/postgresql/.openpublishing.redirection.postgresql.json

@@ -1624,6 +1624,16 @@
         "source_path_from_root": "/articles/postgresql/migrate/migration-service/tutorial-migration-service-aws.md",
         "redirect_url": "/azure/postgresql/migrate/migration-service/tutorial-migration-service-aws-offline",
         "redirect_document_id": true
+    },
+    {
+        "source_path_from_root": "/articles/postgresql/migrate/migration-service/tutorial-migration-service-iaas.md",
+        "redirect_url": "/azure/postgresql/migrate/migration-service/tutorial-migration-service-iaas-offline",
+        "redirect_document_id": true
+    },
+    {
+        "source_path_from_root": "/articles/postgresql/migrate/migration-service/concepts-migration-service-postgresql.md",
+        "redirect_url": "/azure/postgresql/migrate/migration-service/overview-migration-service-postgresql",
+        "redirect_document_id": true
     }
     ]
 }

articles/postgresql/TOC.yml

@@ -543,13 +543,15 @@
   - name: Migration Service
     items:
     - name: Overview
-      href: migrate/migration-service/concepts-migration-service-postgresql.md
+      href: migrate/migration-service/overview-migration-service-postgresql.md
     - name: Tutorials
       items:
       - name: Migrate from Single Server
         href: migrate/migration-service/tutorial-migration-service-single-to-flexible.md
-      - name: Migrate from on-premises or Azure VM
-        href: migrate/migration-service/tutorial-migration-service-iaas.md
+      - name: Migrate from an Azure VM or an on-premises PostgreSQL server offline
+        href: migrate/migration-service/tutorial-migration-service-iaas-offline.md
+      - name: Migrate from an Azure VM or an on-premises PostgreSQL server online
+        href: migrate/migration-service/tutorial-migration-service-iaas-online.md
       - name: Migrate from AWS RDS offline
         href: migrate/migration-service/tutorial-migration-service-aws-offline.md
       - name: Migrate from AWS RDS online
@@ -568,11 +570,13 @@
         href: migrate/migration-service/concepts-known-issues-migration-service.md
     - name: How-to guides
       items:
-      - name: Network setup
+      - name: How to set up Azure CLI
+        href: migrate/migration-service/how-to-setup-azure-cli-commands-postgresql.md
+      - name: How to set up the network
         href: migrate/migration-service/how-to-network-setup-migration-service.md
     - name: Troubleshoot
       items:
-      - name: Premigration error codes
+      - name: Known errors
         href: migrate/migration-service/troubleshoot-error-codes-premigration.md
   - name: Migrate data with pg_dump
     href: migrate/how-to-migrate-using-dump-and-restore.md

articles/postgresql/migrate/migration-service/concepts-known-issues-migration-service.md

@@ -53,6 +53,8 @@ Here are common limitations that apply to migration scenarios:
 
 - Migration to burstable SKUs isn't supported; databases must first be migrated to a non-burstable SKU and then scaled down if needed.
 
+- The Migration Runtime Server is specifically designed to operate with the default DNS servers/private DNS zones i.e., **privatelink.postgres.database.azure.com**. Custom DNS names/DNS servers are not supported by the migration service when utilizing the migration runtime server feature. When configuring private endpoints for both the source and target databases, it is imperative to use the default private DNS zone provided by Azure for the private link service. The use of custom DNS configurations is not yet supported and may lead to connectivity issues during the migration process.
+
 ## Limitations migrating from Azure Database for PostgreSQL single server
 
 - It's required that both the Azure Database for PostgreSQL single server and the Azure Database for PostgreSQL flexible server are in the same Azure region. Exceptions to this limitation are made for Azure Database for PostgreSQL flexible server situated in India, China, and UAE, where cross-region migrations are permitted.

articles/postgresql/migrate/migration-service/concepts-migration-service-runtime-server.md

@@ -30,7 +30,7 @@ The migration runtime server is essential for transferring data between differen
 
 To use the Migration Runtime Server feature within the migration service in Azure Database for PostgreSQL, follow these steps in the Azure portal:
 
-- Sign in to the Azure portal and access the migration service in the Azure Database for PostgreSQL.
+- Sign in to the Azure portal and access the migration service (from the target server) in the Azure Database for PostgreSQL instance.
 - Begin a new migration workflow within the service.
 - When you reach the "Select runtime server" tab, use the Migration Runtime Server by selecting "Yes."
 Choose your Azure subscription and resource group and the location of the VNet-integrated Azure Database for PostgreSQL—Flexible server.

articles/postgresql/migrate/migration-service/how-to-network-setup-migration-service.md

@@ -27,9 +27,9 @@ The table below summarizes the scenarios for connecting a source database to an
 | PostgreSQL installed Azure VM in same/different virtual network | VNet-integrated Azure Database for PostgreSQL - Flexible Server in same/different virtual network | Yes |
 | Azure Database for PostgreSQL - Single Server with public access | VNet-integrated Azure Database for PostgreSQL - Flexible Server | Yes |
 | Azure Database for PostgreSQL - Single Server with private endpoint | VNet-integrated Azure Database for PostgreSQL - Flexible Server | Yes |
-| Azure Database for PostgreSQL - Single Server with private endpoint | Azure Database for PostgreSQL - Flexible Server with private endpoint | Planned for future release |
-| On-premises/Azure VM/AWS with private access | Azure Database for PostgreSQL - Flexible Server with private endpoint | Planned for future release |
-| On-premises/Azure VM/AWS with private access | Azure Database for PostgreSQL - Flexible Server with public access | Planned for future release |
+| Azure Database for PostgreSQL - Single Server with private endpoint | Azure Database for PostgreSQL - Flexible Server with private endpoint | Yes |
+| On-premises/Azure VM/AWS with private access | Azure Database for PostgreSQL - Flexible Server with private endpoint | Yes |
+| On-premises/Azure VM/AWS with private access | Azure Database for PostgreSQL - Flexible Server with public access | No |
 
 ## Scenario 1: On-premises source to Azure Database for PostgreSQL with public access
 
@@ -212,17 +212,6 @@ Below are the networking steps for migrating a PostgreSQL database from an on-pr
     - Attach a private DNS zone to the Flexible Server's private endpoint if not already configured.
     - Add virtual network links for the Flexible Server and Migration Runtime Server to the private DNS zone.
 
-## Scenario 10: Custom private DNS Zone attached to the private endpoint
-
-This scenario addresses the networking challenge when migrating databases to Azure PostgreSQL, which requires using Azure's default private DNS zone. Custom DNS configurations are incompatible with the Migration Runtime Server, necessitating the temporary use of Azure's default private DNS zone.
-
-- Check if the source and target databases use custom DNS zones with private endpoints.
-- For both the source and target databases, set up new private endpoints that are linked to Azure's default private DNS zone.
-- Connect the new private endpoints to the **privatelink.postgres.database.azure.com** DNS zone for proper name resolution.
-- Ensure the Migration Runtime Server can resolve and communicate with both databases using the new private endpoints.
-- Carry out the database migration through the secondary endpoints that use the default private DNS zone.
-- Remove the secondary private endpoints after successful migration and restore any custom DNS configurations if needed.
-
 ## Resources for network setup
 
 - To establish an **ExpressRoute** connection, refer to the [What is Azure ExpressRoute?](../../../expressroute/expressroute-introduction.md).

articles/postgresql/migrate/migration-service/how-to-setup-azure-cli-commands-postgresql.md

@@ -0,0 +1,113 @@
+---
+title: How to set up Azure CLI for migration service in Azure Database for PostgreSQL - Flexible Server
+description: Learn how to set up Azure CLI for migration service in Azure Database for PostgreSQL - Flexible Server and begin migrating your data.
+author: markingmyname
+ms.author: maghan
+ms.date: 06/13/2024
+ms.service: postgresql
+ms.subservice: flexible-server
+ms.topic: how-to
+ms.custom:
+  - devx-track-azurecli
+---
+
+# How to set up Azure CLI for migration service in Azure Database for PostgreSQL - Flexible Server
+
+To begin migrating using Azure CLI, you need to install the Azure CLI on your local machine.
+
+## Prerequisites
+
+- [Choose the right command line tool](/CLI/azure/choose-the-right-azure-command-line-tool)
+- [Install Azure CLI](/cli/azure/install-azure-cli)
+- [Review the az postgresql commands](/cli/azure/postgres)
+
+## Migrate commands
+
+Migrating with the Azure CLI uses the following commands:
+
+- [az postgres flexible-server migration create](/cli/azure/postgres/flexible-server/migration#az-postgres-flexible-server-migration-create)
+- [az postgres flexible-server migration list](/cli/azure/postgres/flexible-server/migration#az-postgres-flexible-server-migration-list)
+- [az postgres flexible-server migration show](/cli/azure/postgres/flexible-server/migration#az-postgres-flexible-server-migration-show)
+
+The following table defines the parameters and the commands that use them:
+
+| Parameter | Relevant commands | Description |
+| --- | --- | --- |
+| `subscription` | Create, List, Show | Subscription ID of PostgreSQL Flexible server |
+| `resource-group` | Create, List, Show | Resource group of PostgreSQL Flexible server |
+| `name` | Create, List, Show | Name of the PostgreSQL Flexible server |
+| `migration-name` | Create, Show | Unique identifier to migrations attempted to Flexible Server. This field accepts only alphanumeric characters and doesn't accept any special characters except a hyphen (-). The name can't start with -, and no two migrations to a Flexible Server target can have the same name. |
+| `filter` | List | To filter migrations, two values are supported – Active and All  
+| `help` | Create, List, Show | Provides information about each command. |
+| `migration-mode` | Create | This is an optional parameter. Default value: Offline. Offline migration involves copying your source databases to your target server at a point in time. |
+| `migration-option` | Create | Allows you to perform validations before triggering a migration. Default is ValidateAndMigrate. Allowed values are - Migrate, Validate, ValidateAndMigrate.
+| `properties` | Create | Absolute path to a JSON file that has the information about the source server |
+
+The **create** command requires a JSON file, which contains the following information:
+
+| Property Name | Description |
+| --- | --- |
+| `sourceDbServerResourceId` | Source server details in the format for on-premises, Azure virtual machines (VMs), AWS - `<<hostname or IP address>>:<<port>>@<<username>>` |
+| `AdminCredentials` | This parameter lists passwords for admin users for both the source server and the target PostgreSQL flexible server. These passwords help to authenticate against the source and target servers |
+| `targetServerUserName` | The default value is the admin user created during the creation of the flexible server, and the password provided is used for authentication against this user. If you aren't using the default user, this parameter is the user or role on the target server for migration. This user should be a member of azure_pg_admin, pg_read_all_settings, pg_read_all_stats, and pg_stat_scan_tables roles and should have the Create role, Create DB attributes |
+| `dbsToMigrate` | Specify the list of databases that you want to migrate to Flexible Server. You can include a maximum of eight database names at a time. Providing the list of DBs in array format. |
+| `OverwriteDBsInTarget` | When set to true (default), if the target server happens to have an existing database with the same name as the one you're trying to migrate, the migration tool automatically overwrites the database |
+| `MigrationMode` | Mode of the migration. The supported value is "Offline" |
+| `sourceType` | Required parameter. Values can be - on-premises, AWS, AzureVM, PostgreSQLSingleServer |
+| `sslMode` | SSL modes for migration. SSL mode for PostgreSQLSingleServer is VerifyFull and Prefer/Require for other source types |
+
+## Cutover the migration
+
+After the base data migration is complete, the migration task moves to the `WaitingForCutoverTrigger` substate. In this state, users can trigger the cutover from the portal by selecting the migration name in the migration grid or through the CLI using the command below.
+
+For example:
+
+```azurecli-interactive
+az postgres flexible-server migration update --subscription 11111111-1111-1111-1111-111111111111 --resource-group my-learning-rg --name myflexibleserver --migration-name CLIMigrationExample --cutover
+```
+
+Before initiating cutover, it's important to ensure that:
+
+- Writes to the source are stopped
+- `latency` value decreases to 0 or close to 0
+- `latency` value indicates when the target last synced with the source. At this point, writes to the source can be stopped and cutover initiated. In case there's heavy traffic at the source, it's recommended to stop writes first so that `Latency` can come close to 0, and then a cutover is initiated.
+- The Cutover operation applies all pending changes from the Source to the Target and completes the migration. If you trigger a "Cutover" even with nonzero `Latency`, the replication stops until that point in time. All the data on source until the cutover point is then applied on the target. Say a latency was 15 minutes at the cutover point, so all the changed data in the last 15 minutes applies to the target.
+
+Time depends on the backlog of changes occurring in the last 15 minutes. Hence, it's recommended that the latency go to zero or near zero before triggering the cutover.
+
+The `latency` information can be obtained using the migration show command.
+Here's a snapshot of the migration before initiating the cutover:
+
+After the cutover is initiated, pending data captured during CDC is written to the target, and migration is now complete.
+
+If the cutover isn't successful, the migration moves to the `Failed` state.
+
+For more information about this command, use the `help` parameter.
+
+## Cancel the migration
+
+To cancel the migration task, use the following command:
+
+```azurecli
+az postgres flexible-server migration update cancel [--subscription]
+                                            [--resource-group]
+                                            [--name]
+                                            [--migration-name]
+```
+
+## The --help command
+
+For more information about this command, use the `help` parameter:
+
+```azurecli-interactive
+az postgres flexible-server migration update cancel -- help
+```
+
+## Related content
+
+- [Migration service in Azure Database for PostgreSQL](overview-migration-service-postgresql.md)
+- [Migrate from Single Server to Flexible Server](tutorial-migration-service-single-to-flexible.md)
+- [Migrate offline from AWS RDS PostgreSQL](tutorial-migration-service-aws-offline.md)
+- [Migrate online from AWS RDS PostgreSQL](tutorial-migration-service-aws-online.md)
+- [Migrate offline from on-premises or an Azure VM hosted PostgreSQL](tutorial-migration-service-iaas-offline.md)
+- [Migrate online from on-premises or an Azure VM hosted PostgreSQL](tutorial-migration-service-iaas-online.md)

articles/postgresql/migrate/migration-service/includes/aws/prerequisites-migration-service-postgresql-offline-aws.md

@@ -33,34 +33,9 @@ The SKU chosen for the Azure Database for PostgreSQL should correspond with the
 
 ### Configure network setup
 
-Proper networking setup is essential to ensure successful migration connectivity between the source and target. Here's a guide to help you establish the network connection for different scenarios:
+Network setup is crucial for the migration service to function correctly. Ensure that the source PostgreSQL server can communicate with the target Azure Database for PostgreSQL server. The following network configurations are essential for a successful migration.
 
-- **Networking requirements for migration:**
-
-  - **ExpressRoute/IPsec VPN/VPN tunneling**: When connecting your on-premises/AWS source to Azure, you might need to set up an ExpressRoute, IPsec VPN, or VPN tunneling to facilitate secure data transfer.
-
-  - **VNET peering**: Establish virtual network peering between the two distinct VNets to enable direct network connectivity, a prerequisite for migration between the Azure VM and the Azure Database for PostgreSQL.
-
-- **Connectivity Scenarios:**
-
-The following table can help set up the network between the source and target.
-
-| Source | Target | Connectivity Tips |
-| --- | --- | --- |
-| Public | Public | No other action is required if the source is whitelisted in the target's firewall rules. |
-| Private | Public | This configuration isn't supported; use pg_dump/pg_restore for data transfer. |
-| Public | Private | No other action is required if the source is whitelisted in the target's firewall rules. |
-| Private | Private | Establish an ExpressRoute, IPsec VPN, VPN Tunneling, or virtual network Peering between the source and target. |
-| Private | Private Endpoint | This configuration isn't supported; contact [Microsoft support](https://support.microsoft.com/en-US). |
-
-- **Additional networking considerations:**
-
-pg_hba.conf Configuration: To facilitate connectivity between the source and target PostgreSQL instances, it's essential to verify and potentially modify the pg_hba.conf file. This file includes client authentication and must be configured to allow the target PostgreSQL to connect to the source. Changes to the pg_hba.conf file typically require a restart of the source PostgreSQL instance to take effect.
-
-> [!NOTE]  
-> The pg_hba.conf file is located in the data directory of the PostgreSQL installation. This file should be checked and configured if the source database is an on-premises PostgreSQL server or a PostgreSQL server hosted on an Azure VM. For PostgreSQL instances on AWS RDS or similar managed services, the pg_hba.conf file is not directly accessible or applicable. Instead, access is controlled through the service's security and network access configurations.
-
-For more information about network setup, visit [Network guide for migration service](../../how-to-network-setup-migration-service.md).
+For information about network setup, visit [Network guide for migration service](../../how-to-network-setup-migration-service.md).
 
 ### Enable extensions