diff --git a/src/current/_includes/molt/molt-setup.md b/src/current/_includes/molt/molt-setup.md index 00f9f642a3b..caf275bf2cb 100644 --- a/src/current/_includes/molt/molt-setup.md +++ b/src/current/_includes/molt/molt-setup.md @@ -74,6 +74,12 @@ When you run `molt fetch`, you can configure the following options for data load - [Data load mode](#data-load-mode): Choose between `IMPORT INTO` and `COPY FROM`. - [Fetch metrics](#fetch-metrics): Configure metrics collection during initial data load. +
+ + + +
+ ### Connection strings {% include molt/molt-connection-strings.md %} diff --git a/src/current/_includes/molt/replicator-flags-usage.md b/src/current/_includes/molt/replicator-flags-usage.md index 99f93b4ba0b..559d77e3a9e 100644 --- a/src/current/_includes/molt/replicator-flags-usage.md +++ b/src/current/_includes/molt/replicator-flags-usage.md @@ -5,7 +5,7 @@ The following [MOLT Replicator]({% link molt/molt-replicator.md %}) flags are ** | Flag | Description | |-------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `--slotName` | **Required.** PostgreSQL replication slot name. Must match the slot name specified with `--pglogical-replication-slot-name` in the [MOLT Fetch command](#start-fetch). | -| `--targetSchema` | **Required.** Target schema name on CockroachDB where tables will be replicated. | +| `--targetSchema` | **Required.** Target schema name on CockroachDB where tables will be replicated. Schema name must be fully qualified in the format `database.schema`. | | `--stagingSchema` | **Required.** Staging schema name for replication metadata and checkpoints. | | `--stagingCreateSchema` | **Required.** Automatically create the staging schema if it does not exist. Include this flag when starting replication for the first time. | | `--metricsAddr` | Enable Prometheus metrics at a specified `{host}:{port}`. Metrics are served at `http://{host}:{port}/_/varz`. | @@ -14,7 +14,7 @@ The following [MOLT Replicator]({% link molt/molt-replicator.md %}) flags are **
| Flag | Description | |-------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `--targetSchema` | **Required.** Target schema name on CockroachDB where tables will be replicated. | +| `--targetSchema` | **Required.** Target schema name on CockroachDB where tables will be replicated. Schema name must be fully qualified in the format `database.schema`. | | `--defaultGTIDSet` | **Required.** Default GTID set for changefeed. | | `--stagingSchema` | **Required.** Staging schema name for replication metadata and checkpoints. | | `--stagingCreateSchema` | **Required.** Automatically create the staging schema if it does not exist. Include this flag when starting replication for the first time. | @@ -26,16 +26,16 @@ You can find the starting GTID in the `cdc_cursor` field of the `fetch complete`
-| Flag | Description | -|-------------------------|---------------------------------------------------------------------------------------------------------------------------------------------| -| `--sourceSchema` | **Required.** Source schema name on Oracle where tables will be replicated from. | -| `--targetSchema` | **Required.** Target schema name on CockroachDB where tables will be replicated. | -| `--scn` | **Required.** Snapshot System Change Number (SCN) for the initial changefeed starting point. | -| `--backfillFromSCN` | **Required.** SCN of the earliest active transaction at the time of the snapshot. Ensures no transactions are skipped. | -| `--stagingSchema` | **Required.** Staging schema name for replication metadata and checkpoints. | -| `--stagingCreateSchema` | **Required.** Automatically create the staging schema if it does not exist. Include this flag when starting replication for the first time. | -| `--metricsAddr` | Enable Prometheus metrics at a specified `{host}:{port}`. Metrics are served at `http://{host}:{port}/_/varz`. | -| `--userscript` | Path to a userscript that enables table filtering from Oracle sources. Refer to [Table filter userscript](#table-filter-userscript). | +| Flag | Description | +|-------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------| +| `--sourceSchema` | **Required.** Source schema name on Oracle where tables will be replicated from. | +| `--targetSchema` | **Required.** Target schema name on CockroachDB where tables will be replicated. Schema name must be fully qualified in the format `database.schema`. | +| `--scn` | **Required.** Snapshot System Change Number (SCN) for the initial changefeed starting point. | +| `--backfillFromSCN` | **Required.** SCN of the earliest active transaction at the time of the snapshot. Ensures no transactions are skipped. | +| `--stagingSchema` | **Required.** Staging schema name for replication metadata and checkpoints. | +| `--stagingCreateSchema` | **Required.** Automatically create the staging schema if it does not exist. Include this flag when starting replication for the first time. | +| `--metricsAddr` | Enable Prometheus metrics at a specified `{host}:{port}`. Metrics are served at `http://{host}:{port}/_/varz`. | +| `--userscript` | Path to a userscript that enables table filtering from Oracle sources. Refer to [Table filter userscript](#table-filter-userscript). | You can find the SCN values in the message `replication-only mode should include the following replicator flags` after the [initial data load](#start-fetch) completes.
diff --git a/src/current/_includes/molt/replicator-flags.md b/src/current/_includes/molt/replicator-flags.md index 4dea042a028..e62875a0b2b 100644 --- a/src/current/_includes/molt/replicator-flags.md +++ b/src/current/_includes/molt/replicator-flags.md @@ -39,7 +39,7 @@ | `--targetJitterTime` | `DURATION` | The time over which to jitter database pool disconnections.

**Default:** `15s` | | `--targetMaxLifetime` | `DURATION` | The maximum lifetime of a database connection.

**Default:** `5m0s` | | `--targetMaxPoolSize` | `INT` | The maximum number of target database connections.

**Default:** `128` | -| `--targetSchema` | `STRING` | The SQL database schema in the target cluster to update. | +| `--targetSchema` | `STRING` | The SQL database schema in the target cluster to update. CockroachDB schema names must be fully qualified in the format `database.schema`. | | `--targetStatementCacheSize` | `INT` | The maximum number of prepared statements to retain.

**Default:** `128` | | `--taskGracePeriod` | `DURATION` | How long to allow for task cleanup when recovering from errors.

**Default:** `1m0s` | | `--timestampLimit` | `INT` | The maximum number of source timestamps to coalesce into a target transaction.

**Default:** `1000` | diff --git a/src/current/molt/migrate-failback.md b/src/current/molt/migrate-failback.md index b6a39d78d24..2e145c7d66c 100644 --- a/src/current/molt/migrate-failback.md +++ b/src/current/molt/migrate-failback.md @@ -101,6 +101,12 @@ When you run `replicator`, you can configure the following options for replicati - [Replicator metrics](#replicator-metrics): Monitor failback replication performance. +
+ + + +
+ ### Connection strings For failback, MOLT Replicator uses `--targetConn` to specify the destination database where you want to replicate CockroachDB changes, and `--stagingConn` for the CockroachDB staging database. @@ -225,11 +231,19 @@ Create a CockroachDB changefeed to send changes to MOLT Replicator. 1. Create the CockroachDB changefeed pointing to the MOLT Replicator webhook endpoint. Use `cursor` to specify the logical timestamp from the preceding step. - {{site.data.alerts.callout_info}} - Ensure that only **one** changefeed points to MOLT Replicator at a time to avoid mixing streams of incoming data. - {{site.data.alerts.end}} +
+ The target schema is specified in the webhook URL path in the fully-qualified format `/database/schema`. For example, `/migration_schema/public` routes changes to the `public` schema in the `migration_schema` database. +
- {{site.data.alerts.callout_success}} +
+ The target schema is specified in the webhook URL path. For example, `/migration_schema` routes changes to the `migration_schema` schema. +
+ +
+ The target schema is specified in the webhook URL path. For example, `/MIGRATION_SCHEMA` routes changes to the `MIGRATION_SCHEMA` schema. +
+ + {{site.data.alerts.callout_info}} For details on the webhook sink URI, refer to [Webhook sink]({% link {{ site.current_cloud_version }}/changefeed-sinks.md %}#webhook-sink). {{site.data.alerts.end}} @@ -268,6 +282,10 @@ Create a CockroachDB changefeed to send changes to MOLT Replicator. 1101234051444375553 ~~~ + {{site.data.alerts.callout_success}} + Ensure that only **one** changefeed points to MOLT Replicator at a time to avoid mixing streams of incoming data. + {{site.data.alerts.end}} + 1. Monitor the changefeed status, specifying the job ID: ~~~ sql diff --git a/src/current/molt/migrate-load-replicate.md b/src/current/molt/migrate-load-replicate.md index d883607d518..e7a24f1d8c0 100644 --- a/src/current/molt/migrate-load-replicate.md +++ b/src/current/molt/migrate-load-replicate.md @@ -82,6 +82,12 @@ When you run `replicator`, you can configure the following options for replicati - [Replicator metrics](#replicator-metrics): Monitor replication progress and performance. +
+ + + +
+ ### Replication connection strings MOLT Replicator uses `--sourceConn` and `--targetConn` to specify the source and target database connections. diff --git a/src/current/molt/migrate-resume-replication.md b/src/current/molt/migrate-resume-replication.md index 2ccd0b68cf3..2d700822754 100644 --- a/src/current/molt/migrate-resume-replication.md +++ b/src/current/molt/migrate-resume-replication.md @@ -70,10 +70,12 @@ replicator oraclelogminer \ --sourceConn $SOURCE \ --sourcePDBConn $SOURCE_PDB \ --sourceSchema migration_schema \ +--targetSchema defaultdb.public \ --targetConn $TARGET \ --stagingSchema _replicator \ --metricsAddr :30005 \ ---userscript table_filter.ts +--userscript table_filter.ts \ +-v ~~~ {{site.data.alerts.callout_info}} diff --git a/src/current/molt/molt-replicator.md b/src/current/molt/molt-replicator.md index 5a0bfacbaee..ac1d1079694 100644 --- a/src/current/molt/molt-replicator.md +++ b/src/current/molt/molt-replicator.md @@ -269,8 +269,8 @@ These server certificates must correspond to the client certificates specified i Encode client certificates for changefeed webhook URLs: -- **Webhook URLs**: Use both URL encoding and base64 encoding: `base64 -i ./client.crt | jq -R -r '@uri'` -- **Non-webhook contexts**: Use base64 encoding only: `base64 -w 0 ca.cert` +- Webhook URLs: Use both URL encoding and base64 encoding: `base64 -i ./client.crt | jq -R -r '@uri'` +- Non-webhook contexts: Use base64 encoding only: `base64 -w 0 ca.cert` #### JWT authentication @@ -430,6 +430,13 @@ Specify the source Oracle schema to replicate from: ~~~ +Specify the target schema on CockroachDB in fully-qualified `database.schema` format: + +{% include_cached copy-clipboard.html %} +~~~ +--targetSchema defaultdb.public +~~~ + To replicate from the correct position, specify the appropriate checkpoint value.
@@ -476,6 +483,7 @@ At minimum, the `replicator` command should include the following flags: replicator pglogical \ --sourceConn $SOURCE \ --targetConn $TARGET \ +--targetSchema defaultdb.public \ --slotName molt_slot \ --stagingSchema _replicator \ --stagingCreateSchema @@ -490,6 +498,7 @@ For detailed steps, refer to [Load and replicate]({% link molt/migrate-load-repl replicator mylogical \ --sourceConn $SOURCE \ --targetConn $TARGET \ +--targetSchema defaultdb.public \ --defaultGTIDSet '4c658ae6-e8ad-11ef-8449-0242ac140006:1-29' \ --stagingSchema _replicator \ --stagingCreateSchema @@ -504,8 +513,9 @@ For detailed steps, refer to [Load and replicate]({% link molt/migrate-load-repl replicator oraclelogminer \ --sourceConn $SOURCE \ --sourcePDBConn $SOURCE_PDB \ ---sourceSchema migration_schema \ --targetConn $TARGET \ +--sourceSchema migration_schema \ +--targetSchema defaultdb.public \ --scn 26685786 \ --backfillFromSCN 26685444 \ --stagingSchema _replicator \ @@ -626,7 +636,11 @@ replicator start \ --tlsPrivateKey ./certs/server.key ~~~ -For detailed steps, refer to [Migration failback]({% link molt/migrate-failback.md %}). +After starting `replicator`, create a CockroachDB changefeed to send changes to MOLT Replicator. For detailed steps, refer to [Migration failback]({% link molt/migrate-failback.md %}). + +{{site.data.alerts.callout_info}} +When [creating the CockroachDB changefeed]({% link molt/migrate-failback.md %}#create-the-cockroachdb-changefeed) , you specify the target schema in the webhook URL path. For PostgreSQL targets, use the fully-qualified format `/database/schema` (for example, `/migration_schema/public`). For MySQL and Oracle targets, use just the schema name (for example, `/migration_schema`). +{{site.data.alerts.end}} ## Monitoring