Skip to content

Commit bc7fe84

Browse files
tarofacetaroface
committed
restructure replication-only doc
1 parent a32aac3 commit bc7fe84

File tree

2 files changed

+101
-116
lines changed

2 files changed

+101
-116
lines changed

‎src/current/molt/migrate-resume-replication.md‎

Lines changed: 95 additions & 114 deletions
Original file line numberDiff line numberDiff line change
@@ -22,133 +22,114 @@ To resume replication after an interruption, refer to [Resume replication after
2222
If you want to start replication without first running [MOLT Fetch]({% link molt/molt-fetch.md %}), you need to manually obtain a replication checkpoint from the source database and then start MOLT Replicator with the appropriate checkpoint flags.
2323

2424
<section class="filter-content" markdown="1" data-scope="postgres">
25-
### Create publication and replication slot
26-
27-
Create a publication for the tables you want to replicate:
28-
29-
{% include_cached copy-clipboard.html %}
30-
~~~ sql
31-
CREATE PUBLICATION molt_publication FOR TABLE employees, payments, orders;
32-
~~~
33-
34-
Create a replication slot to track the LSN checkpoint:
35-
36-
{% include_cached copy-clipboard.html %}
37-
~~~ sql
38-
SELECT pg_create_logical_replication_slot('molt_slot', 'pgoutput');
39-
~~~
40-
41-
~~~
42-
pg_create_logical_replication_slot
43-
-------------------------------------
44-
(molt_slot,0/167A220)
45-
~~~
46-
47-
### Start Replicator
48-
49-
Run the `replicator pglogical` command, specifying the slot name with `--slotName`. The replication slot automatically tracks the LSN checkpoint, so you don't need to specify an LSN value. Use `--stagingSchema` to specify a unique name for the staging database, and include `--stagingCreateSchema` to have MOLT Replicator automatically create the staging database:
50-
51-
{% include_cached copy-clipboard.html %}
52-
~~~ shell
53-
replicator pglogical \
54-
--sourceConn $SOURCE \
55-
--targetConn $TARGET \
56-
--targetSchema defaultdb.public \
57-
--slotName molt_slot \
58-
--stagingSchema _replicator \
59-
--stagingCreateSchema \
60-
--metricsAddr :30005 \
61-
--verbose
62-
~~~
25+
1. Create a publication for the tables you want to replicate:
26+
27+
{% include_cached copy-clipboard.html %}
28+
~~~ sql
29+
CREATE PUBLICATION molt_publication FOR TABLE employees, payments, orders;
30+
~~~
31+
32+
1. Create a replication slot to track the LSN checkpoint:
33+
34+
{% include_cached copy-clipboard.html %}
35+
~~~ sql
36+
SELECT pg_create_logical_replication_slot('molt_slot', 'pgoutput');
37+
~~~
38+
39+
~~~
40+
pg_create_logical_replication_slot
41+
-------------------------------------
42+
(molt_slot,0/167A220)
43+
~~~
44+
45+
1. Run the `replicator pglogical` command, specifying the slot name with `--slotName`. The replication slot automatically tracks the LSN checkpoint, so you don't need to specify an LSN value. Use `--stagingSchema` to specify a unique name for the staging database, and include `--stagingCreateSchema` to have MOLT Replicator automatically create the staging database:
46+
47+
{% include_cached copy-clipboard.html %}
48+
~~~ shell
49+
replicator pglogical \
50+
--sourceConn $SOURCE \
51+
--targetConn $TARGET \
52+
--targetSchema defaultdb.public \
53+
--slotName molt_slot \
54+
--stagingSchema _replicator \
55+
--stagingCreateSchema \
56+
--metricsAddr :30005 \
57+
--verbose
58+
~~~
6359
</section>
6460

6561
<section class="filter-content" markdown="1" data-scope="mysql">
66-
### Get GTID checkpoint
67-
68-
Get the current GTID set to use as the starting point for replication:
69-
70-
{% include_cached copy-clipboard.html %}
71-
~~~ sql
72-
-- For MySQL < 8.0:
73-
SHOW MASTER STATUS;
74-
-- For MySQL 8.0+:
75-
SHOW BINARY LOG STATUS;
76-
~~~
77-
78-
~~~
79-
+---------------+----------+--------------+------------------+-------------------------------------------+
80-
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
81-
+---------------+----------+--------------+------------------+-------------------------------------------+
82-
| binlog.000005 | 197 | | | 77263736-7899-11f0-81a5-0242ac120002:1-38 |
83-
+---------------+----------+--------------+------------------+-------------------------------------------+
84-
~~~
85-
86-
### Start Replicator
87-
88-
Run the `replicator mylogical` command, specifying the GTID from the `Executed_Gtid_Set` value with `--defaultGTIDSet`. Use `--stagingSchema` to specify a unique name for the staging database, and include `--stagingCreateSchema` to have MOLT Replicator automatically create the staging database:
89-
90-
{% include_cached copy-clipboard.html %}
91-
~~~ shell
92-
replicator mylogical \
93-
--sourceConn $SOURCE \
94-
--targetConn $TARGET \
95-
--targetSchema defaultdb.public \
96-
--defaultGTIDSet 77263736-7899-11f0-81a5-0242ac120002:1-38 \
97-
--stagingSchema _replicator \
98-
--stagingCreateSchema \
99-
--metricsAddr :30005 \
100-
--userscript table_filter.ts \
101-
--verbose
102-
~~~
103-
104-
{{site.data.alerts.callout_success}}
105-
For MySQL versions that do not support `binlog_row_metadata`, include `--fetchMetadata` to explicitly fetch column metadata. This requires additional permissions on the source MySQL database. Grant `SELECT` permissions with `GRANT SELECT ON source_database.* TO 'migration_user'@'localhost';`. If that is insufficient for your deployment, use `GRANT PROCESS ON *.* TO 'migration_user'@'localhost';`, though this is more permissive and allows seeing processes and server status.
106-
{{site.data.alerts.end}}
107-
</section>
62+
1. Get the current GTID set to use as the starting point for replication:
10863

109-
<section class="filter-content" markdown="1" data-scope="oracle">
110-
### Get SCN checkpoint
64+
{% include_cached copy-clipboard.html %}
65+
~~~ sql
66+
-- For MySQL < 8.0:
67+
SHOW MASTER STATUS;
68+
-- For MySQL 8.0+:
69+
SHOW BINARY LOG STATUS;
70+
~~~
11171

112-
Obtain the correct SCNs to use as the starting point for replication. Run the following queries on the PDB in the order shown:
72+
~~~
73+
+---------------+----------+--------------+------------------+-------------------------------------------+
74+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
75+
+---------------+----------+--------------+------------------+-------------------------------------------+
76+
| binlog.000005 | 197 | | | 77263736-7899-11f0-81a5-0242ac120002:1-38 |
77+
+---------------+----------+--------------+------------------+-------------------------------------------+
78+
~~~
79+
80+
1. Run the `replicator mylogical` command, specifying the GTID from the `Executed_Gtid_Set` value with `--defaultGTIDSet`. Use `--stagingSchema` to specify a unique name for the staging database, and include `--stagingCreateSchema` to have MOLT Replicator automatically create the staging database:
11381

114-
{% include_cached copy-clipboard.html %}
115-
~~~ sql
116-
-- Query the current SCN from Oracle
117-
SELECT CURRENT_SCN FROM V$DATABASE;
82+
{% include_cached copy-clipboard.html %}
83+
~~~ shell
84+
replicator mylogical \
85+
--sourceConn $SOURCE \
86+
--targetConn $TARGET \
87+
--targetSchema defaultdb.public \
88+
--defaultGTIDSet 77263736-7899-11f0-81a5-0242ac120002:1-38 \
89+
--stagingSchema _replicator \
90+
--stagingCreateSchema \
91+
--metricsAddr :30005 \
92+
--userscript table_filter.ts \
93+
--verbose
94+
~~~
11895

119-
-- Query the starting SCN of the earliest active transaction
120-
SELECT MIN(t.START_SCNB) FROM V$TRANSACTION t;
121-
~~~
96+
{{site.data.alerts.callout_success}}
97+
For MySQL versions that do not support `binlog_row_metadata`, include `--fetchMetadata` to explicitly fetch column metadata. This requires additional permissions on the source MySQL database. Grant `SELECT` permissions with `GRANT SELECT ON source_database.* TO 'migration_user'@'localhost';`. If that is insufficient for your deployment, use `GRANT PROCESS ON *.* TO 'migration_user'@'localhost';`, though this is more permissive and allows seeing processes and server status.
98+
{{site.data.alerts.end}}
99+
</section>
122100

123-
Use the results as follows:
101+
<section class="filter-content" markdown="1" data-scope="oracle">
102+
1. Get the current SCN to use as the starting point for replication:
124103

125-
- `--scn`: Use the result from the first query (current SCN)
126-
- `--backfillFromSCN`: Use the result from the second query (earliest active transaction SCN). If the second query returns no results, use the result from the first query instead.
104+
{% include_cached copy-clipboard.html %}
105+
~~~ sql
106+
SELECT CURRENT_SCN FROM V$DATABASE;
107+
~~~
127108

128-
### Start Replicator
109+
Use this SCN value for both `--backfillFromSCN` and `--scn` flags.
129110

130-
Run the `replicator oraclelogminer` command, specifying the SCN values from the checkpoint queries. Use `--stagingSchema` to specify a unique name for the staging database, and include `--stagingCreateSchema` to have MOLT Replicator automatically create the staging database:
111+
1. Run the `replicator oraclelogminer` command, specifying the SCN values from the checkpoint queries. Use `--stagingSchema` to specify a unique name for the staging database, and include `--stagingCreateSchema` to have MOLT Replicator automatically create the staging database:
131112

132-
{% include_cached copy-clipboard.html %}
133-
~~~ shell
134-
replicator oraclelogminer \
135-
--sourceConn $SOURCE \
136-
--sourcePDBConn $SOURCE_PDB \
137-
--targetConn $TARGET \
138-
--sourceSchema migration_schema \
139-
--targetSchema defaultdb.public \
140-
--backfillFromSCN 26685444 \
141-
--scn 26685786 \
142-
--stagingSchema _replicator \
143-
--stagingCreateSchema \
144-
--metricsAddr :30005 \
145-
--userscript table_filter.ts \
146-
--verbose
147-
~~~
113+
{% include_cached copy-clipboard.html %}
114+
~~~ shell
115+
replicator oraclelogminer \
116+
--sourceConn $SOURCE \
117+
--sourcePDBConn $SOURCE_PDB \
118+
--targetConn $TARGET \
119+
--sourceSchema migration_schema \
120+
--targetSchema defaultdb.public \
121+
--backfillFromSCN 26685444 \
122+
--scn 26685786 \
123+
--stagingSchema _replicator \
124+
--stagingCreateSchema \
125+
--metricsAddr :30005 \
126+
--userscript table_filter.ts \
127+
--verbose
128+
~~~
148129

149-
{{site.data.alerts.callout_info}}
150-
When filtering out tables in a schema with a userscript, replication performance may decrease because filtered tables are still included in LogMiner queries and processed before being discarded.
151-
{{site.data.alerts.end}}
130+
{{site.data.alerts.callout_info}}
131+
When filtering out tables in a schema with a userscript, replication performance may decrease because filtered tables are still included in LogMiner queries and processed before being discarded.
132+
{{site.data.alerts.end}}
152133
</section>
153134

154135
## Resume replication after interruption

‎src/current/molt/molt-replicator.md‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -57,9 +57,13 @@ Oracle source (`oraclelogminer`): MOLT Replicator uses Oracle LogMiner to captur
5757

5858
MOLT Replicator offers three consistency modes for balancing throughput and transactional guarantees:
5959

60-
1. Consistent (default for CockroachDB sources): Preserves per-row order and source transaction atomicity. Concurrent transactions are controlled by `--parallelism`.
60+
1. Consistent (default for CockroachDB sources, failback mode only): Preserves per-row order and source transaction atomicity. Concurrent transactions are controlled by `--parallelism`.
6161

62-
1. BestEffort: Relaxes atomicity across tables that do not have foreign key constraints between them (maintains coherence within FK-connected groups). Enable with `--bestEffortOnly` or allow auto-entry via `--bestEffortWindow` set to a positive duration (e.g., `1s`).
62+
1. BestEffort (failback mode only): Relaxes atomicity across tables that do not have foreign key constraints between them (maintains coherence within FK-connected groups). Enable with `--bestEffortOnly` or allow auto-entry via `--bestEffortWindow` set to a positive duration (e.g., `1s`).
63+
64+
{{site.data.alerts.callout_info}}
65+
For independent tables (with no foreign key constraints), BestEffort mode applies changes immediately as they arrive, without waiting for the resolved timestamp. This provides higher throughput for tables that have no relationships with other tables.
66+
{{site.data.alerts.end}}
6367

6468
1. Immediate (default for PostgreSQL, MySQL, and Oracle sources): Applies updates as they arrive to Replicator with no buffering or waiting for resolved timestamps. Provides highest throughput but requires no foreign keys on the target schema.
6569

0 commit comments

Comments
 (0)