new Replicator metrics; reviewer comments

Author: taroface

src/current/_includes/molt/molt-troubleshooting-failback.md

@@ -12,20 +12,23 @@ This indicates that Replicator is down, the webhook URL is incorrect, or the por
 
 **Resolution:** Verify that MOLT Replicator is running on the port specified in the changefeed `INTO` configuration. Confirm the host and port are correct.
 
-##### Unknown schema error
+##### Incorrect schema path errors
+
+This error occurs when the [CockroachDB changefeed]({% link {{ site.current_cloud_version }}/create-and-configure-changefeeds.md %}) webhook URL path does not match the target database schema naming convention:
 
 ~~~
 transient error: 400 Bad Request: unknown schema:
 ~~~
 
-This indicates the webhook URL path is incorrectly formatted. Common causes include using the wrong path format for your target database type or incorrect database names.
+The webhook URL path is specified in the `INTO` clause when you [create the changefeed](#create-the-cockroachdb-changefeed). For example: `webhook-https://replicator-host:30004/database/schema`.
 
-**Resolution:** Check the webhook URL path mapping:
+**Resolution:** Verify the webhook path format matches your target database type:
 
-- **PostgreSQL targets:** Use `/database/schema` format (for example, `/molt_db/public`).
-- **MySQL/Oracle targets:** Use `/SCHEMA` format (for example, `/MOLT_DB`). Use only the schema name (for example, `molt` instead of `molt/public`).
+- PostgreSQL or CockroachDB targets: Use `/database/schema` format. For example, `webhook-https://replicator-host:30004/migration_schema/public`.
+- MySQL targets: Use `/database` format (schema is implicit). For example, `webhook-https://replicator-host:30004/migration_schema`.
+- Oracle targets: Use `/DATABASE` format in uppercase. For example, `webhook-https://replicator-host:30004/MIGRATION_SCHEMA`.
 
-Verify that the target database and schema names match the webhook URL.
+For details on configuring the webhook sink URI, refer to [Webhook sink]({% link {{ site.current_cloud_version }}/changefeed-sinks.md %}#webhook-sink).
 
 ##### GC threshold error
 

src/current/_includes/molt/molt-troubleshooting-fetch.md

@@ -2,10 +2,14 @@
 
 ##### Fetch exits early due to mismatches
 
-`molt fetch` exits early in the following cases, and will output a log with a corresponding `mismatch_tag` and `failable_mismatch` set to `true`:
+When run in `none` or `truncate-if-exists` mode, `molt fetch` exits early in the following cases, and will output a log with a corresponding `mismatch_tag` and `failable_mismatch` set to `true`:
 
 - A source table is missing a primary key.
 - A source primary key and target primary key have mismatching types.
+    {{site.data.alerts.callout_success}}
+    These restrictions (missing or mismatching primary keys) can be bypassed with [`--skip-pk-check`]({% link molt/molt-fetch.md %}#skip-primary-key-matching).
+    {{site.data.alerts.end}}
+
 - A [`STRING`]({% link {{site.current_cloud_version}}/string.md %}) primary key has a different [collation]({% link {{site.current_cloud_version}}/collate.md %}) on the source and target.
 - A source and target column have mismatching types that are not [allowable mappings]({% link molt/molt-fetch.md %}#type-mapping).
 - A target table is missing a column that is in the corresponding source table.

src/current/_includes/molt/molt-troubleshooting-replication.md

@@ -6,14 +6,14 @@ If MOLT Replicator appears hung or performs poorly:
 
 1. Enable trace logging with `-vv` to get more visibility into the replicator's state and behavior.
 
-1. If MOLT Replicator is in an unknown, hung, or erroneous state, collect performance profiles to include with support tickets:
+1. If MOLT Replicator is in an unknown, hung, or erroneous state, collect performance profiles to include with support tickets. Replace `{host}` and `{metrics-port}` with your Replicator host and the port specified by `--metricsAddr`:
 
     {% include_cached copy-clipboard.html %}
     ~~~shell
-    curl 'localhost:30005/debug/pprof/trace?seconds=15' > trace.out
-    curl 'localhost:30005/debug/pprof/profile?seconds=15' > profile.out
-    curl 'localhost:30005/debug/pprof/goroutine?seconds=15' > gr.out
-    curl 'localhost:30005/debug/pprof/heap?seconds=15' > heap.out
+    curl '{host}:{metrics-port}/debug/pprof/trace?seconds=15' > trace.out
+    curl '{host}:{metrics-port}/debug/pprof/profile?seconds=15' > profile.out
+    curl '{host}:{metrics-port}/debug/pprof/goroutine?seconds=15' > gr.out
+    curl '{host}:{metrics-port}/debug/pprof/heap?seconds=15' > heap.out
     ~~~
 
 1. Monitor lag metrics and adjust performance parameters as needed.
@@ -62,7 +62,7 @@ Dropping a replication slot can be destructive and delete data that is not yet r
 run CREATE PUBLICATION molt_fetch FOR ALL TABLES;
 ~~~
 
-**Resolution:** {% if page.name != "migrate-load-replicate.md" %}[Create the publication]({% link molt/migrate-load-replicate.md %}#configure-source-database-for-replication){% else %}[Create the publication](#configure-source-database-for-replication){% endif %} on the source database. Ensure you also create the replication slot:
+**Resolution:** Create the publication on the source database. Ensure you also create the replication slot:
 
 {% include_cached copy-clipboard.html %}
 ~~~ sql
@@ -139,14 +139,18 @@ Interpret the results as follows:
 
 If the GTID is purged or invalid, follow these steps:
 
-1. Increase binlog retention by configuring `binlog_expire_logs_seconds` in MySQL or through your cloud provider:
+1. Increase binlog retention by configuring `binlog_expire_logs_seconds` in MySQL:
 
     {% include_cached copy-clipboard.html %}
     ~~~ sql
     -- Increase binlog retention (example: 7 days = 604800 seconds)
     SET GLOBAL binlog_expire_logs_seconds = 604800;
     ~~~
 
+    {{site.data.alerts.callout_info}}
+    For managed MySQL services (such as Amazon RDS, Google Cloud SQL, or Azure Database for MySQL), binlog retention is typically configured through the provider's console or CLI. Consult your provider's documentation for how to adjust binlog retention settings.
+    {{site.data.alerts.end}}
+
 1. Get a current GTID set to restart replication:
 
     {% include_cached copy-clipboard.html %}
@@ -195,10 +199,12 @@ Oracle LogMiner excludes tables and columns with names longer than 30 characters
 ##### Unsupported data types
 
 LogMiner and replication do not support:
-- Long BLOB/CLOBs (4000+ characters)
+
+- Long `BLOB`/`CLOB`s (4000+ characters)
 - User-defined types (UDTs)
 - Nested tables
 - Varrays
+- `GEOGRAPHY` and `GEOMETRY`
 
 **Resolution:** Convert unsupported data types or exclude affected tables from replication.
 
@@ -218,7 +224,7 @@ SQL NULL and JSON null values are not distinguishable in JSON payloads during re
 
 If the Oracle redo log files are too small or do not retain enough history, you may get errors indicating that required log files are missing for a given SCN range, or that a specific SCN is unavailable.
 
-Increase the number and size of online redo log files, and verify that archived log files are being generated and retained correctly in your Oracle environment.
+**Resolution:** Increase the number and size of online redo log files, and verify that archived log files are being generated and retained correctly in your Oracle environment.
 
 ##### Replicator lag
 
@@ -244,14 +250,4 @@ WARNING: warning during tryCommit: ERROR: duplicate key value violates unique co
 ERROR: maximum number of retries (10) exceeded
 ~~~
 
-**Resolution:** Check target database constraints and connection stability. MOLT Replicator will log warnings for each retry attempt. If you see warnings but no final error, the apply succeeded after retrying. If all retry attempts are exhausted, Replicator will surface a final error and restart the apply loop to continue processing.
-
-##### Incorrect schema path errors
-
-Schema path mismatches in changefeed URLs:
-
-~~~
-transient error: 400 Bad Request: unknown schema:
-~~~
-
-**Resolution:** Verify the webhook path matches your target database schema. Use `/database/schema` for CockroachDB/PostgreSQL targets and `/DATABASE` for MySQL/Oracle targets.
+**Resolution:** Check target database constraints and connection stability. MOLT Replicator will log warnings for each retry attempt. If you see warnings but no final error, the apply succeeded after retrying. If all retry attempts are exhausted, Replicator will surface a final error and restart the apply loop to continue processing.

src/current/_includes/molt/replicator-flags.md

@@ -44,7 +44,7 @@
 | `--taskGracePeriod`                    | `DURATION` | How long to allow for task cleanup when recovering from errors.<br><br>**Default:** `1m0s`                                                                                                                                                                                                                                      |
 | `--timestampLimit`                     | `INT`      | The maximum number of source timestamps to coalesce into a target transaction.<br><br>**Default:** `1000`                                                                                                                                                                                                                       |
 | `--userscript`                         | `STRING`   | The path to a TypeScript configuration script. For example, `--userscript 'script.ts'`.                                                                                                                                                                                                                                         |
-| `-v`, `--verbose`                      | `COUNT`    | Increase logging verbosity to `debug`; repeat for `trace`.                                                                                                                                                                                                                                                                      |
+| `-v`, `--verbose`                      | `COUNT`    | Increase logging verbosity. Use `-v` for `debug` logging or `-vv` for `trace` logging.                                                                                                                                                                                                                                          |
 
 ### `pglogical` replication flags
 

src/current/_includes/molt/replicator-metrics.md

@@ -7,21 +7,31 @@ Cockroach Labs recommends monitoring the following metrics during replication:
 {% if page.name == "migrate-failback.md" %}
 |              Metric Name              |                                                         Description                                                         |
 |---------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|
-| `source_lag_seconds`                  | Time between when an incoming resolved MVCC timestamp originated on the source CockroachDB cluster and when it was received by Replicator. |
-| `target_lag_seconds`                  | End-to-end lag from when an incoming resolved MVCC timestamp originated on the source CockroachDB to when all data changes up to that timestamp were written to the target database. |
-| `source_lag_seconds_histogram`        | Same as `source_lag_seconds` but stored as a histogram for analyzing distributions over time. |
-| `target_lag_seconds_histogram`        | Same as `target_lag_seconds` but stored as a histogram for analyzing distributions over time. |
-| `replicator_applier_mutations_staged` | Number of mutations that have been staged for application to the target database.                                          |
-| `replicator_applier_mutations_applied` | Number of mutations that have been successfully applied to the target database.                                           |
+| `commit_to_stage_lag_seconds`         | Time between when a mutation is written to the source CockroachDB cluster and when it is written to the staging database. |
+| `source_commit_to_apply_lag_seconds`  | End-to-end lag from when a mutation is written to the source CockroachDB cluster to when it is applied to the target database. |
+| `stage_mutations_total`               | Number of mutations staged for application to the target database.                                                         |
+| `apply_conflicts_total`               | Number of rows that experienced a compare-and-set (CAS) conflict.                                                           |
+| `apply_deletes_total`                 | Number of rows deleted.                                                                                                     |
+| `apply_duration_seconds`              | Length of time it took to successfully apply mutations.                                                                     |
+| `apply_errors_total`                  | Number of times an error was encountered while applying mutations.                                                          |
+| `apply_resolves_total`                | Number of rows that experienced a compare-and-set (CAS) conflict and which were resolved.                                   |
+| `apply_upserts_total`                 | Number of rows upserted.                                                                                                    |
+| `target_apply_queue_depth`            | Number of batches in the target apply queue. Indicates how backed up the applier flow is between receiving changefeed data and applying it to the target database. |
+| `target_apply_queue_utilization_percent` | Utilization percentage (0.0-100.0) of the target apply queue capacity. Use this to understand how close the queue is to capacity and to set alerting thresholds for backpressure conditions. |
+| `core_parallelism_utilization_percent` | Current utilization percentage of the applier flow parallelism capacity. Shows what percentage of the configured parallelism is actively being used. |
 {% else %}
 |              Metric Name              |                                                         Description                                                         |
 |---------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|
-| `source_lag_seconds_histogram`        | Time between when a source transaction is committed and when its COMMIT transaction log arrives at Replicator. |
-| `target_lag_seconds_histogram`        | End-to-end lag from when a source transaction is committed to when its changes are fully written to the target CockroachDB. |
-| `replicator_applier_mutations_staged` | Number of mutations that have been staged for application to the target database.                                          |
-| `replicator_applier_mutations_applied` | Number of mutations that have been successfully applied to the target database.                                           |
+| `commit_to_stage_lag_seconds`         | Time between when a mutation is written to the source database and when it is written to the staging database.             |
+| `source_commit_to_apply_lag_seconds`  | End-to-end lag from when a mutation is written to the source database to when it is applied to the target CockroachDB.     |
+| `apply_conflicts_total`               | Number of rows that experienced a compare-and-set (CAS) conflict.                                                           |
+| `apply_deletes_total`                 | Number of rows deleted.                                                                                                     |
+| `apply_duration_seconds`              | Length of time it took to successfully apply mutations.                                                                     |
+| `apply_errors_total`                  | Number of times an error was encountered while applying mutations.                                                          |
+| `apply_resolves_total`                | Number of rows that experienced a compare-and-set (CAS) conflict and which were resolved.                                   |
+| `apply_upserts_total`                 | Number of rows upserted.                                                                                                    |
 {% endif %}
 
-You can use the [Replicator Grafana dashboard](https://replicator.cockroachdb.com/replicator_grafana_dashboard.json) to visualize these metrics. <section class="filter-content" markdown="1" data-scope="oracle">For Oracle-specific metrics, import [this Oracle Grafana dashboard](https://replicator.cockroachdb.com/replicator_oracle_grafana_dashboard.json).</section>
+You can use the [Replicator Grafana dashboard](https://replicator.cockroachdb.com/replicator_grafana_dashboard.json) to visualize the metrics. <section class="filter-content" markdown="1" data-scope="oracle">For Oracle-specific metrics, import the [Oracle Grafana dashboard](https://replicator.cockroachdb.com/replicator_oracle_grafana_dashboard.json).</section>
 
 To check MOLT Replicator health when metrics are enabled, run `curl http://localhost:30005/_/healthz` (replacing the port with your `--metricsAddr` value). This returns a status code of `200` if Replicator is running.