Is TiCDC compatible with TiDB v4.0.14

[sykp241095]

Original author: prateekrckstr
Original time: 2025-01-14 08:59:26.673855
Original ID/Slug: 10372 / is-ticdc-compatible-with-tidb-v4-0-14

Application environment:

Production

TiDB version: 4.0.14

####Problem
We’re having trouble setting up TiCDC replication and are looking for assistance. Has anyone successfully implemented TiCDC replication? If so, we’d appreciate it if you could share your setup process. Since we’re constrained to TiDB v4.0.14 and cannot upgrade, we’re particularly interested in any workarounds or specific configurations for this version.

[sykp241095]

Original author: TiDB-AI-Assistant
Original time: 2025-01-14 09:00:20.075283

To set up TiCDC replication with TiDB v4.0.14, follow this detailed guide, which includes compatibility considerations, known issues, and specific configurations or workarounds.

Compatibility

TiCDC is compatible with TiDB versions 4.0.6 and later, including v4.0.14. It is designed to capture and replicate incremental data changes from TiDB clusters to various downstream systems such as MySQL, Kafka, and other storage services. Ensure that your TiDB version is at least 4.0.6 to utilize TiCDC effectively.

Known Issues in TiDB v4.0.14

1. Partition Dispatching: There was an issue with extra partition dispatching after adding new table partitions. This has been fixed in TiDB v4.0.14. Source

2. Memory Consumption: TiCDC’s runtime memory consumption has been reduced in this version, addressing previous concerns about high memory usage. Source

3. MySQL Connection Leaks: A bug that caused some MySQL connections to leak after encountering errors has been fixed. Source

4. Changefeed Creation: An issue where TiCDC changefeed could not be created when the start timestamp was less than the current timestamp minus GC TTL has been resolved. Source

5. Replication Task Stops: A bug that could cause the replication task to stop when moving a table has been fixed. Source

Configuration and Setup

1. Installation: TiCDC can be installed and managed using TiUP, a cluster management tool. Ensure that TiUP is installed and updated to the latest version.

2. Security: TiCDC supports TLS for secure communication. Configure TLS settings in the TiCDC configuration files or via command-line arguments to ensure secure data transmission.

3. Configuration Files: TiCDC can be configured using configuration files. Ensure that the configuration aligns with your network setup and downstream requirements. Pay attention to settings like max-message-bytes and partition-num if using Kafka as a sink.

4. High Availability: TiCDC provides high availability through PD’s etcd. Ensure that your PD cluster is properly configured and monitored to maintain high availability.

5. Data Consistency: TiCDC ensures data consistency using tidb_external_ts. Make sure this setting is correctly configured to avoid data inconsistency issues.

6. Limitations: Be aware that TiCDC has limitations regarding DDL operations within imported SQL statements. Plan your schema changes accordingly to avoid disruptions.

Workarounds

• If you encounter issues with memory usage or replication tasks stopping unexpectedly, consider adjusting the memory allocation and monitoring the system resources closely.
• For any issues related to changefeed creation, ensure that the GC TTL settings are appropriately configured to prevent timestamp conflicts.

By following these guidelines, you can effectively set up and manage TiCDC replication with TiDB v4.0.14. For further details, refer to the official TiDB documentation.