Implement heartbeat in Debezium for Informix connector

[dumcatalin81]

Feature request or enhancement

Add heartbeat feature in Debezium for the Informix connector. We also have a ticket (TS021250785) requesting a workaround to prevent de-synchronization.

For feature requests or enhancements, please provide the following information:

Which use case/requirement will be addressed by the proposed feature?
The reason why this feature would be useful is that the connectors are not always connected to the database and we run in a de-synchronization issue that breaks data continuity. As other flows fill up the CDC logs, the logs then get deleted, between our connections.

Thank you,

[jpechane]

Hi, would you be able to send a PR? Also which ticket do you refer to?

[Naros]

Hi @jpechane, the ticket is an internal customer ticket.

@dumcatalin81 the discussion for TS021250785 indicated the user was getting this error

Cannot invoke "io.debezium.heartbeat.HeartbeatConnectionProvider.get()" because "connectionProvider" is null

Heartbeats aim to address two core issues.

The first is when you just set heartbeat.interval.ms. This ensures that the offset topic is periodically updated to reflect the current state of the Debezium connector. This has no direct interaction with the database; it's simply a small Kafka Connect event to the Kafka broker.

The second is when you pair the former with heartbeat.action.query. This is where the connector also writes to a database table based on the heartbeat interval. This triggers an event that the connector captures and sends to Kafka, updating the connector offsets in the process.

The action query aims to address the issue when the connector is running, but there is significantly more activity on a secondary database on the same instance or in non-captured tables. Some connectors do not update their internal offset context unless changes are captured. The action query ensures the offset context details are updated periodically, so that upon a connector restart, the offsets are not stale.

Based on the description and conversation, the user states that the connector is not always connected to the database. This sounds to me as though the user is using the connector in a "batch" fashion to periodically process changes and leaves the connector down in between those "batch" runs. If that's true, heartbeats won't resolve the user's issue.

Debezium is designed to be started and to run uninterrupted for as long as possible. If Debezium's execution is interrupted for any reason, the responsibility for maintaining transaction logs with the last offset position to support a successful restart rests with the user, not Debezium.

To help us better understand the circumstances, could you elaborate on how the user is using Debezium? Is it indeed in this "batch" like scenario, or is it running indefinitely as expected?