Skip to content

Add documentation for MilliSecondsBehindSource growing exponentially. #700

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rnatarajan wants to merge 2 commits into
base: develop
Choose a base branch
from
Open

Add documentation for MilliSecondsBehindSource growing exponentially. #700

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6c0d342
Update faq.asciidoc
rnatarajan Aug 6, 2021
9ecaf67
Merge branch 'develop' into develop
rnatarajan Jan 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 18 additions & 1 deletion documentation/faq.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -413,6 +413,23 @@ If the global change is not desirable then the connector can override the defaul

In the latter case it is also necessary to configure `connector.client.config.override.policy=ALL` option in Kafka Connect worker config file `connect-distributed.properties`. For Debezium `connect` Docker image the environment variable `CONNECT_CONNECTOR_CLIENT_CONFIG_OVERRIDE_POLICY` can be used to configure the option.

== Why MilliSecondsBehindSource is growing exponentially ?

When Debezium Connector is replicating binlog, it is possible that Debezium Connector is not able to keep up with rate at which CDC events are generated in the upstream database.
Observe the streaming metrics MilliSecondsBehindSource. MilliSecondsBehindSource would increase exponentially.

To solve the issue, identify the round trip time of a packet from the machine on which Kafka connect is running to database host.

```
ping -c 10 <database.hostname>
```

If the round trip time is in few milliseconds(For Example 20 or 30 milliseconds and not 0.1 or 0.5 milliseconds), then time taken by Kafka Connect to connect with upstream database is high.

For streams generating fewer CDC events, Kafka connect will be able to keep up with CDC events even with high round trip time.
However for a stream generating high volume of CDC data, Kafka connect will not be able to keep up with CDC data and hence MilliSecondsBehindSource will grow exponentially.

Move Kafka Connect with Debezium connector to a host or machine from which database can be reached faster or round trip time is less than a millisecond.
== Why do JSON messages not contain schema?

If you are using `JsonConverter` to convert and serialize the messages emitted by Debezium, the schema is not included in the messages by default.
Expand Down Expand Up @@ -471,4 +488,4 @@ add the https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-tr
</configuration>
----

Alternatively, if you use the https://maven.apache.org/plugins/maven-assembly-plugin/index.html[Maven Assembly plug-in], you can use the https://maven.apache.org/plugins/maven-assembly-plugin/examples/single/using-container-descriptor-handlers.html[metaInf-services container descriptor handlers].
Alternatively, if you use the https://maven.apache.org/plugins/maven-assembly-plugin/index.html[Maven Assembly plug-in], you can use the https://maven.apache.org/plugins/maven-assembly-plugin/examples/single/using-container-descriptor-handlers.html[metaInf-services container descriptor handlers].
Loading