Node <Number> disconnected INFOs

[bahaeddinoz]

Hello,

when i check the logs of kafka nodes, producer (debezium), consumers i saw that nodes are getting disconnected in some invervals 9-10mins. I think each time connection got lost. Upgraded strimzi (0.31.0) and kafka (3.2.0) versions to 0.33.2 and 3.2.3 but nothing changed. I have 3 kafka brokers configured and maybe that's why do not feel anything for now but do you know what do we have this disconnection INFOs on logs.

"2023-05-04 08:08:44,481 INFO [TransactionCoordinator id=0] Node 1 disconnected. (org.apache.kafka.clients.NetworkClient) [TxnMarkerSenderThread-0]"
"2023-05-04 08:08:45,694 INFO [TransactionCoordinator id=0] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient) [TxnMarkerSenderThread-0]"
"2023-05-04 08:08:48,415 INFO [TransactionCoordinator id=2] Node 0 disconnected. (org.apache.kafka.clients.NetworkClient) [TxnMarkerSenderThread-2]"
"2023-05-04 08:08:56,482 INFO [Producer clientId=producer-1] Node 0 disconnected. (org.apache.kafka.clients.NetworkClient) [kafka-producer-network-thread | producer-1]"
"2023-05-04 08:09:06,324 INFO [dev-connector|task-0] [AdminClient clientId=connector-adminclient-dev-connector-0] Node 0 disconnected. (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | connector-adminclient-dev-connector-0]"

When i change the connections.max.idle.ms the INFO's interval also changes but it i don't wanna change this value. The kafka brokers are not idle actually.

Kafka config:
config:
default.replication.factor: 3
min.insync.replicas: 2
offsets.topic.replication.factor: 3
offsets.retention.minutes: 525600
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
inter.broker.protocol.version: "3.2.3"

Connector config:
config:
group.id: connect-cluster
offset.storage.topic: connect-cluster-offsets
config.storage.topic: connect-cluster-configs
status.storage.topic: connect-cluster-status
config.storage.replication.factor: 3
offset.storage.replication.factor: 3
status.storage.replication.factor: 3
offsets.topic.replication.factor: 3
config.providers: directory
config.providers.directory.class: org.apache.kafka.common.config.provider.DirectoryConfigProvider
producer.max.in.flight.requests.per.connection: 5
producer.retries: 2147483647
producer.acks: all
producer.enable.idempotence: true

[scholzj]

When i change the connections.max.idle.ms the INFO's interval also changes but it i don't wanna change this value. The kafka brokers are not idle actually.

The connections.max.idle.ms is not about the brokers being idle but the connection being idle. So if that impacts these connections, they are probably idle. I think the connections can be idle for various reasons - for example, they do not have anything to send for a long time to the given broker because the keys of your messages now point to partitions on other brokers etc. So I do not think these mean necessarily some issue. It is just Kafka trying to conserve resources by not maintaining connections that are not needed right now.

[bahaeddinoz]

Thanks for the quick help @scholzj. But i have heartbeat events created by producer every 1 minute. So it shouldn't be idle connection between debezium connector and brokers. That's why i said actually i am sure why it should not be idle. I think i can check the idle logs of kafka after enabled Trace logging on kafka but what do you think about this ?

[scholzj]

But i have heartbeat events created by producer every 1 minute. So it shouldn't be idle connection between debezium connector and brokers. That's why i said actually i am sure why it should not be idle. I think i can check the idle logs of kafka after enabled Trace logging on kafka but what do you think about this ?

Do you mean that you are sending fake Kafka messages to generate some traffic? How do you do that? Given how the Kafka protocol works, it is non-trivial to design the fake messages to target all the different brokers in the cluster where the connection might have been opened. So unless you somehow added that logic to every client, you would still have some idle connections from time to time. Also, keep in mind that for example in the case of Connect there are many connections -> the Connect's Admin API connections to manage its own topics, Connect's own consumers and producers to synchronize the Connect cluster, the producers and consumers related to your connectors, Debezium at least in some cases uses some additional connections for its own tasks it does directly etc. Each consumer / producer also has the metadata connection to get the metadata which you don't fully control. So there are many connections and your log snippet does not make it really clear which of those are actually affected. Your fake events will cover only a part of them.

[bahaeddinoz]

Using heartbeat.interval.ms and heartbeat.action.query which it sends message to the related topic every 1 min but yeah you are right. thank you @scholzj 🙏