Producer error

[barryzhounb]

I am using Srimizi Kafka on OpenShift platform.

I saw the warning from my producer, and producer could not public any messages.

2021-07-19 14:44:27,207 [kafka-producer-network-thread | dlcclient1] org.apache.kafka.clients.NetworkClient: [WARN] [Producer clientId=dlcclient1] Error connecting to node xtest-cluster-kafka-1-atest.apps.irondev-1.cp.fyre.ibm.com:443 (id: 1 rack: null)
java.net.UnknownHostException: xtest-cluster-kafka-1-atest.apps.irondev-1.cp.fyre.ibm.com: xtest-cluster-kafka-1-atest.apps.irondev-1.cp.fyre.ibm.com: unknown error
        at java.net.InetAddress.getAllByName0(InetAddress.java:1414) ~[?:1.8.0]
        at java.net.InetAddress.getAllByName(InetAddress.java:1323) ~[?:1.8.0]
        at java.net.InetAddress.getAllByName(InetAddress.java:1246) ~[?:1.8.0]
        at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110) ~[kafka-clients-2.5.0.jar:?]
        at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403) ~[kafka-clients-2.5.0.jar:?]
        at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363) ~[kafka-clients-2.5.0.jar:?]
        at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151) ~[kafka-clients-2.5.0.jar:?]
        at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:955) [kafka-clients-2.5.0.jar:?]
        at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:73) [kafka-clients-2.5.0.jar:?]
        at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1128) [kafka-clients-2.5.0.jar:?]
        at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1016) [kafka-clients-2.5.0.jar:?]
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:547) [kafka-clients-2.5.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:324) [kafka-clients-2.5.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239) [kafka-clients-2.5.0.jar:?]
        at java.lang.Thread.run(Thread.java:812) [?:2.9 (12-19-2018)]

Then I looked into kafka-broker-0 Pod and view its logs, I saw information like the following. I am wondering what happens in Kafka broker side and why does it cause producer not work? And how to fix or avoid it under this situation?

2021-07-19 18:29:57,664 DEBUG [Controller id=0] Topics not in preferred replica for broker 1 Map() (kafka.controller.KafkaController) [controller-event-thread]
2021-07-19 18:29:57,664 TRACE [Controller id=0] Leader imbalance ratio for broker 1 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]
2021-07-19 18:34:57,664 INFO [Controller id=0] Processing automatic preferred replica leader election (kafka.controller.KafkaController) [controller-event-thread]
2021-07-19 18:34:57,664 TRACE [Controller id=0] Checking need to trigger auto leader balancing (kafka.controller.KafkaController) [controller-event-thread]
2021-07-19 18:34:57,665 DEBUG [Controller id=0] Topics not in preferred replica for broker 0 Map() (kafka.controller.KafkaController) [controller-event-thread]
2021-07-19 18:34:57,665 TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]
2021-07-19 18:34:57,665 DEBUG [Controller id=0] Topics not in preferred replica for broker 2 Map() (kafka.controller.KafkaController) [controller-event-thread]
2021-07-19 18:34:57,665 TRACE [Controller id=0] Leader imbalance ratio for broker 2 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]
2021-07-19 18:34:57,665 DEBUG [Controller id=0] Topics not in preferred replica for broker 1 Map() (kafka.controller.KafkaController) [controller-event-thread]
2021-07-19 18:34:57,665 TRACE [Controller id=0] Leader imbalance ratio for broker 1 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]
2021-07-19 18:39:57,665 INFO [Controller id=0] Processing automatic preferred replica leader election (kafka.controller.KafkaController) [controller-event-thread]
2021-07-19 18:39:57,666 TRACE [Controller id=0] Checking need to trigger auto leader balancing (kafka.controller.KafkaController) [controller-event-thread]
2021-07-19 18:39:57,666 DEBUG [Controller id=0] Topics not in preferred replica for broker 0 Map() (kafka.controller.KafkaController) [controller-event-thread]
2021-07-19 18:39:57,666 TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]
2021-07-19 18:39:57,666 DEBUG [Controller id=0] Topics not in preferred replica for broker 2 Map() (kafka.controller.KafkaController) [controller-event-thread]
2021-07-19 18:39:57,666 TRACE [Controller id=0] Leader imbalance ratio for broker 2 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]
2021-07-19 18:39:57,666 DEBUG [Controller id=0] Topics not in preferred replica for broker 1 Map() (kafka.controller.KafkaController) [controller-event-thread]
2021-07-19 18:39:57,666 TRACE [Controller id=0] Leader imbalance ratio for broker 1 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]

[scholzj]

The broker messages are IMHO just regular messages which show up regularly in the broker. From the client issue, it looks like it cannot resolve the broker address. You should double check if it is correct and check the DNS etc.

[barryzhounb]

@hi @scholzj , thanks your reply. But I still hit this issue.

I added bootstrap server to /etc/hosts

9.35.14.205     xtest-cluster-kafka-bootstrap-atest.apps.irondev-1.cp.fyre.ibm.com

But I still see the following warns, seem it could not recognize sever. Is the above way correct to add bootstarp server to /etc/hosts? Or is there something missing?

2021-07-20 08:41:47,178 [KafkaMessageReceiver] org.apache.kafka.clients.NetworkClient: [WARN] [Consumer clientId=dlcConsumer1, groupId=xtest-group-management-cp4s-to-dlc] Error connecting to node xtest-cluster-kafka-0-atest.apps.irondev-1.cp.fyre.ibm.com:443 (id: 0 rack: null)
java.net.UnknownHostException: xtest-cluster-kafka-0-atest.apps.irondev-1.cp.fyre.ibm.com
	at java.net.InetAddress.getAllByName0(InetAddress.java:1419) ~[?:1.8.0]
	at java.net.InetAddress.getAllByName(InetAddress.java:1323) ~[?:1.8.0]
	at java.net.InetAddress.getAllByName(InetAddress.java:1246) ~[?:1.8.0]
	at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110) ~[kafka-clients-2.5.0.jar:?]
	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403) ~[kafka-clients-2.5.0.jar:?]
	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363) ~[kafka-clients-2.5.0.jar:?]
	at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151) ~[kafka-clients-2.5.0.jar:?]
	at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:955) [kafka-clients-2.5.0.jar:?]
	at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:293) [kafka-clients-2.5.0.jar:?]
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.trySend(ConsumerNetworkClient.java:495) [kafka-clients-2.5.0.jar:?]
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:252) [kafka-clients-2.5.0.jar:?]
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233) [kafka-clients-2.5.0.jar:?]
	at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1308) [kafka-clients-2.5.0.jar:?]
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1248) [kafka-clients-2.5.0.jar:?]
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216) [kafka-clients-2.5.0.jar:?]
	at com.ibm.si.managementservice.communicator.MessageReceiverProcessor.readFromKafka(MessageReceiverProcessor.java:63) [q1labs_sem.jar:?]
	at com.ibm.si.managementservice.communicator.MessageReceiverProcessor.run(MessageReceiverProcessor.java:38) [q1labs_sem.jar:?]
	at java.lang.Thread.run(Thread.java:812) [?:2.9 (12-19-2018)]

[scholzj]

Well, you set it for the bootstrap address. But you will need to configure the hostname for all broker addresses as well. The error complains about xtest-cluster-kafka-0-atest.apps.irondev-1.cp.fyre.ibm.com ... so I guess xtest-cluster-kafka-bootstrap-atest.apps.irondev-1.cp.fyre.ibm.com worked, you just need to add the brokers as well.