Kafka admin disconnections from cluster after moving to SSL

[Yedidyah27]

Hey,
Recently we have upgraded our strimzi kafka cluster to version 0.27.1 (kafka version 2.8). During this upgrade we changed the communication of our admin client to the cluster to SSL. Since this change we have been seeing constant disconnections between the client and the cluster.
Our admin client communicates with kafka bootstrap in order to discover the kafka brokers.
We use the admin client to create and delete topics. We are seeing the following exception multiple times:
Confluent.Kafka.KafkaException: Failed while waiting for response from broker: Local: Broker transport failure at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Gigya.DispatcherService.Core.KafkaConnect.KafkaAdminClientRepository.<>c__DisplayClass8_0.<<CreateTopics>b__0>d.MoveNext()

Based on kafka admin documentation the client recovers on its own and manages to reconnect to the brokers.
We added verbose logging to our client and we see the following logs:
6|1646050429.311|FAIL|rdkafka#producer-1| [thrd:ssl://stage-kafka1.net:9095/bootstrap]: ssl://stage-kafka1.net:9095/bootstrap: Disconnected (after 299978ms in state UP, 1 identical error(s) suppressed) %3|1646050429.311|ERROR|rdkafka#producer-1| [thrd:app]: rdkafka#producer-1: ssl://stage-kafka1.net:9095/bootstrap: Disconnected (after 299978ms in state UP, 1 identical error(s) suppressed) %6|1646050729.317|FAIL|rdkafka#producer-1| [thrd:ssl://50.22.32.38:9095/3]: ssl://50.22.32.38:9095/3: Disconnected (after 294335ms in state UP) %3|1646050729.317|ERROR|rdkafka#producer-1| [thrd:app]: rdkafka#producer-1: ssl://50.22.32.38:9095/3: Disconnected (after 294335ms in state UP) %6|1646051029.321|FAIL|rdkafka#producer-1| [thrd:ssl://50.22.32.38:9095/3]: ssl://50.22.32.38:9095/3: Disconnected (after 299974ms in state UP, 1 identical error(s) suppressed) %3|1646051029.321|ERROR|rdkafka#producer-1| [thrd:app]: rdkafka#producer-1: ssl://50.22.32.38:9095/3: Disconnected (after 299974ms in state UP, 1 identical error(s) suppressed) %6|1646051329.327|FAIL|rdkafka#producer-1| [thrd:ssl://50.22.32.38:9095/3]: ssl://50.22.32.38:9095/3: Disconnected (after 299972ms in state UP, 1 identical error(s) suppressed)

After this, usually the client recovers but it causes inconsistency in our application.
I was wondering of anyone has encountered something similar and managed to solve this?

[scholzj]

Did you upgraded Kafka as well? I have no experience with librdkafka or whatever client it is you are using. But it looks like some incompatibility between the client and broker? It seems to always timeout after 5 minutes.

[BenB196]

@Yedidyah27 were you ever able to solve this issue? We're running into a similar one when switching from an non-SSL cluster to an SSL cluster.

[Yedidyah27]

We upgraded to the latest kafka client and the problem seemed to be resolved. From the documentation, it seemed to be a bug on the client side.
https://github.com/confluentinc/confluent-kafka-dotnet/releases

[adarsh-ns]

@Yedidyah27 Do not see anything obvious on the release notes. Which version and which PR fixed this issue?

I still see this issue on v1.9.2, although on the python client.

[Yedidyah27]

Im guessing that the AdminClient poll loop fix, that appears in the release notes, resolved this issue.