Can't connect to Kafka Connect for PUT request, but can to GET #10361

vladislav-bakanov · 2024-07-19T10:02:35Z

vladislav-bakanov
Jul 19, 2024

Hi!

I deployed Strimzi Kafka Connect but met weird behaviour:

$ curl -X GET http://dwh-kafka-connect-connect-api.dwh.svc:8083/connectors
["uat-dwh-clickhouse-sink-mt4-live-loginfo"]

$ curl -X GET http://dwh-kafka-connect-connect-api.dwh.svc:8083/connectors/uat-dwh-clickhouse-sink-mt4-live-loginfo/status
{"name":"uat-dwh-clickhouse-sink-mt4-live-loginfo","connector":{"state":"RUNNING","worker_id":"10.129.6.54:8083"},"tasks":[{"id":0,"state":"RUNNING","worker_id":"10.129.6.54:8083"}],"type":"sink"}

$ curl -X PUT http://dwh-kafka-connect-connect-api.dwh.svc:8083/connectors/uat-dwh-clickhouse-sink-mt4-live-loginfo/stop
{"error_code":500,"message":"IO Error trying to forward REST request: java.net.SocketTimeoutException: Connect Timeout"}

$ curl -X DELETE http://dwh-kafka-connect-connect-api.dwh.svc:8083/connectors/uat-dwh-clickhouse-sink-mt4-live-loginfo
{"error_code":500,"message":"IO Error trying to forward REST request: java.net.SocketTimeoutException: Connect Timeout"}

I use Strimzi cluster operator 0.36.0

I use following Docker image:

 Check following issue to get to know why we use this approach:
# https://github.com/strimzi/strimzi-kafka-operator/issues/3047

FROM confluentinc/cp-kafka-connect-base:7.6.1 as cp

# Free connectors from Confluent Hub
RUN confluent-hub install --no-prompt clickhouse/clickhouse-kafka-connect:v1.0.17
RUN confluent-hub install --no-prompt wepay/kafka-connect-bigquery:2.5.6
RUN confluent-hub install --no-prompt confluentinc/kafka-connect-avro-converter:7.5.4
RUN confluent-hub install --no-prompt debezium/debezium-connector-mysql:2.4.2
RUN confluent-hub install --no-prompt debezium/debezium-connector-postgresql:2.5.4


FROM quay.io/strimzi/kafka:0.36.0-kafka-3.5.0
USER root:root

ARG CONFLUENT_IMAGE_PLUGINS_DIRECTORY=/usr/share/confluent-hub-components/


# -------------------------------------------------------------------------------------------------------------- #
#  Connectors installation
# -------------------------------------------------------------------------------------------------------------- #
# Avro Converter
ARG AVRO_PLUGIN_PATH=/opt/kafka/plugins/avro/
RUN mkdir -p mkdir -p $AVRO_PLUGIN_PATH
COPY --from=cp $CONFLUENT_IMAGE_PLUGINS_DIRECTORY/confluentinc-kafka-connect-avro-converter/lib $AVRO_PLUGIN_PATH

# Clickhouse Plugin
ARG CLICKHOUSE_PLUGIN_PATH=/opt/kafka/plugins/clickhouse/
RUN mkdir -p mkdir -p $CLICKHOUSE_PLUGIN_PATH
COPY --from=cp $CONFLUENT_IMAGE_PLUGINS_DIRECTORY/clickhouse-clickhouse-kafka-connect/lib $CLICKHOUSE_PLUGIN_PATH

# BigQuery Plugin
ARG BIGQUERY_PLUGIN_PATH=/opt/kafka/plugins/bigquery/
RUN mkdir -p mkdir -p $BIGQUERY_PLUGIN_PATH
COPY --from=cp $CONFLUENT_IMAGE_PLUGINS_DIRECTORY/wepay-kafka-connect-bigquery/lib $BIGQUERY_PLUGIN_PATH

# Debezium MySQL Plugin
ARG DEBEZIUM_MYSQL_PLUGIN_PATH=/opt/kafka/plugins/debezium-mysql/
RUN mkdir -p mkdir -p $DEBEZIUM_MYSQL_PLUGIN_PATH
COPY --from=cp $CONFLUENT_IMAGE_PLUGINS_DIRECTORY/debezium-debezium-connector-mysql/lib $DEBEZIUM_MYSQL_PLUGIN_PATH

# Debezium PostgreSQL Plugin
ARG DEBEZIUM_POSTGRESQL_PLUGIN_PATH=/opt/kafka/plugins/debezium-postgresql/
RUN mkdir -p mkdir -p $DEBEZIUM_POSTGRESQL_PLUGIN_PATH
COPY --from=cp $CONFLUENT_IMAGE_PLUGINS_DIRECTORY/debezium-debezium-connector-postgresql/lib $DEBEZIUM_POSTGRESQL_PLUGIN_PATH
# -------------------------------------------------------------------------------------------------------------- #


# ---------------------------------------------- #
# Fix security issues
# ---------------------------------------------- #
RUN microdnf install -y yum
RUN yum update -y
RUN yum install -y glibc-2.28
RUN yum install -y glibc-common-2.28
RUN yum install -y glibc-minimal-langpack-2.28
RUN yum install -y python3
RUN yum clean all
# ---------------------------------------------- #

USER 1001

Strimzi Cluster Operator gives me following trace:

io.strimzi.operator.cluster.operator.assembly.ConnectRestException: DELETE /connectors/uat-dwh-clickhouse-sink-mt4-live-loginfo returned 500 (Internal Server Error): IO Error trying to forward REST request: java.net.SocketTimeoutException: Connect Timeout
	at io.strimzi.operator.cluster.operator.assembly.KafkaConnectApiImpl.lambda$delete$13(KafkaConnectApiImpl.java:194) ~[io.strimzi.cluster-operator-0.36.0.jar:0.36.0]
	at io.vertx.core.impl.future.FutureImpl$1.onSuccess(FutureImpl.java:91) ~[io.vertx.vertx-core-4.4.4.jar:4.4.4]
	at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:54) ~[io.vertx.vertx-core-4.4.4.jar:4.4.4]
	at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:86) ~[io.vertx.vertx-core-4.4.4.jar:4.4.4]
	at io.vertx.core.impl.ContextBase.execute(ContextBase.java:225) ~[io.vertx.vertx-core-4.4.4.jar:4.4.4]
	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:51) ~[io.vertx.vertx-core-4.4.4.jar:4.4.4]
	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211) ~[io.vertx.vertx-core-4.4.4.jar:4.4.4]
	at io.vertx.core.impl.future.PromiseImpl.tryComplete(PromiseImpl.java:23) ~[io.vertx.vertx-core-4.4.4.jar:4.4.4]
	at io.vertx.core.http.impl.HttpEventHandler.handleEnd(HttpEventHandler.java:79) ~[io.vertx.vertx-core-4.4.4.jar:4.4.4]
	at io.vertx.core.http.impl.HttpClientResponseImpl.handleEnd(HttpClientResponseImpl.java:250) ~[io.vertx.vertx-core-4.4.4.jar:4.4.4]
	at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$new$0(Http1xClientConnection.java:444) ~[io.vertx.vertx-core-4.4.4.jar:4.4.4]
	at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:255) ~[io.vertx.vertx-core-4.4.4.jar:4.4.4]
	at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:134) ~[io.vertx.vertx-core-4.4.4.jar:4.4.4]
	at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleEnd(Http1xClientConnection.java:708) ~[io.vertx.vertx-core-4.4.4.jar:4.4.4]
	at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:76) ~[io.vertx.vertx-core-4.4.4.jar:4.4.4]
	at io.vertx.core.impl.ContextBase.execute(ContextBase.java:232) ~[io.vertx.vertx-core-4.4.4.jar:4.4.4]
	at io.vertx.core.http.impl.Http1xClientConnection.handleResponseEnd(Http1xClientConnection.java:945) ~[io.vertx.vertx-core-4.4.4.jar:4.4.4]
	at io.vertx.core.http.impl.Http1xClientConnection.handleHttpMessage(Http1xClientConnection.java:814) ~[io.vertx.vertx-core-4.4.4.jar:4.4.4]
	at io.vertx.core.http.impl.Http1xClientConnection.handleMessage(Http1xClientConnection.java:778) ~[io.vertx.vertx-core-4.4.4.jar:4.4.4]
	at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:158) ~[io.vertx.vertx-core-4.4.4.jar:4.4.4]
	at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:153) ~[io.vertx.vertx-core-4.4.4.jar:4.4.4]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty.netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty.netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[io.netty.netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[io.netty.netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[io.netty.netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[io.netty.netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty.netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty.netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[io.netty.netty-handler-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty.netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty.netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty.netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[io.netty.netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty.netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[io.netty.netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[io.netty.netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[io.netty.netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[io.netty.netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[io.netty.netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty.netty-common-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.94.Final.jar:4.1.94.Final]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]

The weirdest thing that is yesterday this deployment worked well, but I renamed my connector and it started to fail.

Can you suggest something?

Answered by scholzj

Jul 19, 2024

That typically suggests some misconfiguration or networking issue. If you have multiple Connect clusters using the same Kafka cluster you should make sure they are configured independently: https://strimzi.io/docs/operators/latest/full/deploying.html#con-config-kafka-connect-multiple-instances-str But if you have multiple replcias, this can be also caused by some networking issue etc.

View full answer

scholzj · 2024-07-19T10:23:29Z

scholzj
Jul 19, 2024
Maintainer

That typically suggests some misconfiguration or networking issue. If you have multiple Connect clusters using the same Kafka cluster you should make sure they are configured independently: https://strimzi.io/docs/operators/latest/full/deploying.html#con-config-kafka-connect-multiple-instances-str But if you have multiple replcias, this can be also caused by some networking issue etc.

0 replies

vladislav-bakanov · 2024-07-19T10:47:18Z

vladislav-bakanov
Jul 19, 2024
Author

@scholzj, thanks!

Actually problem was that Kafka Connect resource was deployed twice with the same configuration but to different namespaces and it was conflicting with the deployment that I expected.
Logs from the Cluster Operator helped me to find, that it check connector in different namespace that I excpect.

Thanks for help ❤️

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

Can't connect to Kafka Connect for PUT request, but can to GET #10361

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Strimzi

Can't connect to Kafka Connect for PUT request, but can to GET #10361

Uh oh!

Uh oh!

vladislav-bakanov Jul 19, 2024

Strimzi Cluster Operator gives me following trace:

Replies: 2 comments

Uh oh!

scholzj Jul 19, 2024 Maintainer

Uh oh!

vladislav-bakanov Jul 19, 2024 Author

vladislav-bakanov
Jul 19, 2024

scholzj
Jul 19, 2024
Maintainer

vladislav-bakanov
Jul 19, 2024
Author