[Bug]: ERROR IO error forwarding REST request: (org.apache.kafka.connect.runtime.rest.RestClient) [qtp2025444884-163671] #9900

areebemumba · 2024-04-01T08:11:29Z

areebemumba
Apr 1, 2024

Bug Description

I deployed Kafka MirrorMaker2 on Kubernetes. It's not able to mirror messages from the src Kafka to the target Kafka. These are the error logs from the pod: my-mirror-maker-2-mirrormaker2-0:

024-04-01 07:55:15,934 ERROR IO error forwarding REST request: (org.apache.kafka.connect.runtime.rest.RestClient) [qtp2025444884-163671]
java.util.concurrent.ExecutionException: java.net.UnknownHostException: my-mirror-maker-2-mirrormaker2-0.my-mirror-maker-2-mirrormaker2.strimzi-kafka.svc
at org.eclipse.jetty.client.util.FutureResponseListener.getResult(FutureResponseListener.java:118)
at org.eclipse.jetty.client.util.FutureResponseListener.get(FutureResponseListener.java:101)
at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:732)
at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:154)
at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:116)
at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:80)
at org.apache.kafka.connect.runtime.rest.HerderRequestHandler.completeOrForwardRequest(HerderRequestHandler.java:131)
at org.apache.kafka.connect.runtime.rest.HerderRequestHandler.completeOrForwardRequest(HerderRequestHandler.java:148)
at org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.putConnectorConfig(ConnectorsResource.java:232)
at jdk.internal.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:81)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.net.UnknownHostException: my-mirror-maker-2-mirrormaker2-0.my-mirror-maker-2-mirrormaker2.strimzi-kafka.svc
at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:801)
at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
at org.eclipse.jetty.util.SocketAddressResolver$Async.lambda$resolve$1(SocketAddressResolver.java:170)
... 3 more

k get svc -n kafka

my-mirror-maker-2-mirrormaker2 ClusterIP None 8083/TCP 58m
my-mirror-maker-2-mirrormaker2-api ClusterIP xxx.xx.xxx.xxx 8083/TCP 58m

k get pods -n kafka

my-mirror-maker-2-mirrormaker2-0 1/1 Running 0 50m
strimzi-cluster-operator-79fb75cb8d-zglkp 1/1 Running 0 56m

Steps to reproduce

Deploy Strimzi operator
kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka
Deploy KafkaMirror Maker 2 with the below mentioned manifest.

Expected behavior

Messages should be replicated from src to target kafka topic.

Strimzi version

quay.io/strimzi/operator:0.40.0

Kubernetes version

1.27

Installation method

Helm Chart

Infrastructure

Amazon EKS

Configuration files and logs

Deploy Kafka MirrorMaker2:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: my-mirror-maker-2
  namespace: kafka
spec:
  version: 3.7.0
  replicas: 1
  connectCluster: "xiq-kafka"
  clusters:
  - alias: "msk-kafka" # Source cluster
    bootstrapServers: "b-1.abcmskcluster.1uc516.k6.kafka.eu-central-1.amazonaws.com:9092,b-1.abcmskcluster.1uc516.k6.kafka.eu-central-1.amazonaws.com:9092"
  - alias: "xiq-kafka" # Target cluster
    bootstrapServers: "mtxb9g8-okl-kafka.xy.abc.com:30092,mtxb9g8-okl-kafka.xy.abc.com:30094:30093,mtxb9g8-okl-kafka.xy.abc.com:30094"
    config:
      # -1 means it will use the default replication factor configured in the broker
      config.storage.replication.factor: -1
      offset.storage.replication.factor: -1
      status.storage.replication.factor: -1
  mirrors:
  - sourceCluster: "msk-kafka"
    targetCluster: "xiq-kafka"
    sourceConnector:
      tasksMax: 1
      config:
        # -1 means it will use the default replication factor configured in the broker
        replication.factor: -1
        offset-syncs.topic.replication.factor: -1
        sync.topic.acls.enabled: "false"
        replication.policy.class: "org.apache.kafka.connect.mirror.IdentityReplicationPolicy"
        refresh.topics.interval.seconds: 600
    checkpointConnector:
      tasksMax: 1
      config:
        # -1 means it will use the default replication factor configured in the broker
        checkpoints.topic.replication.factor: -1
        sync.group.offsets.enabled: "false"
        replication.policy.class: "org.apache.kafka.connect.mirror.IdentityReplicationPolicy"
        refresh.groups.interval.seconds: 600
    topicsPattern: "events-topic"
    groupsPattern: "dev-replicator"

Additional context

No response

scholzj · 2024-04-01T09:57:28Z

scholzj
Apr 1, 2024
Maintainer

Please share the full logs. You should probably also share what you have in the strimzi-kafka namespase.

6 replies

areebemumba Apr 1, 2024
Author

Sorry @scholzj, it was a typo I have made the configuration valid.

Full Error log: https://gist.github.com/areebemumba/d71d6cc345ea8185ff48e07b0bcbe239

Also I don't have a strimzi-kafka namespace.
k get all -n strimzi-kafka

No resources found in strimzi-kafka namespace.

scholzj Apr 1, 2024
Maintainer

That is not a full log as it is not from the start. Also, the log suggests there is such namespace somewhere:

java.util.concurrent.ExecutionException: java.net.UnknownHostException: my-mirror-maker-2-mirrormaker2-0.my-mirror-maker-2-mirrormaker2.strimzi-kafka.svc

Maybe it is in a different cluster? You should make sure you follow this: https://strimzi.io/docs/operators/latest/full/deploying.html#con-config-mm2-multiple-instances-str

areebemumba Apr 1, 2024
Author

Full Logs: https://gist.github.com/areebemumba/136588020452f5993b07abb90938c1b6

Yes it's weird, the namespace doesn't exist also I am using the same cluster:

I have followed the example form official strimzi repo for deployment: https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/mirror-maker/kafka-mirror-maker-2.yaml

scholzj Apr 1, 2024
Maintainer

Well, as I said, the namespace might be in a different Kubernetes cluster as well.

scholzj Apr 1, 2024
Maintainer

The log seems to match that assumption -> you have somewhere else running another MM2 with the same config. You have to follow the docs I linked to make them independent.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

[Bug]: ERROR IO error forwarding REST request: (org.apache.kafka.connect.runtime.rest.RestClient) [qtp2025444884-163671] #9900

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment 6 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Strimzi

[Bug]: ERROR IO error forwarding REST request: (org.apache.kafka.connect.runtime.rest.RestClient) [qtp2025444884-163671] #9900

Uh oh!

Uh oh!

areebemumba Apr 1, 2024

Bug Description

Steps to reproduce

Expected behavior

Strimzi version

Kubernetes version

Installation method

Infrastructure

Configuration files and logs

Additional context

Replies: 1 comment · 6 replies

Uh oh!

scholzj Apr 1, 2024 Maintainer

Uh oh!

Uh oh!

areebemumba Apr 1, 2024 Author

Uh oh!

scholzj Apr 1, 2024 Maintainer

Uh oh!

areebemumba Apr 1, 2024 Author

Uh oh!

scholzj Apr 1, 2024 Maintainer

Uh oh!

scholzj Apr 1, 2024 Maintainer

areebemumba
Apr 1, 2024

Replies: 1 comment 6 replies

scholzj
Apr 1, 2024
Maintainer

areebemumba Apr 1, 2024
Author

scholzj Apr 1, 2024
Maintainer

areebemumba Apr 1, 2024
Author

scholzj Apr 1, 2024
Maintainer

scholzj Apr 1, 2024
Maintainer