Uncaught exception in REST call to /topics endpoint in KafkaConnect

[yonieilon]

Hi,

I've deployed the strimzi-kafka-operator on our k8s cluster using a Docker image built using this Dockerfile:

FROM strimzi/kafka:0.16.1-kafka-2.4.0
USER root:root
RUN mkdir -p /opt/kafka/plugins/debezium
COPY ./debezium-connector-postgres/ /opt/kafka/plugins/debezium/
USER 1001

Then I have the Operator deployed with all the CRDs, and I defined 2 resources - KafkaConnect & KafkaConnector:

---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: debezium-connect-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  replicas: 1
  image: <OUR IMAGE>
  authentication:
    type: plain
    username: <USER>
    passwordSecret:
      secretName: debezium-kafka-secret
      password: kafka-password
  bootstrapServers: <OUR KAFKA BROKER>
  tls:
    trustedCertificates: []
  config:
    group.id: debezium-connect-cluster
    config.providers: file
    config.providers.file.class: org.apache.kafka.common.config.provider.FileConfigProvider
  externalConfiguration:
    volumes:
    - name: connector-config
      secret:
        secretName: <SECRET_NAME>

---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: inventory-connector
  labels:
    strimzi.io/cluster: debezium-connect-cluster
spec:
  class: io.debezium.connector.postgresql.PostgresConnector
  tasksMax: 1
  config:
    plugin.name: pgoutput
    database.hostname: <DB_HOSTNAME>
    database.port: 5432
    database.user: ${file:/opt/kafka/external-configuration/connector-config/debezium-postgresql-credentials.properties:user}
    database.password: ${file:/opt/kafka/external-configuration/connector-config/debezium-postgresql-credentials.properties:password}
    database.dbname: <DB_NAME>
    database.server.name: <SERVER_NAME>
    table.include.list: <TABLE_NAME>
    heartbeat.interval.ms: 5000
    slot.name: <SLOT_NAME>
    publication.name: <PUBLICATION_NAME>

the deployment is successful and Debezium is able to capture INSERT\UPDATE\DELETE on the DB table and send event to our Kafka cluster.

However, we keep getting ERROR in the KafkaConnect logs:

2021-11-16 10:38:12,292 ERROR Uncaught exception in REST call to /connectors/inventory-connector/topics (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper) [qtp608817143-22]
javax.ws.rs.NotFoundException: HTTP 404 Not Found
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:250)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:679)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:392)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:365)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:318)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:852)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:544)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1581)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1307)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:482)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1549)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1204)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
	at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:494)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:374)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:268)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:367)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
	at java.lang.Thread.run(Thread.java:748)

Where inventory-connector mentioned in the error is the name used for the Postgresql KafkaConnector in the config above.

It seems that this endpoint isn't part of the KafkaConnect REST API (https://kafka.apache.org/documentation/#connect_rest), but rather part of the Confluent REST proxy (https://docs.confluent.io/platform/current/kafka-rest/api.html#topics)

So my question is: why is the KafkaConnect trying to reach the /topics endpoint for this Connector?
And if it is needed, how do we enable Confluent's REST Proxy, to make this endpoint available?

Any help would be appreciated...

[scholzj]

The endpoint is part of Kafka Connect. You can see it in the link you shared:

GET /connectors/{name}/topics - get the set of topics that a specific connector is using since the connector was created or since a request to reset its set of active topics was issued

What version of operator are you using? You need to use the same version of the operator also for the base image. The error like this is usually caused by using old version of Kafka Connect. You Docker file uses Kafka 2.4 / Strimzi 0.16. So, are you sure your operator uses the same version (0.16)?

[scholzj]

In the operator logs, the operator prints it when it is starting. Normally, you can also see if from the tag of the operator container image.

[yonieilon]

@scholzj Found it, thanks...

2021-11-15 22:08:09 INFO Main:60 - ClusterOperator 0.26.0 is starting

So seems like I need to upgrade my Docker image to use Strimzi version 0.26.0...

I'll do that and update if it solved the issue.

Thanks again for your help.

[yonieilon]

Hi @scholzj,

Tried to look for version 0.26.0 of the base image, but couldn't find it here:
https://hub.docker.com/r/strimzi/kafka/tags

So took the tag latest-kafka-2.4.0, since our Kafka is v2.4.0

Built the new image and deployed it, but still getting the same error:

Uncaught exception in REST call to /connectors/inventory-connector/topics

Is this docker image also running a version that's too old?

Looks like the /topics endpoint was introduced in v5.5.0 of KafkaConnect:
https://docs.confluent.io/5.5.0/connect/references/restapi.html

How can I know which Connect version the Strimzi docker image is running?

Thanks,
Yoni.

[scholzj]

Tried to look for version 0.26.0 of the base image, but couldn't find it here:
https://hub.docker.com/r/strimzi/kafka/tags

We do not use Docker Hub for quite some time because of all the different limits they introduced. The up-to-date images can be found on Quay.io: https://quay.io/organization/strimzi ... the right image is also used in the docs: https://strimzi.io/docs/operators/latest/full/deploying.html#creating-new-image-from-base-str

So took the tag latest-kafka-2.4.0, since our Kafka is v2.4.0

Each Strimzi version supports only some Kafka versions. You can find the list of which Strimzi versions supports which Kafka versions on the website: https://strimzi.io/downloads/ ... 0.26 supports Kafka 2.8.0, 2.8.1 and 3.0.0. So you need to use older version of the operator or newer version of Kafka.

Looks like the /topics endpoint was introduced in v5.5.0 of KafkaConnect:
https://docs.confluent.io/5.5.0/connect/references/restapi.html

Strimzi is based on the Apache Kafka project. It does not use Confluent Platform.

[yonieilon]

Hi @scholzj Sorry the delayed response...

As you suggested, was able to solve this issue by using the same version for both the Docker image and the k8s operator, in our case v0.19.0.

Once we did this, error stopped and everything is working well...

Thanks, marking your initial reply as the answer.