[Bug] Amazon MSK multi-VPC private connectivity(Privatelink) KafkaMirrorMaker2 error

[moanass]

Bug Description

When I write a privatelink endpoint in bootstrapServers in my KafkaMirrorMaker2 yaml file, it doesn't connect from the mirrormaker2 pod.
It works fine as a regular msk endpoint.

I want to run mirrormaker with my msk privatelink endpoint

================================================
KafkaMirrorMaker2 pod error log

2024-01-26 03:49:47,253 INFO [AdminClient clientId=adminclient-1] Cancelled in-flight API_VERSIONS request with correlation id 110 due to node -1 being disconnected (elapsed time since creation: 40ms, elapsed time since send: 40ms, request timeout: 3600000ms) (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]
2024-01-26 03:49:47,696 INFO [AdminClient clientId=adminclient-1] Node -2 disconnected. (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]
2024-01-26 03:49:47,696 INFO [AdminClient clientId=adminclient-1] Cancelled in-flight API_VERSIONS request with correlation id 111 due to node -2 being disconnected (elapsed time since creation: 40ms, elapsed time since send: 40ms, request timeout: 3600000ms) (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]
2024-01-26 03:49:48,189 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata
2024-01-26 03:49:48,189 INFO App info kafka.admin.client for adminclient-1 unregistered (org.apache.kafka.common.utils.AppInfoParser) [kafka-admin-client-thread | adminclient-1]
2024-01-26 03:49:48,189 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata
2024-01-26 03:49:48,189 INFO [AdminClient clientId=adminclient-1] Timed out 1 remaining operation(s) during close. (org.apache.kafka.clients.admin.KafkaAdminClient) [kafka-admin-client-thread | adminclient-1]
2024-01-26 03:49:48,193 INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2024-01-26 03:49:48,193 INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2024-01-26 03:49:48,193 INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2024-01-26 03:49:48,209 ERROR Stopping due to error (org.apache.kafka.connect.cli.AbstractConnectCli) [main]
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
	at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:305)
	at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:285)
	at org.apache.kafka.connect.runtime.WorkerConfig.kafkaClusterId(WorkerConfig.java:415)
	at org.apache.kafka.connect.cli.AbstractConnectCli.startConnect(AbstractConnectCli.java:124)

my KafkaMirrorMaker2 yaml file

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: my-mirror-maker2
spec:
  version: 3.6.1
  connectCluster: "my-cluster-target"
  clusters:
  - alias: "my-cluster-source"
    bootstrapServers: msk-endpoint:9092
  - alias: "my-cluster-target"
    bootstrapServers: msk-privatelink-endpoint:14001
    config:
      config.storage.replication.factor: 2
      offset.storage.replication.factor: 2
      status.storage.replication.factor: 2 
  mirrors:
  - sourceCluster: "my-cluster-source"
    targetCluster: "my-cluster-target"
    topicsPattern: "test"
    sourceConnector: {}

Steps to reproduce

No response

Expected behavior

No response

Strimzi version

3.6.1

Kubernetes version

1.28

Installation method

No response

Infrastructure

No response

Configuration files and logs

No response

Additional context

No response

[scholzj]

Strimzi just deploys Mirror Maker 2 -> it is up to you to configure the Kafka cluster and configure how to connect to it. Without more details and better understanding of how your infrastructure looks like, it is hard to guess what the issue is. I guess the two obvious things to look into are:

• Are you really using no security?
• That the cluster is properly exposed over the Privatelink (I'm not a MSK user, so no idea if this is part of the MSK service or if this is something you setup your self). You have to make sure that:
  • Every broker is exposed
  • The adveertised hosts / ports for every broker are configured to the address of the private-link endpoint