Kafka mirror maker strimzi to docker #6957

izalul · 2022-06-18T10:25:14Z

izalul
Jun 18, 2022

is it possible to use kafka mirror maker between strimzi to kafka docker?

scholzj · 2022-06-18T10:55:50Z

scholzj
Jun 18, 2022
Maintainer

You mean ... to have a Strimzi operated Kafka cluster and Mirror Maker and use it to mirror messages to some other cluster running somewhere else with Docker containers?

6 replies

scholzj Jun 18, 2022
Maintainer

That should be possible in general. Of course:

Your Kafka cluster needs to be accessible from the Kubernetes cluster where you run the Mirror Maker
It needs to use supported authentication mechanism.

But apart form that, you can use it with any Kafka cluster you want.

izalul Jun 18, 2022
Author

I've tried strimzi mm2 but it doesn't mirror the data, mirror maker only creates some topics on my kafka docker.

Strimzi MM2 Manifest:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: mirrormaker2-k8s-docker
  namespace: kafka
spec:
  version: 3.0.0
  replicas: 1
  connectCluster: "kafka-docker"
  clusters:
  - alias: "kafka-strimzi"
    bootstrapServers: my-cluster-kafka-bootstrap:9092
  - alias: "kafka-docker"
    bootstrapServers: kafka-docker:9092
    config:
      config.storage.replication.factor: 1
      offset.storage.replication.factor: 1
      status.storage.replication.factor: 1
      security.protocol: PLAIN
  mirrors:
  - sourceCluster: "kafka-strimzi"
    targetCluster: "kafka-docker"
    sourceConnector:
      config:
        replication.factor: 1
        offset-syncs.topic.replication.factor: 1
        sync.topic.acls.enabled: "false"
    heartbeatConnector:
      config:
        heartbeats.topic.replication.factor: 1
    checkpointConnector:
      config:
        config.storage.replication.factor: 1
        offset.storage.replication.factor: 1
        status.storage.replication.factor: 1
        sync.topic.acls.enabled: "false"
    topicsPattern: ".*"
    groupsPattern: ".*"
  logging: 
    type: inline
    loggers:
      connect.root.logger.level: "INFO"
  readinessProbe: 
    initialDelaySeconds: 25
    timeoutSeconds: 25
  livenessProbe:
    initialDelaySeconds: 25
    timeoutSeconds: 25

MirrroMaker Logs:

2022-06-18 15:20:00,312 WARN The configuration 'metrics.context.connect.kafka.cluster.id' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig) [StartAndStopExecutor-connect-1-4]
2022-06-18 15:20:00,312 WARN The configuration 'metrics.context.connect.group.id' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig) [StartAndStopExecutor-connect-1-4]
2022-06-18 15:20:00,312 INFO Kafka version: 3.0.0 (org.apache.kafka.common.utils.AppInfoParser) [StartAndStopExecutor-connect-1-4]
2022-06-18 15:20:00,312 INFO Kafka commitId: 8cb0a5e9d3441962 (org.apache.kafka.common.utils.AppInfoParser) [StartAndStopExecutor-connect-1-4]
2022-06-18 15:20:00,312 INFO Kafka startTimeMs: 1655565600312 (org.apache.kafka.common.utils.AppInfoParser) [StartAndStopExecutor-connect-1-4]
2022-06-18 15:20:00,316 INFO [Producer clientId=connector-producer-kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0] Cluster ID: GmXR-ouqT4-tiMCU4C74Ww (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | connector-producer-kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0]
2022-06-18 15:20:00,319 INFO [Worker clientId=connect-1, groupId=mirrormaker2-cluster] Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2022-06-18 15:20:00,320 INFO SourceConnectorConfig values: 
	config.action.reload = restart
	connector.class = org.apache.kafka.connect.mirror.MirrorCheckpointConnector
	errors.log.enable = false
	errors.log.include.messages = false
	errors.retry.delay.max.ms = 60000
	errors.retry.timeout = 0
	errors.tolerance = none
	header.converter = null
	key.converter = null
	name = kafka-strimzi->kafka-docker.MirrorCheckpointConnector
	predicates = []
	tasks.max = 1
	topic.creation.groups = []
	transforms = []
	value.converter = null
 (org.apache.kafka.connect.runtime.SourceConnectorConfig) [DistributedHerder-connect-1-1]
2022-06-18 15:20:00,320 INFO EnrichedConnectorConfig values: 
	config.action.reload = restart
	connector.class = org.apache.kafka.connect.mirror.MirrorCheckpointConnector
	errors.log.enable = false
	errors.log.include.messages = false
	errors.retry.delay.max.ms = 60000
	errors.retry.timeout = 0
	errors.tolerance = none
	header.converter = null
	key.converter = null
	name = kafka-strimzi->kafka-docker.MirrorCheckpointConnector
	predicates = []
	tasks.max = 1
	topic.creation.groups = []
	transforms = []
	value.converter = null
 (org.apache.kafka.connect.runtime.ConnectorConfig$EnrichedConnectorConfig) [DistributedHerder-connect-1-1]
2022-06-18 15:20:00,321 INFO MirrorTaskConfig values: 
	admin.timeout.ms = 60000
	checkpoints.topic.replication.factor = 3
	config.action.reload = restart
	config.properties.blacklist = null
	config.properties.exclude = [follower\.replication\.throttled\.replicas, leader\.replication\.throttled\.replicas, message\.timestamp\.difference\.max\.ms, message\.timestamp\.type, unclean\.leader\.election\.enable, min\.insync\.replicas]
	config.property.filter.class = class org.apache.kafka.connect.mirror.DefaultConfigPropertyFilter
	connector.class = org.apache.kafka.connect.mirror.MirrorHeartbeatConnector
	consumer.poll.timeout.ms = 1000
	emit.checkpoints.enabled = true
	emit.checkpoints.interval.seconds = 60
	emit.heartbeats.enabled = true
	emit.heartbeats.interval.seconds = 1
	enabled = true
	errors.log.enable = false
	errors.log.include.messages = false
	errors.retry.delay.max.ms = 60000
	errors.retry.timeout = 0
	errors.tolerance = none
	group.filter.class = class org.apache.kafka.connect.mirror.DefaultGroupFilter
	groups = [.*]
	groups.blacklist = null
	groups.exclude = [console-consumer-.*, connect-.*, __.*]
	header.converter = null
	heartbeats.topic.replication.factor = 1
	key.converter = null
	metric.reporters = null
	name = kafka-strimzi->kafka-docker.MirrorHeartbeatConnector
	offset-syncs.topic.location = source
	offset-syncs.topic.replication.factor = 3
	offset.lag.max = 100
	predicates = []
	refresh.groups.enabled = true
	refresh.groups.interval.seconds = 600
	refresh.topics.enabled = true
	refresh.topics.interval.seconds = 600
	replication.factor = 2
	replication.policy.class = class org.apache.kafka.connect.mirror.DefaultReplicationPolicy
	replication.policy.separator = .
	sasl.client.callback.handler.class = null
	sasl.jaas.config = null
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.min.time.before.relogin = 60000
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	sasl.kerberos.ticket.renew.window.factor = 0.8
	sasl.login.callback.handler.class = null
	sasl.login.class = null
	sasl.login.refresh.buffer.seconds = 300
	sasl.login.refresh.min.period.seconds = 60
	sasl.login.refresh.window.factor = 0.8
	sasl.login.refresh.window.jitter = 0.05
	sasl.mechanism = GSSAPI
	security.protocol = PLAINTEXT
	source.cluster.alias = kafka-strimzi
	ssl.cipher.suites = null
	ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
	ssl.endpoint.identification.algorithm = https
	ssl.engine.factory.class = null
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.certificate.chain = null
	ssl.keystore.key = null
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.protocol = TLSv1.3
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.certificates = null
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS
	sync.group.offsets.enabled = false
	sync.group.offsets.interval.seconds = 60
	sync.topic.acls.enabled = true
	sync.topic.acls.interval.seconds = 600
	sync.topic.configs.enabled = true
	sync.topic.configs.interval.seconds = 600
	target.cluster.alias = kafka-docker
	task.assigned.groups = null
	task.assigned.partitions = null
	tasks.max = 1
	topic.filter.class = class org.apache.kafka.connect.mirror.DefaultTopicFilter
	topics = [.*]
	topics.blacklist = null
	topics.exclude = [.*[\-\.]internal, .*\.replica, __.*]
	transforms = []
	value.converter = null
 (org.apache.kafka.connect.mirror.MirrorTaskConfig) [task-thread-kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0]
2022-06-18 15:20:00,321 INFO WorkerSourceTask{id=kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0} Source task finished initialization and start (org.apache.kafka.connect.runtime.WorkerSourceTask) [task-thread-kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0]
2022-06-18 15:20:01,336 WARN [Producer clientId=connector-producer-kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0] Error while fetching metadata with correlation id 3 : {heartbeats=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [kafka-producer-network-thread | connector-producer-kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0]
2022-06-18 15:21:00,320 INFO WorkerSourceTask{id=kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [SourceTaskOffsetCommitter-1]
2022-06-18 15:22:00,338 INFO WorkerSourceTask{id=kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [SourceTaskOffsetCommitter-1]
2022-06-18 15:23:00,342 INFO WorkerSourceTask{id=kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [SourceTaskOffsetCommitter-1]
2022-06-18 15:24:00,345 INFO WorkerSourceTask{id=kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [SourceTaskOffsetCommitter-1]
2022-06-18 15:25:00,348 INFO WorkerSourceTask{id=kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [SourceTaskOffsetCommitter-1]
2022-06-18 15:26:00,351 INFO WorkerSourceTask{id=kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [SourceTaskOffsetCommitter-1]
2022-06-18 15:27:00,354 INFO WorkerSourceTask{id=kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [SourceTaskOffsetCommitter-1]
2022-06-18 15:28:00,356 INFO WorkerSourceTask{id=kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [SourceTaskOffsetCommitter-1]
2022-06-18 15:29:00,358 INFO WorkerSourceTask{id=kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [SourceTaskOffsetCommitter-1]
2022-06-18 15:30:00,293 INFO refreshing consumer groups took 7 ms (org.apache.kafka.connect.mirror.Scheduler) [Scheduler for MirrorCheckpointConnector-refreshing consumer groups]
2022-06-18 15:30:00,362 INFO WorkerSourceTask{id=kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [SourceTaskOffsetCommitter-1]
2022-06-18 15:31:00,363 INFO WorkerSourceTask{id=kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [SourceTaskOffsetCommitter-1]
2022-06-18 15:32:00,365 INFO WorkerSourceTask{id=kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [SourceTaskOffsetCommitter-1]
2022-06-18 15:33:00,367 INFO WorkerSourceTask{id=kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [SourceTaskOffsetCommitter-1]
2022-06-18 15:34:00,369 INFO WorkerSourceTask{id=kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [SourceTaskOffsetCommitter-1]
2022-06-18 15:35:00,372 INFO WorkerSourceTask{id=kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [SourceTaskOffsetCommitter-1]
2022-06-18 15:36:00,374 INFO WorkerSourceTask{id=kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [SourceTaskOffsetCommitter-1]
2022-06-18 15:37:00,375 INFO WorkerSourceTask{id=kafka-strimzi->kafka-docker.MirrorHeartbeatConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [SourceTaskOffsetCommitter-1]

scholzj Jun 18, 2022
Maintainer

I guess that shows that it connected successfully. But you might need to check the logs to see why it isn't mirroring the data. I'm not really a MM2 expert, so not sure I can help with that.

izalul Jun 19, 2022
Author

I tried using the previous version of mirror maker, but still having problems with tls authentication.

Strimzi mirror maker user

apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaUser
metadata:
  name: user-mirror-maker
  namespace: kafka
  labels:
    strimzi.io/cluster: my-cluster
spec:
  authentication:
    type: tls

Strimzi mirror maker manifest

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker
metadata:
  name: my-mirror-maker
  namespace: kafka
spec:
  version: 3.0.0
  replicas: 1
  consumer:
    bootstrapServers: my-cluster-kafka-external-bootstrap:9094
    groupId: ".*"
    tls:
      trustedCertificates:
        - secretName: my-cluster-cluster-ca-cert
          certificate: ca.crt
    authentication:
      type: tls
      certificateAndKey:
        secretName: user-mirror-maker
        certificate: user.crt
        key: user.key
  producer:
    bootstrapServers: kafka-docker:9092
  include: ".*"

Mirror maker logs

2022-06-19 03:30:20,312 INFO [Producer clientId=producer-1] Cluster ID: 111-uqQLRnmPs1LX41BXmA (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | producer-1]
2022-06-19 03:30:20,409 INFO Kafka version: 3.0.0 (org.apache.kafka.common.utils.AppInfoParser) [main]
2022-06-19 03:30:20,409 INFO Kafka commitId: 8cb0a5e9d3441962 (org.apache.kafka.common.utils.AppInfoParser) [main]
2022-06-19 03:30:20,409 INFO Kafka startTimeMs: 1655609420408 (org.apache.kafka.common.utils.AppInfoParser) [main]
2022-06-19 03:30:20,433 INFO [mirrormaker-thread-0] Starting mirror maker thread mirrormaker-thread-0 (kafka.tools.MirrorMaker$MirrorMakerThread) [mirrormaker-thread-0]
2022-06-19 03:30:20,434 INFO [Consumer clientId=.*-0, groupId=.*] Subscribed to pattern: '.*' (org.apache.kafka.clients.consumer.KafkaConsumer) [mirrormaker-thread-0]
2022-06-19 03:30:20,580 INFO [Consumer clientId=.*-0, groupId=.*] Failed authentication with my-cluster-kafka-external-bootstrap/10.97.250.230 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [mirrormaker-thread-0]
2022-06-19 03:30:20,582 ERROR [Consumer clientId=.*-0, groupId=.*] Connection to node -1 (my-cluster-kafka-external-bootstrap/10.97.250.230:9094) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient) [mirrormaker-thread-0]
2022-06-19 03:30:20,582 WARN [Consumer clientId=.*-0, groupId=.*] Bootstrap broker my-cluster-kafka-external-bootstrap:9094 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient) [mirrormaker-thread-0]
2022-06-19 03:30:20,587 ERROR [mirrormaker-thread-0] Mirror maker thread failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread) [mirrormaker-thread-0]
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching my-cluster-kafka-external-bootstrap found.
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:349)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)
	at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430)
	at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514)
	at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368)
	at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291)
	at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:551)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215)
	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:246)
	at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:480)
	at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1262)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1231)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1211)
	at kafka.tools.MirrorMaker$ConsumerWrapper.receive(MirrorMaker.scala:327)
	at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:216)
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching my-cluster-kafka-external-bootstrap found.
	at java.base/sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:212)
	at java.base/sun.security.util.HostnameChecker.match(HostnameChecker.java:103)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:429)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:283)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335)
	... 26 more
2022-06-19 03:30:20,590 INFO [mirrormaker-thread-0] Flushing producer. (kafka.tools.MirrorMaker$MirrorMakerThread) [mirrormaker-thread-0]
2022-06-19 03:30:20,592 INFO [mirrormaker-thread-0] Committing consumer offsets. (kafka.tools.MirrorMaker$MirrorMakerThread) [mirrormaker-thread-0]
2022-06-19 03:30:20,592 INFO Exiting on send failure, skip committing offsets. (kafka.tools.MirrorMaker$) [mirrormaker-thread-0]
2022-06-19 03:30:20,592 INFO [mirrormaker-thread-0] Shutting down consumer connectors. (kafka.tools.MirrorMaker$MirrorMakerThread) [mirrormaker-thread-0]
2022-06-19 03:30:20,593 INFO [Consumer clientId=.*-0, groupId=.*] Resetting generation due to: consumer pro-actively leaving the group (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [mirrormaker-thread-0]
2022-06-19 03:30:20,593 INFO [Consumer clientId=.*-0, groupId=.*] Request joining group due to: consumer pro-actively leaving the group (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [mirrormaker-thread-0]
2022-06-19 03:30:20,594 INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics) [mirrormaker-thread-0]
2022-06-19 03:30:20,594 INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics) [mirrormaker-thread-0]
2022-06-19 03:30:20,594 INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics) [mirrormaker-thread-0]
2022-06-19 03:30:20,599 INFO App info kafka.consumer for .*-0 unregistered (org.apache.kafka.common.utils.AppInfoParser) [mirrormaker-thread-0]
2022-06-19 03:30:20,599 INFO [mirrormaker-thread-0] Mirror maker thread stopped (kafka.tools.MirrorMaker$MirrorMakerThread) [mirrormaker-thread-0]
2022-06-19 03:30:20,599 INFO [mirrormaker-thread-0] Mirror maker thread shutdown complete (kafka.tools.MirrorMaker$MirrorMakerThread) [main]
2022-06-19 03:30:20,599 ERROR [mirrormaker-thread-0] Mirror maker thread exited abnormally, stopping the whole mirror maker. (kafka.tools.MirrorMaker$MirrorMakerThread) [mirrormaker-thread-0]
2022-06-19 03:30:20,601 INFO Start clean shutdown. (kafka.tools.MirrorMaker$) [MirrorMakerShutdownHook]
2022-06-19 03:30:20,601 INFO Shutting down consumer threads. (kafka.tools.MirrorMaker$) [MirrorMakerShutdownHook]
2022-06-19 03:30:20,602 INFO [mirrormaker-thread-0] mirrormaker-thread-0 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread) [MirrorMakerShutdownHook]

izalul Jun 19, 2022
Author

I also tried using plain authentication but still no luck, Not authorized to access group: .*

Strimzi mirror maker manifest

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker
metadata:
  name: my-mirror-maker
  namespace: kafka
spec:
  version: 2.8.0
  replicas: 1
  consumer:
    bootstrapServers: my-cluster-kafka-bootstrap:9092
    groupId: ".*"
  producer:
    bootstrapServers: kafka-docker:9092
  include: ".*"

Mirror maker logs

2022-06-19 04:17:47,349 INFO [Consumer clientId=.*-0, groupId=.*] Subscribed to pattern: '.*' (org.apache.kafka.clients.consumer.KafkaConsumer) [mirrormaker-thread-0]
2022-06-19 04:17:47,603 INFO [Producer clientId=producer-1] Cluster ID: 111-uqQLRnmPs1LX41BXmA (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | producer-1]
2022-06-19 04:17:47,603 INFO [Consumer clientId=.*-0, groupId=.*] Cluster ID: SptcRy57Q_OOfdyzWl5d8Q (org.apache.kafka.clients.Metadata) [mirrormaker-thread-0]
2022-06-19 04:17:47,605 INFO [Consumer clientId=.*-0, groupId=.*] FindCoordinator request hit fatal exception (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [mirrormaker-thread-0]
org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: .*
2022-06-19 04:17:47,611 ERROR [mirrormaker-thread-0] Mirror maker thread failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread) [mirrormaker-thread-0]
org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: .*
2022-06-19 04:17:47,612 INFO [mirrormaker-thread-0] Flushing producer. (kafka.tools.MirrorMaker$MirrorMakerThread) [mirrormaker-thread-0]
2022-06-19 04:17:47,613 INFO [mirrormaker-thread-0] Committing consumer offsets. (kafka.tools.MirrorMaker$MirrorMakerThread) [mirrormaker-thread-0]
2022-06-19 04:17:47,613 INFO Exiting on send failure, skip committing offsets. (kafka.tools.MirrorMaker$) [mirrormaker-thread-0]
2022-06-19 04:17:47,614 INFO [mirrormaker-thread-0] Shutting down consumer connectors. (kafka.tools.MirrorMaker$MirrorMakerThread) [mirrormaker-thread-0]
2022-06-19 04:17:47,615 INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics) [mirrormaker-thread-0]
2022-06-19 04:17:47,615 INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics) [mirrormaker-thread-0]
2022-06-19 04:17:47,615 INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics) [mirrormaker-thread-0]
2022-06-19 04:17:47,622 INFO App info kafka.consumer for .*-0 unregistered (org.apache.kafka.common.utils.AppInfoParser) [mirrormaker-thread-0]
2022-06-19 04:17:47,623 INFO [mirrormaker-thread-0] Mirror maker thread stopped (kafka.tools.MirrorMaker$MirrorMakerThread) [mirrormaker-thread-0]
2022-06-19 04:17:47,623 INFO [mirrormaker-thread-0] Mirror maker thread shutdown complete (kafka.tools.MirrorMaker$MirrorMakerThread) [main]
2022-06-19 04:17:47,623 ERROR [mirrormaker-thread-0] Mirror maker thread exited abnormally, stopping the whole mirror maker. (kafka.tools.MirrorMaker$MirrorMakerThread) [mirrormaker-thread-0]
2022-06-19 04:17:47,625 INFO Start clean shutdown. (kafka.tools.MirrorMaker$) [MirrorMakerShutdownHook]
2022-06-19 04:17:47,625 INFO Shutting down consumer threads. (kafka.tools.MirrorMaker$) [MirrorMakerShutdownHook]
2022-06-19 04:17:47,626 INFO [mirrormaker-thread-0] mirrormaker-thread-0 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread) [MirrorMakerShutdownHook]
2022-06-19 04:17:47,627 INFO [mirrormaker-thread-0] Mirror maker thread shutdown complete (kafka.tools.MirrorMaker$MirrorMakerThread) [MirrorMakerShutdownHook]
2022-06-19 04:17:47,627 INFO Closing producer. (kafka.tools.MirrorMaker$) [MirrorMakerShutdownHook]
2022-06-19 04:17:47,627 INFO [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer) [MirrorMakerShutdownHook]
2022-06-19 04:17:47,630 INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics) [MirrorMakerShutdownHook]
2022-06-19 04:17:47,630 INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics) [MirrorMakerShutdownHook]
2022-06-19 04:17:47,630 INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics) [MirrorMakerShutdownHook]
2022-06-19 04:17:47,630 INFO App info kafka.producer for producer-1 unregistered (org.apache.kafka.common.utils.AppInfoParser) [MirrorMakerShutdownHook]
2022-06-19 04:17:47,630 INFO Kafka mirror maker shutdown successfully (kafka.tools.MirrorMaker$) [MirrorMakerShutdownHook]

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

Kafka mirror maker strimzi to docker #6957

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 6 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Strimzi

Kafka mirror maker strimzi to docker #6957

Uh oh!

izalul Jun 18, 2022

Replies: 1 comment · 6 replies

Uh oh!

scholzj Jun 18, 2022 Maintainer

Uh oh!

scholzj Jun 18, 2022 Maintainer

Uh oh!

Uh oh!

izalul Jun 18, 2022 Author

Strimzi MM2 Manifest:

MirrroMaker Logs:

Uh oh!

scholzj Jun 18, 2022 Maintainer

Uh oh!

izalul Jun 19, 2022 Author

Strimzi mirror maker user

Strimzi mirror maker manifest

Mirror maker logs

Uh oh!

izalul Jun 19, 2022 Author

Strimzi mirror maker manifest

Mirror maker logs

izalul
Jun 18, 2022

Replies: 1 comment 6 replies

scholzj
Jun 18, 2022
Maintainer

scholzj Jun 18, 2022
Maintainer

izalul Jun 18, 2022
Author

scholzj Jun 18, 2022
Maintainer

izalul Jun 19, 2022
Author

izalul Jun 19, 2022
Author