Strimzi Mirrormaker2 and Event Hub Odd Error, Can't Pinpoint

[walkinverse]

I am trying to sync Kafka to Azure Event Hub, one way. I followed every tutorial I could find to no avail. Nothing seems to work as I keep getting obscure errors. Below is the config used for deployment. We even used the RootManageSharedAccessKey to make sure nothing blocks. I have Kafka, KafkaConnect deployed without issue.

MirrorMaker2 works between Kafka and Kafka, but no dice when I try to sync with EventHub.

To test if port 9093 was accessible, I successfully used telnet to access it.

I have tried the below config with the authentication block, and then with sasl under the config section. Comments below.

MM2 Config

cat <<EOF | kubectl apply -n kafka-cloud -f -
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: mirror-maker-eventhub
spec:
  version: 3.1.0
  replicas: 1
  connectCluster: "eventhub"
  clusters:
  - alias: "my-kafka-cluster"
    bootstrapServers: my-kafka-cluster-kafka-bootstrap:9092
  - alias: "eventhub"
    bootstrapServers: XXXXXXXXXXXXXXXXXXXX.servicebus.windows.net:9093
    config:
      config.storage.replication.factor: 1
      offset.storage.replication.factor: 1
      status.storage.replication.factor: 1
      producer.connections.max.idle.ms: 180000
      producer.metadata.max.age.ms: 180000
      security.protocol: SASL_SSL
      sasl.mechanism: PLAIN
      sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://XXXXXXXXXXXXXXXXXXXX.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=XXXXXXXXXXXXXXXXXXXX";
    tls:
      trustedCertificates: []
  mirrors:
  - sourceCluster: "my-kafka-cluster"
    targetCluster: "eventhub"
    sourceConnector:
      config:
        replication.factor: 1
        offset-syncs.topic.replication.factor: 1
        sync.topic.acls.enabled: "false"
    heartbeatConnector:
      config:
        heartbeats.topic.replication.factor: 1
    checkpointConnector:
      config:
        checkpoints.topic.replication.factor: 1
    topicsPattern: ".*"
    groupsPattern: ".*"
EOF

First issue were these warnings. I know it's only INFO, but the deployment keep crashing and will not stay running. I also know with Kafka these type of logs are vague at best.

2022-03-15 21:21:06,151 INFO [AdminClient clientId=adminclient-1] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]

2022-03-15 21:21:06,152 INFO [AdminClient clientId=adminclient-1] Cancelled in-flight METADATA request with correlation id 59 due to node -1 being disconnected (elapsed time since creation: 87ms, elapsed time since send: 87ms, request timeout: 16401ms) (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]

Then I was able to catch this error in the logs. I can't tell where the problem lies and can't figure out how to resolve it.

2022-03-15 21:22:55,572 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata
2022-03-15 21:22:55,574 INFO App info kafka.admin.client for adminclient-1 unregistered (org.apache.kafka.common.utils.AppInfoParser) [kafka-admin-client-thread | adminclient-1]
2022-03-15 21:22:55,575 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata
2022-03-15 21:22:55,575 INFO [AdminClient clientId=adminclient-1] Timed out 1 remaining operation(s) during close. (org.apache.kafka.clients.admin.KafkaAdminClient) [kafka-admin-client-thread | adminclient-1]
2022-03-15 21:22:55,582 INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2022-03-15 21:22:55,582 INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2022-03-15 21:22:55,582 INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2022-03-15 21:22:55,583 ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed) [main]
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check workers broker connection and security properties.
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:70)
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:51)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:97)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes

When I run a describe I get this error

Status:
  Conditions:
    Last Transition Time:  2022-03-15T21:25:49.149589Z
    Message:               Exceeded timeout of 300000ms while waiting for Deployment resource mirror-maker-eventhub-mirrormaker2 in namespace kafka-cloud to be ready
    Reason:                TimeoutException
    Status:                True
    Type:                  NotReady
  Label Selector:          strimzi.io/cluster=mirror-maker-eventhub,strimzi.io/name=mirror-maker-eventhub-mirrormaker2,strimzi.io/kind=KafkaMirrorMaker2
  Observed Generation:     1
  Replicas:                1
  URL:                     http://mirror-maker-eventhub-mirrormaker2-api.kafka-cloud.svc:8083
Events:                    <none>

When I add the authentication block in mm2:

    authentication: 
      type: plain 
      username: $ConnectionString
      passwordSecret: 
        secretName: eventhubssecret 
        password: eventhubspassword

I get this error, even though the secret exists and is validated.

Status:
  Conditions:
    Last Transition Time:  2022-03-15T23:30:31.200105Z
    Message:               PLAIN authentication selected, but username or password configuration is missing.
    Reason:                InvalidResourceException
    Status:                True
    Type:                  NotReady
  Observed Generation:     2
  Replicas:                0
Events:                    <none>

If anyone has any idea, it'd be greatly appreciated. Spent the entire day to no avail. I removed using secretes as it caused an error that the name and password were not supplied. It's weird.

Thank you.

[scholzj]

How does this differ from https://stackoverflow.com/questions/71489737/strimzi-mirrormaker2-and-event-hub-odd-error-cant-pinpoint/71489915?

[scholzj]

Well, you need to figure out why it cannot read the secret. Either you badly created it or you have some setup issue. But the guide in the blog post works.

[walkinverse]

I cut and pasted from the blog directly and made the changes to my eventhub passwords. Could it be a strimzi operator issue with mirrormaker or a bug? We followed the guide to the letter. Any ideas?

This is the command used to create the secret. I don't see anything wrong with it, do you? Am I missing something in the way I am creating it? Also we noticed in the blog its using v1alpha1. we are on version 0.28 and beta2 for the api verison.

cat <<EOF | kubectl apply -n kafka-cloud -f - 
apiVersion: v1
kind: Secret
metadata:
  name: eventhubssecret
type: Opaque
stringData:
  eventhubspassword: Endpoint=sb://<namespace>.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=xxxxxxxxxx
EOF 

[scholzj]

I do not have Azure to try the whole thing. But the SASL PLAIN configuration and the secret handling work fine for me.

[walkinverse]

If I find an answer I'll let you know. This is an odd one. IF you come across anything please let me know. I have had 3 people now look at this. And even they are perplexed. Makes no sense. Did you create any service accounts for Kafka? Or Roles for MirrorMaker2, or just verbatim as in the blog? I did notice the blog uses an alpha version of the API and 0.28 use beta. Could there be a difference?

I will post another question and point it back to here and see if anyone had any issues running in Azure Kubernetes. I am assuming you run on OpenShift. Thank you again for all your advice, I greatly appreciate it.

[scholzj]

The blog post is old, so it uses older version of the API - if you want to copy it exactly you will need to update it. But the authentication part is still the same. I use Kubernetes - I tried it there earlier today to be sure there is no bug (as I said, I did not tried the whole chain to Azure -> but the secret was recognized and used).

[walkinverse]

Answer found.

The issue was using cat <<EOF | kubectl apply -n kafka-cloud -f - ... EOF

I think it's because of $ in the username. EH needs this as the actual username for the connection. Once I made the above into a file between cat <<EOF and the last EOF it ran from the CLI without changing anything.

It worked.

kubectl apply -n kafka-cloud -f fileName.yaml

When working with EH use a file. Don't cat <<EOF it in.