Amazon MSK (provisioned) cluster is Offline.

[githubeto]

Issue submitter TODO list

• I've looked up my issue in FAQ
• I've searched for an already existing issues here
• I've tried running master-labeled docker image and the issue still persists there
• I'm running a supported version of the application which is listed here

Describe the bug (actual behavior)

We are building MSK (Managed Streaming for Apache Kafka) and trying to connect to MSK from EKS (Elastic Kubernetes Service) pods using IRSA (IAM Roles for Service Accounts). No authentication method is configured for MSK.
However, the cluster appears offline, and brokers and other elements are not displayed on the screen. Additionally, error logs are being output.

---

Expected behavior

No response

Your installation details

App Version = v1.0.0
Helm chart version = kafka-ui-1.4.9
Your application config : none.
Any IAAC configs : none.

Steps to reproduce

helm upgrade --install kafbat-ui -f kafka-ui-values.yaml kafbat-ui/kafka-ui

2. port-forward or create virtualservice(istio)

kafka-ui-values.yaml

replicaCount: 1
autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 2
  targetCPUUtilizationPercentage: 80
serviceAccount: 
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxx:role/odc-stg-kafka-ui
  name: "kafka-ui"
ingress:
  enabled: false
envs:
  config:
    KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: b-1.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092,b-2.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092,b-3.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092
    KAFKA_CLUSTERS_0_NAME: spdkube-stg-cdc-msk
    KAFKA_CLUSTERS_0_READONLY: "true"
    KAFKA_CLUSTERS_0_ZOOKEEPER: z-1.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:2181,z-2.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:2181,z-3.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:2181
    KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SASL_SSL
    KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM: AWS_MSK_IAM
    KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG: 'software.amazon.msk.auth.iam.IAMLoginModule required;'
    KAFKA_CLUSTERS_0_PROPERTIES_SASL_CLIENT_CALLBACK_HANDLER_CLASS: 'software.amazon.msk.auth.iam.IAMClientCallbackHandler'
    LOGGING_LEVEL_ROOT: info
resources:
  limits:
    memory: 512Mi
  requests:
    cpu: 300m
    memory: 512Mi

IAM role (arn:aws:iam::xxxxxxxxxx:role/odc-stg-kafka-ui) policy.

{
    "Statement": [
        {
            "Action": [
                "kafka-cluster:DescribeCluster",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:Connect"
            ],
            "Effect": "Allow",
            "Resource": "<cluster-arn>",
            "Sid": "VisualEditor0"
        },
        {
            "Action": [
                "kafka-cluster:DeleteGroup",
                "kafka-cluster:DescribeCluster",
                "kafka-cluster:ReadData",
                "kafka-cluster:DescribeTopicDynamicConfiguration",
                "kafka-cluster:AlterTopicDynamicConfiguration",
                "kafka-cluster:AlterGroup",
                "kafka-cluster:AlterClusterDynamicConfiguration",
                "kafka-cluster:AlterTopic",
                "kafka-cluster:CreateTopic",
                "kafka-cluster:DescribeTopic",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:DescribeGroup",
                "kafka-cluster:DescribeClusterDynamicConfiguration",
                "kafka-cluster:Connect",
                "kafka-cluster:DeleteTopic",
                "kafka-cluster:WriteData"
            ],
            "Effect": "Allow",
            "Resource": "<cluster-arn>/*",
            "Sid": "VisualEditor1"
        },
        {
            "Action": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Effect": "Allow",
            "Resource": "<cluster-arn>/*",
            "Sid": "VisualEditor2"
        }
    ],
    "Version": "2012-10-17"
}

Screenshots

No response

Logs

2024-11-22 11:24:48,175 ERROR [parallel-2] c.p.k.u.s.StatisticsService: Failed to collect cluster spdkube-stg-cdc-msk info
java.lang.IllegalStateException: Error while creating AdminClient for Cluster spdkube-stg-cdc-msk
	at com.provectus.kafka.ui.service.AdminClientServiceImpl.lambda$createAdminClient$5(AdminClientServiceImpl.java:56)
	at reactor.core.publisher.Mono.lambda$onErrorMap$28(Mono.java:3783)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258)
	at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onError(FluxMap.java:265)
	at reactor.core.publisher.Operators$MonoSubscriber.onError(Operators.java:1886)
	at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.signalCached(MonoCacheTime.java:340)
	at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.onError(MonoCacheTime.java:363)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:180)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onError(MonoIgnoreThen.java:278)
	at reactor.core.publisher.MonoPublishOn$PublishOnSubscriber.run(MonoPublishOn.java:187)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes

Additional context

No response

[github-actions]

Hi githubeto! 👋

Welcome, and thank you for opening your first issue in the repo!

Please wait for triaging by our maintainers.

As development is carried out in our spare time, you can support us by sponsoring our activities or even funding the development of specific issues.
Sponsorship link

If you plan to raise a PR for this issue, please take a look at our contributing guide.

[lsauter-protectline]

Hello,
I am using the same environnent (on eks with a Amazon MSK provisionned) with the image version ghcr.io/kafbat/kafka-ui:v1.1.0
I am using the helm chart kafka-ui-1.4.11
I also tryed a IAM policy/role according this documentation.

I got the same error:

ERROR [parallel-2] i.k.u.s.StatisticsService: Failed to collect cluster kafka-cluster info
java.lang.IllegalStateException: Error while creating AdminClient for the cluster kafka-cluster
        at io.kafbat.ui.service.AdminClientServiceImpl.lambda$createAdminClient$5(AdminClientServiceImpl.java:56)
        at reactor.core.publisher.Mono.lambda$onErrorMap$29(Mono.java:3862)
        at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
        at reactor.core.publisher.Operators.error(Operators.java:198)
        at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:136)
        at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53)
        at reactor.core.publisher.Mono.subscribe(Mono.java:4560)
        at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:82)
        at reactor.core.publisher.Operators.complete(Operators.java:137)
        at reactor.core.publisher.MonoEmpty.subscribe(MonoEmpty.java:46)
        at reactor.core.publisher.Mono.subscribe(Mono.java:4576)
        at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:430)
        at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.runAsync(FluxPublishOn.java:446)
        at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.run(FluxPublishOn.java:533)
        at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)
        at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.NullPointerException: null
        at java.base/java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
        at java.base/java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
        at java.base/java.util.Properties.put(Properties.java:1346)
        at io.kafbat.ui.service.AdminClientServiceImpl.lambda$createAdminClient$2(AdminClientServiceImpl.java:47)
        at reactor.core.publisher.MonoSupplier.call(MonoSupplier.java:67)
        at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:128)
        ... 16 common frames omitted  

Kafka-ui is not able to connect to MSK

My config.yml is like this:

auth:
    type: disabled
  kafka:
    clusters:
    - bootstrap.servers: kafka-id.kafka.region.amazonaws.com:port
      name: kafka-cluster
      properties:
        security.protocol: SASL_SSL
        sasl.mechanism: AWS_MSK_IAM
        sasl.client.callback.handler.class: software.amazon.msk.auth.iam.IAMClientCallbackHandler
        sasl.jaas.config: 'software.amazon.msk.auth.iam.IAMLoginModule required;'
  management:
    health:
      ldap:
        enabled: false

Maybe I forgot something, thank you for the help

[Haarolean]

@lsauter-protectline your config is invalid: your clusters declaration is under 'auth' block which is wrong.

[lsauter-protectline]

thank you @Haarolean for your reply.
I tried again with this config.yml and i got the same error.

kafka:
  clusters:
  - bootstrap.servers: kafka-id.kafka.region.amazonaws.com:port
    name: kafka-cluster
    properties:
      security.protocol: SASL_SSL
      sasl.mechanism: AWS_MSK_IAM
      sasl.client.callback.handler.class: software.amazon.msk.auth.iam.IAMClientCallbackHandler
      sasl.jaas.config: 'software.amazon.msk.auth.iam.IAMLoginModule required;'
management:
  health:
    ldap:
      enabled: false
auth:
  type: disabled

[germanosin]

@lsauter-protectline
It seems like there's an issue with the configuration again. Instead of "bootstrap.servers," it should be "bootstrapServers." I understand that the log message isn't very clear, and we'll definitely work on improving it.

[germanosin]

Hi @githubeto,

It seems like your error may be a bit different from @lsauter-protectline . Here are a couple of things to check:

1. I don't see ACLs in the policy that allow the ListTopics action for this particular role. Could you kindly verify that?
2. Could you please confirm if the following hosts and ports are reachable from the Kafka-UI instance?

• b-1.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092
• b-2.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092
• b-3.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092

Thank you for your help, and feel free to let me know if you need any further assistance!