Cannot list kafka topics & kafka acl

[chary1112004]

1. Follow this Q&A Cannot list kafka topics using bin/kafka-topics.sh --list #1470, I still could not list topic with below command:

From kafka bootstrap server pod

kubectl -n kafka exec -it my-cluster-kafka-0 -- ./bin/kafka-topics.sh --list --zookeeper localhost:2181
Exception in thread "main" joptsimple.UnrecognizedOptionException: zookeeper is not a recognized option
        at joptsimple.OptionException.unrecognizedOption(OptionException.java:108)
        at joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:510)
        at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56)
        at joptsimple.OptionParser.parse(OptionParser.java:396)
        at kafka.admin.TopicCommand$TopicCommandOptions.<init>(TopicCommand.scala:567)
        at kafka.admin.TopicCommand$.main(TopicCommand.scala:47)
        at kafka.admin.TopicCommand.main(TopicCommand.scala)

kubectl -n kafka exec -it my-cluster-kafka-0 -- ./bin/kafka-topics.sh --list --bootstrap-server my-cluster-kafka-bootstrap:9092
Error while executing topic command : Timed out waiting for a node assignment. Call: listTopics
[2023-04-17 04:40:22,652] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics
 (kafka.admin.TopicCommand$)

From kafka zookeeper pod

kubectl -n kafka exec -it my-cluster-zookeeper-0 -- ./bin/kafka-topics.sh --list --zookeeper localhost:2181
Exception in thread "main" joptsimple.UnrecognizedOptionException: zookeeper is not a recognized option
        at joptsimple.OptionException.unrecognizedOption(OptionException.java:108)
        at joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:510)
        at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56)
        at joptsimple.OptionParser.parse(OptionParser.java:396)
        at kafka.admin.TopicCommand$TopicCommandOptions.<init>(TopicCommand.scala:567)
        at kafka.admin.TopicCommand$.main(TopicCommand.scala:47)
        at kafka.admin.TopicCommand.main(TopicCommand.scala)

2. Beside of that, I would like to list kafka acls however it does not work for me as well

kubectl -n kafka exec -it my-cluster-zookeeper-0  -- ./bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --list
Warning: support for ACL configuration directly through the authorizer is deprecated and will be removed in a future release. Please use --bootstrap-server instead to set ACLs through the admin client.
[2023-04-17 04:58:23,905] WARN Session 0x0 for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
EndOfStreamException: Unable to read additional data from server sessionid 0x0, likely server has closed socket
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290)

Kafka version: 3.4.0

Any solution to resolve this issue would be very much appreciated.

Thanks!

[ppatierno]

Which Strimzi and Kafka version are you using? Can you share the Kafka custom resource and mainly the listeners declaration, please?

[chary1112004]

Hi @ppatierno,

I am using kafka version 3.4.0.

Here is kafka custom resource that I am using:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: 3.4.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
        authentication:
          type: scram-sha-512
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: scram-sha-512
    authorization:
      type: simple
    config:
      auto.create.topics.enable: "false"
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.4"
    storage:
      type: ephemeral
  zookeeper:
    replicas: 3
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}

Regards

[chary1112004]

Which Strimzi and Kafka version are you using? Can you share the Kafka custom resource and mainly the listeners declaration, please?

Hi @ppatierno,

I could get list of topic success with below command follow https://stackoverflow.com/questions/69752997/kafka-topics-sh-authentication

/tmp/client_sasl.properties

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
ssl.truststore.location=/tmp/my-cluster-ca.p12
ssl.truststore.password=<trust store pwd>
ssl.truststore.type=PKCS12
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="my-user" password="<password>";

Get list topic:

bin/kafka-topics.sh --list --bootstrap-server my-cluster-kafka-bootstrap:9093 --command-config /tmp/client_sasl.properties

However I still could not get list of acl as it requires to call to zookeeper follow this document https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Authorization+Command+Line+Interface

UPDATED:
I could get success the list of acl as below follow #3715

bin/kafka-acls.sh --list --bootstrap-server my-cluster-kafka-bootstrap:9093 --command-config /tmp/client_sasl.properties

It requires to add permission type: cluster to user my-user allow get list of acl:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: my-user
  labels:
    strimzi.io/cluster: my-cluster
spec:
  authentication:
    type: scram-sha-512
  authorization:
    type: simple
    acls:
      # Example ACL rules for consuming from my-topic using consumer group my-group
      - resource:
          type: topic
          name: my-topic
          patternType: literal
        operations:
          - Describe
          - Read
        host: "*"
      - resource:
          type: group
          name: "*"
          patternType: literal
        operations:
          - All
        host: "*"
      # Example ACL rules for producing to topic my-topic
      - resource:
          type: topic
          name: my-topic
          patternType: literal
        operations:
          - Create
          - Describe
          - Write
        host: "*"
      - resource:
          type: cluster
          name: "*"
          patternType: literal
        operations:
          - Create
          - Describe
          - Write
        host: "*"

Thanks!

[ppatierno]

@chary1112004 yes your problem on port 9092 was the authorization part ...

authentication:
          type: scram-sha-512

so it wasn't able to connect. I see you fixed the issue.
Regarding ACLS, I don't think that page is updated and you should still use --bootstrap-server option.

[chary1112004]

@chary1112004 yes your problem on port 9092 was the authorization part ...

authentication:
          type: scram-sha-512

so it wasn't able to connect. I see you fixed the issue. Regarding ACLS, I don't think that page is updated and you should still use --bootstrap-server option.

Yes, it is a bit strange for me to use bootstrap server as well. When we use aws kafka (msk), it is used zookeeper completely to manage topic/acl as in https://docs.aws.amazon.com/msk/latest/developerguide/msk-acls.html

Then we could close this one!

Thanks!

[ppatierno]

Maybe MSK is still using a Kafka version where not all Kafka tools have the bootstrap server option but still using the ZooKeeper one. Take into account that Kafka is anyway moving to KRaft so ZooKeeper won't be there anymore at some point. I would move all my handling with Kafka tools to use the bootstrap server.

[chary1112004]

Maybe MSK is still using a Kafka version where not all Kafka tools have the bootstrap server option but still using the ZooKeeper one. Take into account that Kafka is anyway moving to KRaft so ZooKeeper won't be there anymore at some point. I would move all my handling with Kafka tools to use the bootstrap server.

Yes, thank you for your information!

Then we could close this one!

Regards!