KafkaUser Secrets Not Created from UserOperator via Entity Operator

[Hemant0601]

Try creating a KafkaUser like

apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaUser metadata: name: super-admin labels: strimzi.io/cluster: my-cluster spec: authentication: type: scram-sha-512

User shows in the created list but secrets are not visible in kubectl get secrets

[sknot-rh]

Hi. Just a silly question as this happened to me a lot of times. Do you execute kubectl get secrets in the namespace operated by the UserOperator?

[Hemant0601]

@sknot-rh Yes I did it with the namespace. I could see my other certificates there, but could not see anything related to this.
I want a scram-sha-512 authentication for my Kafka users remotely.

[scholzj]

You should check:

• if the strimzi.io/cluster has the right value.
• What is the status of the KafkaUser resource (the .status section when you do kubect get kafkausers)
• Check the logs of the user operator to see if it saw the user or now

[scholzj]

So, can you do kubectl get kafkausers -o yaml? Also, can you share the user operator log?

[Hemant0601]

apiVersion: v1
items:
- apiVersion: kafka.strimzi.io/v1beta2
  kind: KafkaUser
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"kafka.strimzi.io/v1beta2","kind":"KafkaUser","metadata":{"annotations":{},"labels":{"strimzi.io/cluster":"my-cluster"},"name":"admin-moxiam","namespace":"kafka"},"spec":{"authentication":{"type":"scram-sha-512"},"authorization":{"acls":[{"host":"*","operation":"Read","resource":{"name":"moxiam-stream-raw-data","patternType":"literal","type":"topic"}},{"host":"*","operation":"Describe","resource":{"name":"moxiam-stream-raw-data","patternType":"literal","type":"topic"}},{"host":"*","operation":"Read","resource":{"name":"moxiam-stream-output-data","patternType":"literal","type":"topic"}},{"host":"*","operation":"Describe","resource":{"name":"moxiam-stream-output-data","patternType":"literal","type":"topic"}},{"host":"*","operation":"Write","resource":{"name":"moxiam-stream-raw-data","patternType":"literal","type":"topic"}},{"host":"*","operation":"Create","resource":{"name":"moxiam-stream-raw-data","patternType":"literal","type":"topic"}},{"host":"*","operation":"Describe","resource":{"name":"moxiam-stream-raw-data","patternType":"literal","type":"topic"}},{"host":"*","operation":"Write","resource":{"name":"moxiam-stream-output-data","patternType":"literal","type":"topic"}},{"host":"*","operation":"Create","resource":{"name":"moxiam-stream-output-data","patternType":"literal","type":"topic"}},{"host":"*","operation":"Describe","resource":{"name":"moxiam-stream-output-data","patternType":"literal","type":"topic"}}],"type":"simple"}}}
    creationTimestamp: "2021-05-11T09:04:54Z"
    generation: 1
    labels:
      strimzi.io/cluster: my-cluster
    managedFields:
    - apiVersion: kafka.strimzi.io/v1beta2
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:labels:
            .: {}
            f:strimzi.io/cluster: {}
        f:spec:
          .: {}
          f:authentication:
            .: {}
            f:type: {}
          f:authorization:
            .: {}
            f:acls: {}
            f:type: {}
      manager: kubectl-client-side-apply
      operation: Update
      time: "2021-05-11T09:04:54Z"
    name: admin-moxiam
    namespace: kafka
    resourceVersion: "25951725"
    selfLink: /apis/kafka.strimzi.io/v1beta2/namespaces/kafka/kafkausers/admin-moxiam
    uid: 01c3699f-bb33-4944-983e-57a5e9bda420
  spec:
    authentication:
      type: scram-sha-512
    authorization:
      acls:
      - host: '*'
        operation: Read
        resource:
          name: moxiam-stream-raw-data
          patternType: literal
          type: topic
      - host: '*'
        operation: Describe
        resource:
          name: moxiam-stream-raw-data
          patternType: literal
          type: topic
      - host: '*'
        operation: Read
        resource:
          name: moxiam-stream-output-data
          patternType: literal
          type: topic
      - host: '*'
        operation: Describe
        resource:
          name: moxiam-stream-output-data
          patternType: literal
          type: topic
      - host: '*'
        operation: Write
        resource:
          name: moxiam-stream-raw-data
          patternType: literal
          type: topic
      - host: '*'
        operation: Create
        resource:
          name: moxiam-stream-raw-data
          patternType: literal
          type: topic
      - host: '*'
        operation: Describe
        resource:
          name: moxiam-stream-raw-data
          patternType: literal
          type: topic
      - host: '*'
        operation: Write
        resource:
          name: moxiam-stream-output-data
          patternType: literal
          type: topic
      - host: '*'
        operation: Create
        resource:
          name: moxiam-stream-output-data
          patternType: literal
          type: topic
      - host: '*'
        operation: Describe
        resource:
          name: moxiam-stream-output-data
          patternType: literal
          type: topic
      type: simple
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

[Hemant0601]

It seems my enitity operator pod is down. I am checking if thats causing the issue.

[Hemant0601]

Here are my logs , Its saying Authorization needs to be enabled in the Kafka custom resource

2021-05-11 09:49:03 INFO  OperatorWatcher:40 - Reconciliation #0(watch) KafkaUser(kafka/admin-moxiam): KafkaUser admin-moxiam in namespace kafka was ADDED
2021-05-11 09:49:03 INFO  AbstractOperator:254 - Reconciliation #0(watch) KafkaUser(kafka/admin-moxiam): KafkaUser admin-moxiam will be checked for creation or modification
2021-05-11 09:49:03 ERROR SimpleAclOperator:81 - Reconciliation failed for user admin-moxiam
io.strimzi.operator.cluster.model.InvalidResourceException: Authorization needs to be enabled in the Kafka custom resource
	at io.strimzi.operator.user.operator.SimpleAclOperator.getAcls(SimpleAclOperator.java:209) ~[io.strimzi.user-operator-0.22.1.jar:0.22.1]
	at io.strimzi.operator.user.operator.SimpleAclOperator.lambda$reconcile$0(SimpleAclOperator.java:73) ~[io.strimzi.user-operator-0.22.1.jar:0.22.1]
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.apache.kafka.common.errors.SecurityDisabledException: No Authorizer is configured on the broker
2021-05-11 09:49:03 ERROR AbstractOperator:275 - Reconciliation #0(watch) KafkaUser(kafka/admin-moxiam): createOrUpdate failed
io.strimzi.operator.cluster.model.InvalidResourceException: Authorization needs to be enabled in the Kafka custom resource
	at io.strimzi.operator.user.operator.SimpleAclOperator.getAcls(SimpleAclOperator.java:209) ~[io.strimzi.user-operator-0.22.1.jar:0.22.1]
	at io.strimzi.operator.user.operator.SimpleAclOperator.lambda$reconcile$0(SimpleAclOperator.java:73) ~[io.strimzi.user-operator-0.22.1.jar:0.22.1]
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.apache.kafka.common.errors.SecurityDisabledException: No Authorizer is configured on the broker
2021-05-11 09:49:04 INFO  OperatorWatcher:40 - Reconciliation #1(watch) KafkaUser(kafka/admin-moxiam): KafkaUser admin-moxiam in namespace kafka was MODIFIED
2021-05-11 09:49:04 INFO  CrdOperator:112 - Status of KafkaUser admin-moxiam in namespace kafka has been updated
2021-05-11 09:49:04 WARN  AbstractOperator:508 - Reconciliation #0(watch) KafkaUser(kafka/admin-moxiam): Failed to reconcile
io.strimzi.operator.cluster.model.InvalidResourceException: Authorization needs to be enabled in the Kafka custom resource
	at io.strimzi.operator.user.operator.SimpleAclOperator.getAcls(SimpleAclOperator.java:209) ~[io.strimzi.user-operator-0.22.1.jar:0.22.1]
	at io.strimzi.operator.user.operator.SimpleAclOperator.lambda$reconcile$0(SimpleAclOperator.java:73) ~[io.strimzi.user-operator-0.22.1.jar:0.22.1]
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.apache.kafka.common.errors.SecurityDisabledException: No Authorizer is configured on the broker
2021-05-11 09:49:04 INFO  AbstractOperator:254 - Reconciliation #1(watch) KafkaUser(kafka/admin-moxiam): KafkaUser admin-moxiam will be checked for creation or modification
2021-05-11 09:49:04 ERROR SimpleAclOperator:81 - Reconciliation failed for user admin-moxiam
io.strimzi.operator.cluster.model.InvalidResourceException: Authorization needs to be enabled in the Kafka custom resource
	at io.strimzi.operator.user.operator.SimpleAclOperator.getAcls(SimpleAclOperator.java:209) ~[io.strimzi.user-operator-0.22.1.jar:0.22.1]
	at io.strimzi.operator.user.operator.SimpleAclOperator.lambda$reconcile$0(SimpleAclOperator.java:73) ~[io.strimzi.user-operator-0.22.1.jar:0.22.1]
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.apache.kafka.common.errors.SecurityDisabledException: No Authorizer is configured on the broker
2021-05-11 09:49:04 ERROR AbstractOperator:275 - Reconciliation #1(watch) KafkaUser(kafka/admin-moxiam): createOrUpdate failed
io.strimzi.operator.cluster.model.InvalidResourceException: Authorization needs to be enabled in the Kafka custom resource
	at io.strimzi.operator.user.operator.SimpleAclOperator.getAcls(SimpleAclOperator.java:209) ~[io.strimzi.user-operator-0.22.1.jar:0.22.1]
	at io.strimzi.operator.user.operator.SimpleAclOperator.lambda$reconcile$0(SimpleAclOperator.java:73) ~[io.strimzi.user-operator-0.22.1.jar:0.22.1]
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.apache.kafka.common.errors.SecurityDisabledException: No Authorizer is configured on the broker
2021-05-11 09:49:04 WARN  AbstractOperator:508 - Reconciliation #1(watch) KafkaUser(kafka/admin-moxiam): Failed to reconcile
io.strimzi.operator.cluster.model.InvalidResourceException: Authorization needs to be enabled in the Kafka custom resource
	at io.strimzi.operator.user.operator.SimpleAclOperator.getAcls(SimpleAclOperator.java:209) ~[io.strimzi.user-operator-0.22.1.jar:0.22.1]
	at io.strimzi.operator.user.operator.SimpleAclOperator.lambda$reconcile$0(SimpleAclOperator.java:73) ~[io.strimzi.user-operator-0.22.1.jar:0.22.1]
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.apache.kafka.common.errors.SecurityDisabledException: No Authorizer is configured on the broker
2021-05-11 09:49:39 INFO  UserOperator:66 - Triggering periodic reconciliation for namespace kafka...
2021-05-11 09:49:39 INFO  AbstractOperator:254 - Reconciliation #2(timer) KafkaUser(kafka/admin-moxiam): KafkaUser admin-moxiam will be checked for creation or modification
2021-05-11 09:49:39 ERROR SimpleAclOperator:81 - Reconciliation failed for user admin-moxiam
io.strimzi.operator.cluster.model.InvalidResourceException: Authorization needs to be enabled in the Kafka custom resource
	at io.strimzi.operator.user.operator.SimpleAclOperator.getAcls(SimpleAclOperator.java:209) ~[io.strimzi.user-operator-0.22.1.jar:0.22.1]
	at io.strimzi.operator.user.operator.SimpleAclOperator.lambda$reconcile$0(SimpleAclOperator.java:73) ~[io.strimzi.user-operator-0.22.1.jar:0.22.1]
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]

[scholzj]

You can either remove the authorization from the KafkaUser or enable authorization in the broker in .spec.kafka:

authorization:
  type: simple

But I'm not sure it explains why the KafkaUser had no status or why the secret was not created 🤔

[sknot-rh]

Can you provide you Kafka resource? It seems you are missing

spec:
  kafka:
    authorization:
      type: simple

[Hemant0601]

Do i have to install Kafka ACL authorizer seperately ?

[sknot-rh]

spec:
  kafka:
    authorization:
      type: simple
      # rest of your cr

Sorry, this should be in Kafka resource.

[Hemant0601]

I think this worked :) Thanks

[Hemant0601]

Can you suggest what are the changes I need to do to connect to my brokers using scram-sha-512 from a client outside the cluster.
I am getting SSL protocol wrong error.

[scholzj]

I think the config looks correct. But the loadbalancer listener has TLS encryption on. So you still need to configure TLS encryption to connect even with the SCRAM-SHA client. If you do not want to use TLS at all, just change tls: true to tls: false.

[Hemant0601]

Actually I need the TLS encryption so I want to keep it true. For authentication I want to use scram-sha-512. I am able to create a user with scram-sha-512. Now I want to connect to the remote brokers, but when I connect it gives me protocol error. I am using kafkcat to test my connection. I am not sure what I am doing wrong.

…

On Tue, 11 May, 2021, 9:15 pm Jakub Scholz, ***@***.***> wrote: I think the config looks correct. But the loadbalancer listener has TLS encryption on. So you still need to configure TLS encryption to connect even with the SCRAM-SHA client. If you do not want to use TLS at all, just change tls: true to tls: false. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4941 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKBWYXA5G2UFZJA364YAZTTNFGLBANCNFSM44USQ7OQ> .

[Hemant0601]

This is basically referring to the local certificate file. Seems fine here

[scholzj]

But you have a space after the = sign. That certainly seems unusual.

[Hemant0601]

Changed it but still same.

[Hemant0601]

I think there was some issue in the password. When I decoded it from kubectl it gave me an extra % sign.
It seems working without TLS : true now.
However, SSL using certificate is still throwing error, saying certificate used is invalid. Cannot authenticate it.

[SupriyaDeepk]

hi @scholzj We have also seen the same error where, we tried creating the Kafkauser, but we are not able to see the secret created automatically, we wanted the user to be created for connecting to the Kafka.