Unable to deploy TLS+ SCRAM using Strimzi

[rohanraz91]

While deploying kafka with TLS+ SCRAM using Strimzi I am not able to connect to KafkaStore using AdminClient configs and the SchemaRegistry fails to start with following error:

[2021-07-21 12:56:46,504] ERROR Error starting the schema registry (io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication)
io.confluent.kafka.schemaregistry.exceptions.SchemaRegistryInitializationException: Error initializing kafka store while initializing schema registry
        at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.init(KafkaSchemaRegistry.java:291)
        at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.initSchemaRegistry(SchemaRegistryRestApplication.java:75)
        at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.configureBaseApplication(SchemaRegistryRestApplication.java:90)
        at io.confluent.rest.Application.configureHandler(Application.java:217)
        at io.confluent.rest.ApplicationServer.doStart(ApplicationServer.java:187)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:43)
Caused by: io.confluent.kafka.schemaregistry.storage.exceptions.StoreInitializationException: Failed trying to create or validate schema topic configuration
        at io.confluent.kafka.schemaregistry.storage.KafkaStore.createOrVerifySchemaTopic(KafkaStore.java:182)
        at io.confluent.kafka.schemaregistry.storage.KafkaStore.init(KafkaStore.java:119)
        at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.init(KafkaSchemaRegistry.java:289)
        ... 6 more
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TopicAuthorizationException: Authorization failed.
        at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
        at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
        at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104)
        at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272)
        at io.confluent.kafka.schemaregistry.storage.KafkaStore.createSchemaTopic(KafkaStore.java:221)
        at io.confluent.kafka.schemaregistry.storage.KafkaStore.createOrVerifySchemaTopic(KafkaStore.java:174)
        ... 8 more
Caused by: org.apache.kafka.common.errors.TopicAuthorizationException: Authorization failed.

Not sure whether the kafka yaml or the SchemaRegistry is missing any other configs. Please Advice, sharing yaml for Kafka and SchemaRegistry.

kafka-ssl-scram.txt
SchemaRegistry.txt

[scholzj]

It says TopicAuthorizationException: Authorization failed. So you will need to investigate what ACL rights does the Schema Registry need and give them to its user.

[rohanraz91]

In kafka yaml(shared) we use

apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaUser
metadata:
  name: kafka-user
  labels:
    strimzi.io/cluster: odf-cluster
spec:
  authentication:
    type: scram-sha-512
  authorization:
    type: simple
    acls:
      # Example consumer Acls for topic lines using consumer group secure-group
      - resource:
          type: topic
          name: lines
          patternType: literal
        operation: Read
        host: "*"
      - resource:
          type: topic
          name: lines
          patternType: literal
        operation: Describe
        host: "*"
      - resource:
          type: group
          name: secure-group
          patternType: literal
        operation: Read
        host: "*"
      - resource:
          type: topic
          name: lines
          patternType: literal
        operation: Write
        host: "*"

The same is being used by schema registry yaml(shared) under section sasl:

sasl:
  configPath: "/etc/kafka-config"
  scram:
    enabled: true
    init:
      image: "xyz.com/confluentinc/cp-schema-registry"
      imageTag: "5.5.0"
      imagePullPolicy: "IfNotPresent"
    clientUser: "kafka-user"
    useExistingSecret:
      clientPassword:
        secretKeyRef:
          name: "kafka-user"
          key: password

Is this not the correct way?

[scholzj]

The code is not formatted properly, so it is hard to say what you actually have in the KafkaUser resource. But I anyway have absolutely no idea what ACL rights does Confluent Schema Registry require. Normally, with the ACL rights, the error on the broker side contains more details including what exactly is missing. So you can check that.

[rohanraz91]

In kafka logs I see no errors but only below info :

2021-07-21 14:33:47,010 INFO Principal = User:kafka-user is Denied Operation = Create from host = 172.17.0.7 on resource = Cluster:LITERAL:kafka-cluster (kafka.authorizer.logger) [data-plane-kafka-request-handler-2]
2021-07-21 14:33:47,011 INFO Principal = User:kafka-user is Denied Operation = Create from host = 172.17.0.7 on resource = Topic:LITERAL:_schemas (kafka.authorizer.logger) [data-plane-kafka-request-handler-2]
2021-07-21 14:33:47,011 INFO Principal = User:kafka-user is Denied Operation = DescribeConfigs from host = 172.17.0.7 on resource = Topic:LITERAL:_schemas (kafka.authorizer.logger) [data-plane-kafka-request-handler-2]
2021-07-21 14:34:38,222 INFO Principal = User:kafka-user is Denied Operation = Create from host = 172.17.0.8 on resource = Cluster:LITERAL:kafka-cluster (kafka.authorizer.logger) [data-plane-kafka-request-handler-0]
2021-07-21 14:34:38,223 INFO Principal = User:kafka-user is Denied Operation = Create from host = 172.17.0.8 on resource = Topic:LITERAL:connect-cluster-offsets (kafka.authorizer.logger) [data-plane-kafka-request-handler-0]
2021-07-21 14:34:38,223 INFO Principal = User:kafka-user is Denied Operation = DescribeConfigs from host = 172.17.0.8 on resource = Topic:LITERAL:connect-cluster-offsets (kafka.authorizer.logger) [data-plane-kafka-request-handler-0]
2021-07-21 14:34:39,316 INFO Principal = User:kafka-user is Denied Operation = Describe from host = 172.17.0.1 on resource = Topic:LITERAL:connect-cluster-offsets (kafka.authorizer.logger) [data-plane-kafka-request-handler-1]
2021-07-21 14:34:48,241 INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager) [group-metadata-manager-0]

[scholzj]

So that should tell you what ACL rights are you missing - you will need to add those (or set the user as super user in the Kafka CR).

[rohanraz91]

thank you for the inputs 👍