tls-external type not available in 0.27 version of the operator? #6117

hari819 · 2021-12-28T13:40:51Z

hari819
Dec 28, 2021

I wanted to provide my own user certificate for mTLS , so since the type "tls-external" is not available in 0.22.1 version which i was running before , i started a clean setup of 0.27.0 ,

https://strimzi.io/docs/operators/in-development/full/using.html#tls_client_authentication_using_a_certificate_issued_outside_the_user_operator

but i see that the kafka CRD is missing this type "tls-external" ,

                           type:
                            type: string
                            enum:
                              - tls
                              - scram-sha-512
                              - oauth

has anyone tried this on latest version or does it work in 0.26 or 0.25 version ?

hari819 · 2021-12-28T13:50:39Z

hari819
Dec 28, 2021
Author

and i get this error while deploying the cluster ,

invalid: spec.kafka.listeners.authentication.type: Unsupported value: "tls-external": supported values: "tls", "scram-sha-512", "oauth"

#https://strimzi.io/docs/operators/latest/full/using.html#overview-str
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka
spec:
  kafka:
    version: 2.8.1
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: external
        port: 9094
        type: ingress
        tls: true
        authentication:
          type: tls-external
        configuration:
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: tls-secret-kafka

0 replies

hari819 · 2021-12-28T13:55:29Z

hari819
Dec 28, 2021
Author

got it , this has to be used in the KafkaUser , not in Kafka ,

4 replies

scholzj Dec 28, 2021
Maintainer

Correct, this is only in the KafkaUser. If you want to use your custom CA for Kafka clients for mTLS, you can configure a custom Clients CA.

hari819 Dec 28, 2021
Author

thankyou @scholzj ,
not custom CA ,but only provide a user certificate by generating a CSR and having CN=username,

scholzj Dec 28, 2021
Maintainer

But you understand that you would still need to sign the CSR with the Clients CA, right?

hari819 Dec 28, 2021
Author

yes i want to use the generated clients-ca to sign the CSR,

hari819 · 2021-12-30T08:41:52Z

hari819
Dec 30, 2021
Author

@scholzj , i have used [(https://github.com/scholzj/strimzi-custom-ca-test)] , created cluster , client certificates , after that i creaated user.crt and user.key which were signed by client CA , but i see that the userOperator is deleting my secret which i created to support type "tls-external" ,

+ exec /usr/bin/tini -w -e 143 -- java -Dlog4j2.configurationFile=file:/opt/user-operator/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom --illegal-access=deny -classpath lib/io.strimzi.user-operator-0.27.0.jar:lib/io.netty.netty-common-4.1.71.Final.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.netty.netty-buffer-4.1.71.Final.jar:lib/io.fabric8.openshift-model-operatorhub-5.10.1.jar:lib/io.fabric8.kubernetes-model-scheduling-5.10.1.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/io.fabric8.kubernetes-model-extensions-5.10.1.jar:lib/io.fabric8.kubernetes-model-discovery-5.10.1.jar:lib/io.netty.netty-codec-http2-4.1.71.Final.jar:lib/io.fabric8.kubernetes-model-events-5.10.1.jar:lib/org.apache.logging.log4j.log4j-core-2.17.0.jar:lib/org.apache.kafka.kafka-clients-3.0.0.jar:lib/io.fabric8.kubernetes-model-core-5.10.1.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.11.3.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/io.fabric8.openshift-model-5.10.1.jar:lib/io.fabric8.kubernetes-model-networking-5.10.1.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/io.fabric8.kubernetes-model-certificates-5.10.1.jar:lib/io.strimzi.crd-annotations-0.27.0.jar:lib/io.fabric8.kubernetes-model-node-5.10.1.jar:lib/org.hdrhistogram.HdrHistogram-2.1.10.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.11.3.jar:lib/io.fabric8.kubernetes-model-common-5.10.1.jar:lib/io.fabric8.openshift-model-miscellaneous-5.10.1.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/io.netty.netty-handler-4.1.71.Final.jar:lib/io.fabric8.kubernetes-model-apiextensions-5.10.1.jar:lib/io.fabric8.openshift-model-whereabouts-5.10.1.jar:lib/io.fabric8.kubernetes-client-5.10.1.jar:lib/io.netty.netty-codec-4.1.71.Final.jar:lib/io.fabric8.openshift-model-machineconfig-5.10.1.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.5.jar:lib/io.fabric8.kubernetes-model-metrics-5.10.1.jar:lib/io.netty.netty-codec-http-4.1.71.Final.jar:lib/io.fabric8.openshift-model-storageversionmigrator-5.10.1.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.strimzi.operator-common-0.27.0.jar:lib/org.xerial.snappy.snappy-java-1.1.8.1.jar:lib/com.fasterxml.jackson.core.jackson-core-2.11.3.jar:lib/io.fabric8.kubernetes-model-apps-5.10.1.jar:lib/org.yaml.snakeyaml-1.26.jar:lib/io.fabric8.kubernetes-model-admissionregistration-5.10.1.jar:lib/io.netty.netty-resolver-dns-4.1.71.Final.jar:lib/io.fabric8.openshift-model-monitoring-5.10.1.jar:lib/io.fabric8.kubernetes-model-storageclass-5.10.1.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/io.fabric8.openshift-model-operator-5.10.1.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.netty.netty-codec-socks-4.1.71.Final.jar:lib/io.netty.netty-handler-proxy-4.1.71.Final.jar:lib/io.netty.netty-resolver-4.1.71.Final.jar:lib/io.fabric8.openshift-model-clusterautoscaling-5.10.1.jar:lib/io.strimzi.certificate-manager-0.27.0.jar:lib/io.fabric8.kubernetes-model-flowcontrol-5.10.1.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/io.fabric8.kubernetes-model-coordination-5.10.1.jar:lib/org.apache.logging.log4j.log4j-api-2.17.0.jar:lib/io.vertx.vertx-micrometer-metrics-4.2.1.jar:lib/io.fabric8.openshift-model-tuned-5.10.1.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.strimzi.api-0.27.0.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.0.jar:lib/io.fabric8.kubernetes-model-batch-5.10.1.jar:lib/io.netty.netty-codec-dns-4.1.71.Final.jar:lib/io.fabric8.kubernetes-model-autoscaling-5.10.1.jar:lib/io.fabric8.openshift-client-5.10.1.jar:lib/io.fabric8.openshift-model-console-5.10.1.jar:lib/io.fabric8.openshift-model-machine-5.10.1.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/com.github.luben.zstd-jni-1.5.0-2.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.13.0.jar:lib/io.netty.netty-tcnative-classes-2.0.46.Final.jar:lib/io.netty.netty-transport-4.1.71.Final.jar:lib/io.vertx.vertx-core-4.2.1.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/io.fabric8.kubernetes-model-rbac-5.10.1.jar:lib/io.fabric8.kubernetes-model-policy-5.10.1.jar io.strimzi.operator.user.Main
2021-12-30 08:05:51 INFO  Main:51 - UserOperator 0.27.0 is starting
2021-12-30 08:05:51 DEBUG LoggerFactory: - Using io.vertx.core.logging.SLF4JLogDelegateFactory
2021-12-30 08:05:51 DEBUG InternalLoggerFactory:61 - Using SLF4J as the default logging framework
2021-12-30 08:05:51 DEBUG InternalLoggerFactory:63 - Using SLF4J as the default logging framework
2021-12-30 08:05:51 DEBUG PlatformDependent0:460 - -Dio.netty.noUnsafe: false
2021-12-30 08:05:51 DEBUG PlatformDependent0:954 - Java version: 11
2021-12-30 08:05:51 DEBUG PlatformDependent0:135 - sun.misc.Unsafe.theUnsafe: available
2021-12-30 08:05:51 DEBUG PlatformDependent0:159 - sun.misc.Unsafe.copyMemory: available
2021-12-30 08:05:51 DEBUG PlatformDependent0:202 - java.nio.Buffer.address: available
2021-12-30 08:05:51 DEBUG PlatformDependent0:285 - direct buffer constructor: unavailable: Reflective setAccessible(true) disabled
2021-12-30 08:05:51 DEBUG PlatformDependent0:350 - java.nio.Bits.unaligned: available, true
2021-12-30 08:05:51 DEBUG PlatformDependent0:417 - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable: class io.netty.util.internal.PlatformDependent0$6 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module @60856961
2021-12-30 08:05:51 DEBUG PlatformDependent0:446 - java.nio.DirectByteBuffer.<init>(long, int): unavailable
2021-12-30 08:05:51 DEBUG PlatformDependent:1116 - sun.misc.Unsafe: available
2021-12-30 08:05:51 DEBUG PlatformDependent:1218 - maxDirectMemory: 26429227008 bytes (maybe)
2021-12-30 08:05:51 DEBUG PlatformDependent:1237 - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
2021-12-30 08:05:51 DEBUG PlatformDependent:1316 - -Dio.netty.bitMode: 64 (sun.arch.data.model)
2021-12-30 08:05:51 DEBUG PlatformDependent:178 - -Dio.netty.maxDirectMemory: -1 bytes
2021-12-30 08:05:51 DEBUG PlatformDependent:185 - -Dio.netty.uninitializedArrayAllocationThreshold: -1
2021-12-30 08:05:51 DEBUG CleanerJava9:71 - java.nio.ByteBuffer.cleaner(): available
2021-12-30 08:05:51 DEBUG PlatformDependent:205 - -Dio.netty.noPreferDirect: false
2021-12-30 08:05:51 DEBUG ResourceLeakDetector:129 - -Dio.netty.leakDetection.level: simple
2021-12-30 08:05:51 DEBUG ResourceLeakDetector:130 - -Dio.netty.leakDetection.targetRecords: 4
2021-12-30 08:05:51 DEBUG MultithreadEventLoopGroup:44 - -Dio.netty.eventLoopThreads: 2
2021-12-30 08:05:51 DEBUG InternalThreadLocalMap:83 - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
2021-12-30 08:05:51 DEBUG InternalThreadLocalMap:86 - -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
2021-12-30 08:05:51 DEBUG NioEventLoop:109 - -Dio.netty.noKeySetOptimization: false
2021-12-30 08:05:51 DEBUG NioEventLoop:110 - -Dio.netty.selectorAutoRebuildThreshold: 512
2021-12-30 08:05:51 DEBUG PlatformDependent:967 - org.jctools-core.MpscChunkedArrayQueue: available
2021-12-30 08:05:51 DEBUG DefaultDnsServerAddressStreamProvider:80 - Default DNS servers: [/10.222.0.10:53] (sun.net.dns.ResolverConfiguration)
2021-12-30 08:05:51 DEBUG NetUtil:135 - -Djava.net.preferIPv4Stack: false
2021-12-30 08:05:51 DEBUG NetUtil:136 - -Djava.net.preferIPv6Addresses: false
2021-12-30 08:05:51 DEBUG NetUtilInitializations:129 - Loopback interface: lo (lo, 127.0.0.1)
2021-12-30 08:05:51 DEBUG NetUtil:169 - /proc/sys/net/core/somaxconn: 4096
2021-12-30 08:05:51 DEBUG Config:542 - Trying to configure client from Kubernetes config...
2021-12-30 08:05:51 DEBUG Config:548 - Did not find Kubernetes config at: [/home/strimzi/.kube/config]. Ignoring.
2021-12-30 08:05:51 DEBUG Config:471 - Trying to configure client from service account...
2021-12-30 08:05:51 DEBUG Config:479 - Found service account host and port: 10.222.0.1:443
2021-12-30 08:05:51 DEBUG Config:485 - Found service account ca cert at: [/var/run/secrets/kubernetes.io/serviceaccount/ca.crt}].
2021-12-30 08:05:51 DEBUG Config:492 - Found service account token at: [/var/run/secrets/kubernetes.io/serviceaccount/token].
2021-12-30 08:05:51 DEBUG Config:748 - Trying to configure client namespace from Kubernetes service account namespace path...
2021-12-30 08:05:51 DEBUG Config:753 - Found service account namespace at: [/var/run/secrets/kubernetes.io/serviceaccount/namespace].
2021-12-30 08:05:52 INFO  Util:303 - Using config:
	PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
	container: oci
	STRIMZI_CA_RENEWAL: 30
	KAFKA_ZOOKEEPER_CLIENT_PORT: tcp://10.223.34.106:2181
	KAFKA_KAFKA_BOOTSTRAP_SERVICE_PORT_TCP_REPLICATION: 9091
	KAFKA_ZOOKEEPER_CLIENT_PORT_2181_TCP_PORT: 2181
	KAFKA_KAFKA_1_PORT_9094_TCP: tcp://10.223.237.237:9094
	STRIMZI_LABELS: strimzi.io/cluster=kafka
	KAFKA_KAFKA_BOOTSTRAP_PORT_9092_TCP_PROTO: tcp
	KAFKA_KAFKA_2_PORT_9094_TCP_PROTO: tcp
	JAVA_OPTS:  -Dlog4j2.configurationFile=file:/opt/user-operator/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom  --illegal-access=deny
	KAFKA_KAFKA_2_SERVICE_PORT_TCP_EXTERNAL: 9094
	KAFKA_KAFKA_2_PORT_9094_TCP_ADDR: 10.222.33.188
	KAFKA_KAFKA_0_PORT_9094_TCP_ADDR: 10.223.44.187
	KAFKA_KAFKA_1_PORT: tcp://10.223.237.237:9094
	KAFKA_ZOOKEEPER_CLIENT_PORT_2181_TCP_ADDR: 10.223.34.106
	KAFKA_ZOOKEEPER_CLIENT_SERVICE_HOST: 10.223.34.106
	KAFKA_KAFKA_1_SERVICE_PORT_TCP_EXTERNAL: 9094
	KAFKA_KAFKA_1_SERVICE_HOST: 10.223.237.237
	STRIMZI_GC_LOG_ENABLED: false
	KAFKA_KAFKA_BOOTSTRAP_PORT_9092_TCP_ADDR: 10.222.133.199
	STRIMZI_EO_KEY_SECRET_NAME: kafka-entity-operator-certs
	STRIMZI_HOME: /opt/strimzi
	PWD: /opt/strimzi
	KUBERNETES_PORT_443_TCP: tcp://10.222.0.1:443
	JAVA_MAIN: io.strimzi.operator.user.Main
	STRIMZI_SECRET_PREFIX: 
	KAFKA_KAFKA_EXTERNAL_BOOTSTRAP_SERVICE_HOST: 10.223.225.60
	STRIMZI_VERSION: 0.27.0
	STRIMZI_KAFKA_BOOTSTRAP_SERVERS: kafka-kafka-bootstrap:9091
	KAFKA_KAFKA_1_PORT_9094_TCP_PROTO: tcp
	STRIMZI_NAMESPACE: kafka-mtls
	TINI_SHA256_S390X: 931b70a182af879ca249ae9de87ef68423121b38d235c78997fafc680ceab32d
	STRIMZI_CA_VALIDITY: 365
	KAFKA_KAFKA_EXTERNAL_BOOTSTRAP_PORT_9094_TCP_ADDR: 10.223.225.60
	KAFKA_KAFKA_0_PORT_9094_TCP: tcp://10.223.44.187:9094
	TINI_SHA256_AMD64: 93dcc18adc78c65a028a84799ecf8ad40c936fdfc5f2a57b1acda5a8117fa82c
	KAFKA_KAFKA_EXTERNAL_BOOTSTRAP_PORT_9094_TCP: tcp://10.223.225.60:9094
	KUBERNETES_SERVICE_PORT_HTTPS: 443
	KAFKA_KAFKA_BOOTSTRAP_SERVICE_HOST: 10.222.133.199
	KAFKA_ZOOKEEPER_CLIENT_SERVICE_PORT: 2181
	SHLVL: 0
	KAFKA_KAFKA_0_PORT_9094_TCP_PROTO: tcp
	KAFKA_KAFKA_0_PORT: tcp://10.223.44.187:9094
	KAFKA_KAFKA_BOOTSTRAP_SERVICE_PORT_TCP_CLIENTS: 9092
	KAFKA_KAFKA_EXTERNAL_BOOTSTRAP_SERVICE_PORT_TCP_EXTERNAL: 9094
	KAFKA_KAFKA_BOOTSTRAP_SERVICE_PORT: 9091
	KUBERNETES_PORT: tcp://10.222.0.1:443
	JAVA_HOME: /usr/lib/jvm/jre-11
	KAFKA_KAFKA_BOOTSTRAP_PORT_9091_TCP: tcp://10.222.133.199:9091
	KAFKA_KAFKA_1_PORT_9094_TCP_ADDR: 10.223.237.237
	STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
	KUBERNETES_SERVICE_HOST: 10.222.0.1
	JAVA_CLASSPATH: lib/io.strimzi.user-operator-0.27.0.jar:lib/io.netty.netty-common-4.1.71.Final.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.netty.netty-buffer-4.1.71.Final.jar:lib/io.fabric8.openshift-model-operatorhub-5.10.1.jar:lib/io.fabric8.kubernetes-model-scheduling-5.10.1.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/io.fabric8.kubernetes-model-extensions-5.10.1.jar:lib/io.fabric8.kubernetes-model-discovery-5.10.1.jar:lib/io.netty.netty-codec-http2-4.1.71.Final.jar:lib/io.fabric8.kubernetes-model-events-5.10.1.jar:lib/org.apache.logging.log4j.log4j-core-2.17.0.jar:lib/org.apache.kafka.kafka-clients-3.0.0.jar:lib/io.fabric8.kubernetes-model-core-5.10.1.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.11.3.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/io.fabric8.openshift-model-5.10.1.jar:lib/io.fabric8.kubernetes-model-networking-5.10.1.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/io.fabric8.kubernetes-model-certificates-5.10.1.jar:lib/io.strimzi.crd-annotations-0.27.0.jar:lib/io.fabric8.kubernetes-model-node-5.10.1.jar:lib/org.hdrhistogram.HdrHistogram-2.1.10.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.11.3.jar:lib/io.fabric8.kubernetes-model-common-5.10.1.jar:lib/io.fabric8.openshift-model-miscellaneous-5.10.1.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/io.netty.netty-handler-4.1.71.Final.jar:lib/io.fabric8.kubernetes-model-apiextensions-5.10.1.jar:lib/io.fabric8.openshift-model-whereabouts-5.10.1.jar:lib/io.fabric8.kubernetes-client-5.10.1.jar:lib/io.netty.netty-codec-4.1.71.Final.jar:lib/io.fabric8.openshift-model-machineconfig-5.10.1.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.5.jar:lib/io.fabric8.kubernetes-model-metrics-5.10.1.jar:lib/io.netty.netty-codec-http-4.1.71.Final.jar:lib/io.fabric8.openshift-model-storageversionmigrator-5.10.1.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.strimzi.operator-common-0.27.0.jar:lib/org.xerial.snappy.snappy-java-1.1.8.1.jar:lib/com.fasterxml.jackson.core.jackson-core-2.11.3.jar:lib/io.fabric8.kubernetes-model-apps-5.10.1.jar:lib/org.yaml.snakeyaml-1.26.jar:lib/io.fabric8.kubernetes-model-admissionregistration-5.10.1.jar:lib/io.netty.netty-resolver-dns-4.1.71.Final.jar:lib/io.fabric8.openshift-model-monitoring-5.10.1.jar:lib/io.fabric8.kubernetes-model-storageclass-5.10.1.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/io.fabric8.openshift-model-operator-5.10.1.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.netty.netty-codec-socks-4.1.71.Final.jar:lib/io.netty.netty-handler-proxy-4.1.71.Final.jar:lib/io.netty.netty-resolver-4.1.71.Final.jar:lib/io.fabric8.openshift-model-clusterautoscaling-5.10.1.jar:lib/io.strimzi.certificate-manager-0.27.0.jar:lib/io.fabric8.kubernetes-model-flowcontrol-5.10.1.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/io.fabric8.kubernetes-model-coordination-5.10.1.jar:lib/org.apache.logging.log4j.log4j-api-2.17.0.jar:lib/io.vertx.vertx-micrometer-metrics-4.2.1.jar:lib/io.fabric8.openshift-model-tuned-5.10.1.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.strimzi.api-0.27.0.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.0.jar:lib/io.fabric8.kubernetes-model-batch-5.10.1.jar:lib/io.netty.netty-codec-dns-4.1.71.Final.jar:lib/io.fabric8.kubernetes-model-autoscaling-5.10.1.jar:lib/io.fabric8.openshift-client-5.10.1.jar:lib/io.fabric8.openshift-model-console-5.10.1.jar:lib/io.fabric8.openshift-model-machine-5.10.1.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/com.github.luben.zstd-jni-1.5.0-2.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.13.0.jar:lib/io.netty.netty-tcnative-classes-2.0.46.Final.jar:lib/io.netty.netty-transport-4.1.71.Final.jar:lib/io.vertx.vertx-core-4.2.1.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/io.fabric8.kubernetes-model-rbac-5.10.1.jar:lib/io.fabric8.kubernetes-model-policy-5.10.1.jar
	KAFKA_KAFKA_EXTERNAL_BOOTSTRAP_PORT: tcp://10.223.225.60:9094
	STRIMZI_CLUSTER_CA_CERT_SECRET_NAME: kafka-cluster-ca-cert
	KAFKA_KAFKA_BOOTSTRAP_PORT: tcp://10.222.133.199:9091
	KAFKA_KAFKA_2_SERVICE_HOST: 10.222.33.188
	KAFKA_KAFKA_BOOTSTRAP_PORT_9092_TCP: tcp://10.222.133.199:9092
	KAFKA_ZOOKEEPER_CLIENT_PORT_2181_TCP: tcp://10.223.34.106:2181
	TINI_VERSION: v0.19.0
	KAFKA_ZOOKEEPER_CLIENT_SERVICE_PORT_TCP_CLIENTS: 2181
	KAFKA_KAFKA_0_SERVICE_HOST: 10.223.44.187
	KAFKA_KAFKA_BOOTSTRAP_PORT_9091_TCP_PORT: 9091
	KUBERNETES_PORT_443_TCP_ADDR: 10.222.0.1
	KAFKA_KAFKA_1_SERVICE_PORT: 9094
	KAFKA_KAFKA_EXTERNAL_BOOTSTRAP_PORT_9094_TCP_PORT: 9094
	KAFKA_KAFKA_2_PORT: tcp://10.222.33.188:9094
	KAFKA_KAFKA_2_PORT_9094_TCP: tcp://10.222.33.188:9094
	KAFKA_KAFKA_2_SERVICE_PORT: 9094
	KUBERNETES_PORT_443_TCP_PROTO: tcp
	STRIMZI_CA_CERT_NAME: kafka-clients-ca-cert
	KAFKA_ZOOKEEPER_CLIENT_PORT_2181_TCP_PROTO: tcp
	STRIMZI_CA_NAMESPACE: kafka-mtls
	KUBERNETES_SERVICE_PORT: 443
	KAFKA_KAFKA_BOOTSTRAP_PORT_9091_TCP_PROTO: tcp
	KAFKA_KAFKA_0_SERVICE_PORT_TCP_EXTERNAL: 9094
	TINI_SHA256_ARM64: 07952557df20bfd2a95f9bef198b445e006171969499a1d361bd9e6f8e5e0e81
	KAFKA_KAFKA_2_PORT_9094_TCP_PORT: 9094
	TINI_SHA256_PPC64LE: 3f658420974768e40810001a038c29d003728c5fe86da211cff5059e48cfdfde
	KAFKA_KAFKA_BOOTSTRAP_PORT_9092_TCP_PORT: 9092
	KAFKA_KAFKA_BOOTSTRAP_PORT_9091_TCP_ADDR: 10.222.133.199
	HOSTNAME: kafka-entity-operator-d89d9985b-d6hp8
	KAFKA_KAFKA_1_PORT_9094_TCP_PORT: 9094
	KAFKA_KAFKA_EXTERNAL_BOOTSTRAP_SERVICE_PORT: 9094
	KAFKA_KAFKA_0_PORT_9094_TCP_PORT: 9094
	STRIMZI_CA_KEY_NAME: kafka-clients-ca
	KUBERNETES_PORT_443_TCP_PORT: 443
	KAFKA_KAFKA_0_SERVICE_PORT: 9094
	KAFKA_KAFKA_EXTERNAL_BOOTSTRAP_PORT_9094_TCP_PROTO: tcp
	HOME: /home/strimzi
	MALLOC_ARENA_MAX: 2
	STRIMZI_ACLS_ADMIN_API_SUPPORTED: false

2021-12-30 08:05:52 INFO  AdminClientConfig:376 - AdminClientConfig values: 
	bootstrap.servers = [kafka-kafka-bootstrap:9091]
	client.dns.lookup = use_all_dns_ips
	client.id = 
	connections.max.idle.ms = 300000
	default.api.timeout.ms = 40000
	metadata.max.age.ms = 30000
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	receive.buffer.bytes = 65536
	reconnect.backoff.max.ms = 1000
	reconnect.backoff.ms = 50
	request.timeout.ms = 10000
	retries = 3
	retry.backoff.ms = 100
	sasl.client.callback.handler.class = null
	sasl.jaas.config = null
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.min.time.before.relogin = 60000
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	sasl.kerberos.ticket.renew.window.factor = 0.8
	sasl.login.callback.handler.class = null
	sasl.login.class = null
	sasl.login.refresh.buffer.seconds = 300
	sasl.login.refresh.min.period.seconds = 60
	sasl.login.refresh.window.factor = 0.8
	sasl.login.refresh.window.jitter = 0.05
	sasl.mechanism = GSSAPI
	security.protocol = SSL
	security.providers = null
	send.buffer.bytes = 131072
	socket.connection.setup.timeout.max.ms = 30000
	socket.connection.setup.timeout.ms = 10000
	ssl.cipher.suites = null
	ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
	ssl.endpoint.identification.algorithm = https
	ssl.engine.factory.class = null
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.certificate.chain = [hidden]
	ssl.keystore.key = [hidden]
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = PEM
	ssl.protocol = TLSv1.3
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.certificates = [hidden]
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = PEM

2021-12-30 08:05:52 DEBUG AdminMetadataManager:245 - [AdminClient clientId=adminclient-1] Setting bootstrap cluster metadata Cluster(id = null, nodes = [kafka-kafka-bootstrap:9091 (id: -1 rack: null)], partitions = [], controller = null).
2021-12-30 08:05:52 DEBUG DefaultSslEngineFactory:264 - Created SSL context with keystore org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore@578bcd59, truststore org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore@46ad463b, provider SunJSSE.
2021-12-30 08:05:52 INFO  AppInfoParser:119 - Kafka version: 3.0.0
2021-12-30 08:05:52 INFO  AppInfoParser:120 - Kafka commitId: 8cb0a5e9d3441962
2021-12-30 08:05:52 INFO  AppInfoParser:121 - Kafka startTimeMs: 1640851552942
2021-12-30 08:05:52 DEBUG KafkaAdminClient:579 - [AdminClient clientId=adminclient-1] Kafka admin client initialized
2021-12-30 08:05:52 DEBUG KafkaAdminClient:1318 - [AdminClient clientId=adminclient-1] Thread starting
2021-12-30 08:05:52 DEBUG NetworkClient:976 - [AdminClient clientId=adminclient-1] Initiating connection to node kafka-kafka-bootstrap:9091 (id: -1 rack: null) using address kafka-kafka-bootstrap/10.222.133.199
2021-12-30 08:05:52 INFO  UserOperator:44 - Creating UserOperator for namespace kafka-mtls
2021-12-30 08:05:52 INFO  UserOperator:54 - Starting UserOperator for namespace kafka-mtls
2021-12-30 08:05:53 DEBUG Selector:531 - [AdminClient clientId=adminclient-1] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -1
2021-12-30 08:05:53 DEBUG AbstractWatchManager:195 - Watching https://10.222.0.1/apis/kafka.strimzi.io/v1beta2/namespaces/kafka-mtls/kafkausers?labelSelector=strimzi.io%2Fcluster%3Dkafka&watch=true...
2021-12-30 08:05:53 DEBUG WatcherWebSocketListener:50 - WebSocket successfully opened
2021-12-30 08:05:53 INFO  UserOperator:61 - Started operator for KafkaUser kind
2021-12-30 08:05:53 INFO  UserOperator:64 - Setting up periodic reconciliation for namespace kafka-mtls
2021-12-30 08:05:53 DEBUG PooledByteBufAllocator:155 - -Dio.netty.allocator.numHeapArenas: 2
2021-12-30 08:05:53 DEBUG PooledByteBufAllocator:156 - -Dio.netty.allocator.numDirectArenas: 2
2021-12-30 08:05:53 DEBUG PooledByteBufAllocator:158 - -Dio.netty.allocator.pageSize: 8192
2021-12-30 08:05:53 DEBUG PooledByteBufAllocator:163 - -Dio.netty.allocator.maxOrder: 11
2021-12-30 08:05:53 DEBUG PooledByteBufAllocator:167 - -Dio.netty.allocator.chunkSize: 16777216
2021-12-30 08:05:53 DEBUG PooledByteBufAllocator:168 - -Dio.netty.allocator.smallCacheSize: 256
2021-12-30 08:05:53 DEBUG PooledByteBufAllocator:169 - -Dio.netty.allocator.normalCacheSize: 64
2021-12-30 08:05:53 DEBUG PooledByteBufAllocator:170 - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
2021-12-30 08:05:53 DEBUG PooledByteBufAllocator:171 - -Dio.netty.allocator.cacheTrimInterval: 8192
2021-12-30 08:05:53 DEBUG PooledByteBufAllocator:172 - -Dio.netty.allocator.cacheTrimIntervalMillis: 0
2021-12-30 08:05:53 DEBUG PooledByteBufAllocator:173 - -Dio.netty.allocator.useCacheForAllThreads: true
2021-12-30 08:05:53 DEBUG PooledByteBufAllocator:174 - -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023
2021-12-30 08:05:53 DEBUG CustomResource:172 - Calling CustomResource#setKind doesn't do anything because the Kind is computed and shouldn't be changed
2021-12-30 08:05:53 INFO  OperatorWatcher:38 - Reconciliation #1(watch) KafkaUser(kafka-mtls/my-user): KafkaUser my-user in namespace kafka-mtls was ADDED
2021-12-30 08:05:53 DEBUG AbstractOperator:366 - Reconciliation #1(watch) KafkaUser(kafka-mtls/my-user): Try to acquire lock lock::kafka-mtls::KafkaUser::my-user
2021-12-30 08:05:53 DEBUG AbstractOperator:369 - Reconciliation #1(watch) KafkaUser(kafka-mtls/my-user): Lock lock::kafka-mtls::KafkaUser::my-user acquired
2021-12-30 08:05:53 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkausers' with unstable version 'v1beta2'
2021-12-30 08:05:53 DEBUG CustomResource:172 - Calling CustomResource#setKind doesn't do anything because the Kind is computed and shouldn't be changed
2021-12-30 08:05:53 DEBUG DefaultChannelId:79 - -Dio.netty.processId: 9 (auto-detected)
2021-12-30 08:05:53 INFO  AbstractOperator:226 - Reconciliation #1(watch) KafkaUser(kafka-mtls/my-user): KafkaUser my-user will be checked for creation or modification
2021-12-30 08:05:53 DEBUG DefaultChannelId:101 - -Dio.netty.machineId: 8e:31:2f:ff:fe:6d:3a:39 (auto-detected)
2021-12-30 08:05:53 DEBUG KafkaUserOperator:120 - Reconciliation #1(watch) KafkaUser(kafka-mtls/my-user): Updating User my-user in namespace kafka-mtls
**2021-12-30 08:05:53 DEBUG ScramCredentialsOperator:58 - Reconciliation #1(watch) KafkaUser(kafka-mtls/my-user): Deleting SCRAM credentials for user my-user**
2021-12-30 08:05:53 DEBUG ByteBufUtil:87 - -Dio.netty.allocator.type: pooled
2021-12-30 08:05:53 DEBUG ByteBufUtil:96 - -Dio.netty.threadLocalDirectBufferSize: 0
2021-12-30 08:05:53 DEBUG ByteBufUtil:99 - -Dio.netty.maxThreadLocalCharBufferSize: 16384
2021-12-30 08:05:53 DEBUG KafkaAdminClient:1419 - [AdminClient clientId=adminclient-1] Queueing Call(callName=alterUserScramCredentials, deadlineMs=1640851593304, tries=0, nextAllowedTryMs=0) with a timeout 10000 ms from now.
2021-12-30 08:05:53 DEBUG KafkaAdminClient:1419 - [AdminClient clientId=adminclient-1] Queueing Call(callName=describeClientQuotas, deadlineMs=1640851593316, tries=0, nextAllowedTryMs=0) with a timeout 10000 ms from now.
2021-12-30 08:05:53 DEBUG KafkaAdminClient:1419 - [AdminClient clientId=adminclient-1] Queueing Call(callName=describeClientQuotas, deadlineMs=1640851593322, tries=0, nextAllowedTryMs=0) with a timeout 10000 ms from now.
2021-12-30 08:05:53 INFO  UserOperator:112 - UserOperator is now ready (health server listening on 8081)
2021-12-30 08:05:53 INFO  Main:97 - User Operator verticle started in namespace kafka-mtls
2021-12-30 08:05:53 DEBUG AbstractResourceOperator:121 - Reconciliation #1(watch) KafkaUser(kafka-mtls/my-user): Secret kafka-mtls/my-user exist, deleting it
2021-12-30 08:05:53 DEBUG AbstractWatchManager:195 - Watching https://10.222.0.1/api/v1/namespaces/kafka-mtls/secrets?fieldSelector=metadata.name%3Dmy-user&watch=true...
2021-12-30 08:05:53 DEBUG WatcherWebSocketListener:50 - WebSocket successfully opened
2021-12-30 08:05:53 DEBUG ResourceSupport:166 - Reconciliation #1(watch) KafkaUser(kafka-mtls/my-user): Opened watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@49da2d43 for evaluation of observe deletion of Secret kafka-mtls/my-user
2021-12-30 08:05:53 DEBUG NetworkClient:943 - [AdminClient clientId=adminclient-1] Completed connection to node -1. Fetching API versions.
2021-12-30 08:05:53 DEBUG ResourceSupport:196 - Reconciliation #1(watch) KafkaUser(kafka-mtls/my-user): Not yet satisfied: observe deletion of Secret kafka-mtls/my-user
2021-12-30 08:05:53 DEBUG ResourceSupport:176 - Reconciliation #1(watch) KafkaUser(kafka-mtls/my-user): Pre-check is not complete yet, let's wait for the watch: observe deletion of Secret kafka-mtls/my-user
2021-12-30 08:05:53 DEBUG ResourceSupport:196 - Reconciliation #1(watch) KafkaUser(kafka-mtls/my-user): Not yet satisfied: observe deletion of Secret kafka-mtls/my-user
2021-12-30 08:05:53 DEBUG SslTransportLayer:452 - [SslTransportLayer channelId=-1 key=channel=java.nio.channels.SocketChannel[connection-pending remote=kafka-kafka-bootstrap/10.222.133.199:9091], selector=sun.nio.ch.EPollSelectorImpl@571ee568, interestOps=8, readyOps=0] SSL handshake completed successfully with peerHost 'kafka-kafka-bootstrap' peerPort 9091 peerPrincipal 'CN=kafka-kafka, O=io.strimzi' cipherSuite 'TLS_AES_256_GCM_SHA384'
2021-12-30 08:05:53 DEBUG Selector:560 - [AdminClient clientId=adminclient-1] Successfully authenticated with kafka-kafka-bootstrap/10.222.133.199
2021-12-30 08:05:53 DEBUG NetworkClient:957 - [AdminClient clientId=adminclient-1] Initiating API versions fetch from node -1.
2021-12-30 08:05:53 DEBUG NetworkClient:512 - [AdminClient clientId=adminclient-1] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-1, correlationId=0) and timeout 3600000 to node -1: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='3.0.0')
2021-12-30 08:05:53 DEBUG NetworkClient:870 - [AdminClient clientId=adminclient-1] Received API_VERSIONS response from node -1 for request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-1, correlationId=0): ApiVersionsResponseData(errorCode=0, apiKeys=[ApiVersion(apiKey=0, minVersion=0, maxVersion=9), ApiVersion(apiKey=1, minVersion=0, maxVersion=12), ApiVersion(apiKey=2, minVersion=0, maxVersion=6), ApiVersion(apiKey=3, minVersion=0, maxVersion=11), ApiVersion(apiKey=4, minVersion=0, maxVersion=5), ApiVersion(apiKey=5, minVersion=0, maxVersion=3), ApiVersion(apiKey=6, minVersion=0, maxVersion=7), ApiVersion(apiKey=7, minVersion=0, maxVersion=3), ApiVersion(apiKey=8, minVersion=0, maxVersion=8), ApiVersion(apiKey=9, minVersion=0, maxVersion=7), ApiVersion(apiKey=10, minVersion=0, maxVersion=3), ApiVersion(apiKey=11, minVersion=0, maxVersion=7), ApiVersion(apiKey=12, minVersion=0, maxVersion=4), ApiVersion(apiKey=13, minVersion=0, maxVersion=4), ApiVersion(apiKey=14, minVersion=0, maxVersion=5), ApiVersion(apiKey=15, minVersion=0, maxVersion=5), ApiVersion(apiKey=16, minVersion=0, maxVersion=4), ApiVersion(apiKey=17, minVersion=0, maxVersion=1), ApiVersion(apiKey=18, minVersion=0, maxVersion=3), ApiVersion(apiKey=19, minVersion=0, maxVersion=7), ApiVersion(apiKey=20, minVersion=0, maxVersion=6), ApiVersion(apiKey=21, minVersion=0, maxVersion=2), ApiVersion(apiKey=22, minVersion=0, maxVersion=4), ApiVersion(apiKey=23, minVersion=0, maxVersion=4), ApiVersion(apiKey=24, minVersion=0, maxVersion=3), ApiVersion(apiKey=25, minVersion=0, maxVersion=3), ApiVersion(apiKey=26, minVersion=0, maxVersion=3), ApiVersion(apiKey=27, minVersion=0, maxVersion=1), ApiVersion(apiKey=28, minVersion=0, maxVersion=3), ApiVersion(apiKey=29, minVersion=0, maxVersion=2), ApiVersion(apiKey=30, minVersion=0, maxVersion=2), ApiVersion(apiKey=31, minVersion=0, maxVersion=2), ApiVersion(apiKey=32, minVersion=0, maxVersion=4), ApiVersion(apiKey=33, minVersion=0, maxVersion=2), ApiVersion(apiKey=34, minVersion=0, maxVersion=2), ApiVersion(apiKey=35, minVersion=0, maxVersion=2), ApiVersion(apiKey=36, minVersion=0, maxVersion=2), ApiVersion(apiKey=37, minVersion=0, maxVersion=3), ApiVersion(apiKey=38, minVersion=0, maxVersion=2), ApiVersion(apiKey=39, minVersion=0, maxVersion=2), ApiVersion(apiKey=40, minVersion=0, maxVersion=2), ApiVersion(apiKey=41, minVersion=0, maxVersion=2), ApiVersion(apiKey=42, minVersion=0, maxVersion=2), ApiVersion(apiKey=43, minVersion=0, maxVersion=2), ApiVersion(apiKey=44, minVersion=0, maxVersion=1), ApiVersion(apiKey=45, minVersion=0, maxVersion=0), ApiVersion(apiKey=46, minVersion=0, maxVersion=0), ApiVersion(apiKey=47, minVersion=0, maxVersion=0), ApiVersion(apiKey=48, minVersion=0, maxVersion=1), ApiVersion(apiKey=49, minVersion=0, maxVersion=1), ApiVersion(apiKey=50, minVersion=0, maxVersion=0), ApiVersion(apiKey=51, minVersion=0, maxVersion=0), ApiVersion(apiKey=56, minVersion=0, maxVersion=0), ApiVersion(apiKey=57, minVersion=0, maxVersion=0), ApiVersion(apiKey=60, minVersion=0, maxVersion=0), ApiVersion(apiKey=61, minVersion=0, maxVersion=0)], throttleTimeMs=0, supportedFeatures=[], finalizedFeaturesEpoch=0, finalizedFeatures=[])
2021-12-30 08:05:53 DEBUG NetworkClient:912 - [AdminClient clientId=adminclient-1] Node -1 has finalized features epoch: 0, finalized features: [], supported features: [], API versions: (Produce(0): 0 to 9 [usable: 9], Fetch(1): 0 to 12 [usable: 12], ListOffsets(2): 0 to 6 [usable: 6], Metadata(3): 0 to 11 [usable: 11], LeaderAndIsr(4): 0 to 5 [usable: 5], StopReplica(5): 0 to 3 [usable: 3], UpdateMetadata(6): 0 to 7 [usable: 7], ControlledShutdown(7): 0 to 3 [usable: 3], OffsetCommit(8): 0 to 8 [usable: 8], OffsetFetch(9): 0 to 7 [usable: 7], FindCoordinator(10): 0 to 3 [usable: 3], JoinGroup(11): 0 to 7 [usable: 7], Heartbeat(12): 0 to 4 [usable: 4], LeaveGroup(13): 0 to 4 [usable: 4], SyncGroup(14): 0 to 5 [usable: 5], DescribeGroups(15): 0 to 5 [usable: 5], ListGroups(16): 0 to 4 [usable: 4], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 3 [usable: 3], CreateTopics(19): 0 to 7 [usable: 7], DeleteTopics(20): 0 to 6 [usable: 6], DeleteRecords(21): 0 to 2 [usable: 2], InitProducerId(22): 0 to 4 [usable: 4], OffsetForLeaderEpoch(23): 0 to 4 [usable: 4], AddPartitionsToTxn(24): 0 to 3 [usable: 3], AddOffsetsToTxn(25): 0 to 3 [usable: 3], EndTxn(26): 0 to 3 [usable: 3], WriteTxnMarkers(27): 0 to 1 [usable: 1], TxnOffsetCommit(28): 0 to 3 [usable: 3], DescribeAcls(29): 0 to 2 [usable: 2], CreateAcls(30): 0 to 2 [usable: 2], DeleteAcls(31): 0 to 2 [usable: 2], DescribeConfigs(32): 0 to 4 [usable: 4], AlterConfigs(33): 0 to 2 [usable: 2], AlterReplicaLogDirs(34): 0 to 2 [usable: 2], DescribeLogDirs(35): 0 to 2 [usable: 2], SaslAuthenticate(36): 0 to 2 [usable: 2], CreatePartitions(37): 0 to 3 [usable: 3], CreateDelegationToken(38): 0 to 2 [usable: 2], RenewDelegationToken(39): 0 to 2 [usable: 2], ExpireDelegationToken(40): 0 to 2 [usable: 2], DescribeDelegationToken(41): 0 to 2 [usable: 2], DeleteGroups(42): 0 to 2 [usable: 2], ElectLeaders(43): 0 to 2 [usable: 2], IncrementalAlterConfigs(44): 0 to 1 [usable: 1], AlterPartitionReassignments(45): 0 [usable: 0], ListPartitionReassignments(46): 0 [usable: 0], OffsetDelete(47): 0 [usable: 0], DescribeClientQuotas(48): 0 to 1 [usable: 1], AlterClientQuotas(49): 0 to 1 [usable: 1], DescribeUserScramCredentials(50): 0 [usable: 0], AlterUserScramCredentials(51): 0 [usable: 0], AlterIsr(56): 0 [usable: 0], UpdateFeatures(57): 0 [usable: 0], DescribeCluster(60): 0 [usable: 0], DescribeProducers(61): 0 [usable: 0], DescribeTransactions(65): UNSUPPORTED, ListTransactions(66): UNSUPPORTED, AllocateProducerIds(67): UNSUPPORTED).
2021-12-30 08:05:53 DEBUG KafkaAdminClient:1162 - [AdminClient clientId=adminclient-1] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to kafka-kafka-bootstrap:9091 (id: -1 rack: null). correlationId=1, timeoutMs=9430
2021-12-30 08:05:53 DEBUG NetworkClient:512 - [AdminClient clientId=adminclient-1] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=11, clientId=adminclient-1, correlationId=1) and timeout 9430 to node -1: MetadataRequestData(topics=[], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false)
2021-12-30 08:05:53 DEBUG NetworkClient:870 - [AdminClient clientId=adminclient-1] Received METADATA response from node -1 for request with header RequestHeader(apiKey=METADATA, apiVersion=11, clientId=adminclient-1, correlationId=1): MetadataResponseData(throttleTimeMs=0, brokers=[MetadataResponseBroker(nodeId=0, host='kafka-kafka-0.kafka-kafka-brokers.kafka-mtls.svc', port=9091, rack=null), MetadataResponseBroker(nodeId=2, host='kafka-kafka-2.kafka-kafka-brokers.kafka-mtls.svc', port=9091, rack=null), MetadataResponseBroker(nodeId=1, host='kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc', port=9091, rack=null)], clusterId='EQcrSilvRPyTDEt9b6pawQ', controllerId=1, topics=[], clusterAuthorizedOperations=-2147483648)
2021-12-30 08:05:53 DEBUG AdminMetadataManager:247 - [AdminClient clientId=adminclient-1] Updating cluster metadata to Cluster(id = EQcrSilvRPyTDEt9b6pawQ, nodes = [kafka-kafka-2.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 2 rack: null), kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 1 rack: null), kafka-kafka-0.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 0 rack: null)], partitions = [], controller = kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 1 rack: null))
2021-12-30 08:05:53 DEBUG NetworkClient:976 - [AdminClient clientId=adminclient-1] Initiating connection to node kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 1 rack: null) using address kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc/10.33.0.19
2021-12-30 08:05:53 DEBUG NetworkClient:976 - [AdminClient clientId=adminclient-1] Initiating connection to node kafka-kafka-0.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 0 rack: null) using address kafka-kafka-0.kafka-kafka-brokers.kafka-mtls.svc/10.38.0.76
2021-12-30 08:05:53 DEBUG Selector:531 - [AdminClient clientId=adminclient-1] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 1
2021-12-30 08:05:53 DEBUG NetworkClient:943 - [AdminClient clientId=adminclient-1] Completed connection to node 1. Fetching API versions.
2021-12-30 08:05:53 DEBUG Selector:531 - [AdminClient clientId=adminclient-1] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 0
2021-12-30 08:05:53 DEBUG NetworkClient:943 - [AdminClient clientId=adminclient-1] Completed connection to node 0. Fetching API versions.
2021-12-30 08:05:53 DEBUG SslTransportLayer:452 - [SslTransportLayer channelId=1 key=channel=java.nio.channels.SocketChannel[connection-pending remote=kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc/10.33.0.19:9091], selector=sun.nio.ch.EPollSelectorImpl@571ee568, interestOps=8, readyOps=0] SSL handshake completed successfully with peerHost 'kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc' peerPort 9091 peerPrincipal 'CN=kafka-kafka, O=io.strimzi' cipherSuite 'TLS_AES_256_GCM_SHA384'
2021-12-30 08:05:53 DEBUG Selector:560 - [AdminClient clientId=adminclient-1] Successfully authenticated with kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc/10.33.0.19
2021-12-30 08:05:53 DEBUG NetworkClient:957 - [AdminClient clientId=adminclient-1] Initiating API versions fetch from node 1.
2021-12-30 08:05:53 DEBUG NetworkClient:512 - [AdminClient clientId=adminclient-1] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-1, correlationId=2) and timeout 3600000 to node 1: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='3.0.0')
2021-12-30 08:05:53 DEBUG SslTransportLayer:452 - [SslTransportLayer channelId=0 key=channel=java.nio.channels.SocketChannel[connection-pending remote=kafka-kafka-0.kafka-kafka-brokers.kafka-mtls.svc/10.38.0.76:9091], selector=sun.nio.ch.EPollSelectorImpl@571ee568, interestOps=8, readyOps=0] SSL handshake completed successfully with peerHost 'kafka-kafka-0.kafka-kafka-brokers.kafka-mtls.svc' peerPort 9091 peerPrincipal 'CN=kafka-kafka, O=io.strimzi' cipherSuite 'TLS_AES_256_GCM_SHA384'
2021-12-30 08:05:53 DEBUG Selector:560 - [AdminClient clientId=adminclient-1] Successfully authenticated with kafka-kafka-0.kafka-kafka-brokers.kafka-mtls.svc/10.38.0.76
2021-12-30 08:05:53 DEBUG NetworkClient:957 - [AdminClient clientId=adminclient-1] Initiating API versions fetch from node 0.
2021-12-30 08:05:53 DEBUG NetworkClient:512 - [AdminClient clientId=adminclient-1] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-1, correlationId=3) and timeout 3600000 to node 0: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='3.0.0')
2021-12-30 08:05:53 DEBUG NetworkClient:870 - [AdminClient clientId=adminclient-1] Received API_VERSIONS response from node 1 for request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-1, correlationId=2): ApiVersionsResponseData(errorCode=0, apiKeys=[ApiVersion(apiKey=0, minVersion=0, maxVersion=9), ApiVersion(apiKey=1, minVersion=0, maxVersion=12), ApiVersion(apiKey=2, minVersion=0, maxVersion=6), ApiVersion(apiKey=3, minVersion=0, maxVersion=11), ApiVersion(apiKey=4, minVersion=0, maxVersion=5), ApiVersion(apiKey=5, minVersion=0, maxVersion=3), ApiVersion(apiKey=6, minVersion=0, maxVersion=7), ApiVersion(apiKey=7, minVersion=0, maxVersion=3), ApiVersion(apiKey=8, minVersion=0, maxVersion=8), ApiVersion(apiKey=9, minVersion=0, maxVersion=7), ApiVersion(apiKey=10, minVersion=0, maxVersion=3), ApiVersion(apiKey=11, minVersion=0, maxVersion=7), ApiVersion(apiKey=12, minVersion=0, maxVersion=4), ApiVersion(apiKey=13, minVersion=0, maxVersion=4), ApiVersion(apiKey=14, minVersion=0, maxVersion=5), ApiVersion(apiKey=15, minVersion=0, maxVersion=5), ApiVersion(apiKey=16, minVersion=0, maxVersion=4), ApiVersion(apiKey=17, minVersion=0, maxVersion=1), ApiVersion(apiKey=18, minVersion=0, maxVersion=3), ApiVersion(apiKey=19, minVersion=0, maxVersion=7), ApiVersion(apiKey=20, minVersion=0, maxVersion=6), ApiVersion(apiKey=21, minVersion=0, maxVersion=2), ApiVersion(apiKey=22, minVersion=0, maxVersion=4), ApiVersion(apiKey=23, minVersion=0, maxVersion=4), ApiVersion(apiKey=24, minVersion=0, maxVersion=3), ApiVersion(apiKey=25, minVersion=0, maxVersion=3), ApiVersion(apiKey=26, minVersion=0, maxVersion=3), ApiVersion(apiKey=27, minVersion=0, maxVersion=1), ApiVersion(apiKey=28, minVersion=0, maxVersion=3), ApiVersion(apiKey=29, minVersion=0, maxVersion=2), ApiVersion(apiKey=30, minVersion=0, maxVersion=2), ApiVersion(apiKey=31, minVersion=0, maxVersion=2), ApiVersion(apiKey=32, minVersion=0, maxVersion=4), ApiVersion(apiKey=33, minVersion=0, maxVersion=2), ApiVersion(apiKey=34, minVersion=0, maxVersion=2), ApiVersion(apiKey=35, minVersion=0, maxVersion=2), ApiVersion(apiKey=36, minVersion=0, maxVersion=2), ApiVersion(apiKey=37, minVersion=0, maxVersion=3), ApiVersion(apiKey=38, minVersion=0, maxVersion=2), ApiVersion(apiKey=39, minVersion=0, maxVersion=2), ApiVersion(apiKey=40, minVersion=0, maxVersion=2), ApiVersion(apiKey=41, minVersion=0, maxVersion=2), ApiVersion(apiKey=42, minVersion=0, maxVersion=2), ApiVersion(apiKey=43, minVersion=0, maxVersion=2), ApiVersion(apiKey=44, minVersion=0, maxVersion=1), ApiVersion(apiKey=45, minVersion=0, maxVersion=0), ApiVersion(apiKey=46, minVersion=0, maxVersion=0), ApiVersion(apiKey=47, minVersion=0, maxVersion=0), ApiVersion(apiKey=48, minVersion=0, maxVersion=1), ApiVersion(apiKey=49, minVersion=0, maxVersion=1), ApiVersion(apiKey=50, minVersion=0, maxVersion=0), ApiVersion(apiKey=51, minVersion=0, maxVersion=0), ApiVersion(apiKey=56, minVersion=0, maxVersion=0), ApiVersion(apiKey=57, minVersion=0, maxVersion=0), ApiVersion(apiKey=60, minVersion=0, maxVersion=0), ApiVersion(apiKey=61, minVersion=0, maxVersion=0)], throttleTimeMs=0, supportedFeatures=[], finalizedFeaturesEpoch=0, finalizedFeatures=[])
2021-12-30 08:05:53 DEBUG NetworkClient:912 - [AdminClient clientId=adminclient-1] Node 1 has finalized features epoch: 0, finalized features: [], supported features: [], API versions: (Produce(0): 0 to 9 [usable: 9], Fetch(1): 0 to 12 [usable: 12], ListOffsets(2): 0 to 6 [usable: 6], Metadata(3): 0 to 11 [usable: 11], LeaderAndIsr(4): 0 to 5 [usable: 5], StopReplica(5): 0 to 3 [usable: 3], UpdateMetadata(6): 0 to 7 [usable: 7], ControlledShutdown(7): 0 to 3 [usable: 3], OffsetCommit(8): 0 to 8 [usable: 8], OffsetFetch(9): 0 to 7 [usable: 7], FindCoordinator(10): 0 to 3 [usable: 3], JoinGroup(11): 0 to 7 [usable: 7], Heartbeat(12): 0 to 4 [usable: 4], LeaveGroup(13): 0 to 4 [usable: 4], SyncGroup(14): 0 to 5 [usable: 5], DescribeGroups(15): 0 to 5 [usable: 5], ListGroups(16): 0 to 4 [usable: 4], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 3 [usable: 3], CreateTopics(19): 0 to 7 [usable: 7], DeleteTopics(20): 0 to 6 [usable: 6], DeleteRecords(21): 0 to 2 [usable: 2], InitProducerId(22): 0 to 4 [usable: 4], OffsetForLeaderEpoch(23): 0 to 4 [usable: 4], AddPartitionsToTxn(24): 0 to 3 [usable: 3], AddOffsetsToTxn(25): 0 to 3 [usable: 3], EndTxn(26): 0 to 3 [usable: 3], WriteTxnMarkers(27): 0 to 1 [usable: 1], TxnOffsetCommit(28): 0 to 3 [usable: 3], DescribeAcls(29): 0 to 2 [usable: 2], CreateAcls(30): 0 to 2 [usable: 2], DeleteAcls(31): 0 to 2 [usable: 2], DescribeConfigs(32): 0 to 4 [usable: 4], AlterConfigs(33): 0 to 2 [usable: 2], AlterReplicaLogDirs(34): 0 to 2 [usable: 2], DescribeLogDirs(35): 0 to 2 [usable: 2], SaslAuthenticate(36): 0 to 2 [usable: 2], CreatePartitions(37): 0 to 3 [usable: 3], CreateDelegationToken(38): 0 to 2 [usable: 2], RenewDelegationToken(39): 0 to 2 [usable: 2], ExpireDelegationToken(40): 0 to 2 [usable: 2], DescribeDelegationToken(41): 0 to 2 [usable: 2], DeleteGroups(42): 0 to 2 [usable: 2], ElectLeaders(43): 0 to 2 [usable: 2], IncrementalAlterConfigs(44): 0 to 1 [usable: 1], AlterPartitionReassignments(45): 0 [usable: 0], ListPartitionReassignments(46): 0 [usable: 0], OffsetDelete(47): 0 [usable: 0], DescribeClientQuotas(48): 0 to 1 [usable: 1], AlterClientQuotas(49): 0 to 1 [usable: 1], DescribeUserScramCredentials(50): 0 [usable: 0], AlterUserScramCredentials(51): 0 [usable: 0], AlterIsr(56): 0 [usable: 0], UpdateFeatures(57): 0 [usable: 0], DescribeCluster(60): 0 [usable: 0], DescribeProducers(61): 0 [usable: 0], DescribeTransactions(65): UNSUPPORTED, ListTransactions(66): UNSUPPORTED, AllocateProducerIds(67): UNSUPPORTED).
2021-12-30 08:05:53 DEBUG KafkaAdminClient:1162 - [AdminClient clientId=adminclient-1] Sending AlterUserScramCredentialsRequestData(deletions=[ScramCredentialDeletion(name='my-user', mechanism=2)], upsertions=[]) to kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 1 rack: null). correlationId=4, timeoutMs=9898
2021-12-30 08:05:53 DEBUG NetworkClient:512 - [AdminClient clientId=adminclient-1] Sending ALTER_USER_SCRAM_CREDENTIALS request with header RequestHeader(apiKey=ALTER_USER_SCRAM_CREDENTIALS, apiVersion=0, clientId=adminclient-1, correlationId=4) and timeout 9898 to node 1: AlterUserScramCredentialsRequestData(deletions=[ScramCredentialDeletion(name='my-user', mechanism=2)], upsertions=[])
2021-12-30 08:05:53 DEBUG NetworkClient:870 - [AdminClient clientId=adminclient-1] Received API_VERSIONS response from node 0 for request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-1, correlationId=3): ApiVersionsResponseData(errorCode=0, apiKeys=[ApiVersion(apiKey=0, minVersion=0, maxVersion=9), ApiVersion(apiKey=1, minVersion=0, maxVersion=12), ApiVersion(apiKey=2, minVersion=0, maxVersion=6), ApiVersion(apiKey=3, minVersion=0, maxVersion=11), ApiVersion(apiKey=4, minVersion=0, maxVersion=5), ApiVersion(apiKey=5, minVersion=0, maxVersion=3), ApiVersion(apiKey=6, minVersion=0, maxVersion=7), ApiVersion(apiKey=7, minVersion=0, maxVersion=3), ApiVersion(apiKey=8, minVersion=0, maxVersion=8), ApiVersion(apiKey=9, minVersion=0, maxVersion=7), ApiVersion(apiKey=10, minVersion=0, maxVersion=3), ApiVersion(apiKey=11, minVersion=0, maxVersion=7), ApiVersion(apiKey=12, minVersion=0, maxVersion=4), ApiVersion(apiKey=13, minVersion=0, maxVersion=4), ApiVersion(apiKey=14, minVersion=0, maxVersion=5), ApiVersion(apiKey=15, minVersion=0, maxVersion=5), ApiVersion(apiKey=16, minVersion=0, maxVersion=4), ApiVersion(apiKey=17, minVersion=0, maxVersion=1), ApiVersion(apiKey=18, minVersion=0, maxVersion=3), ApiVersion(apiKey=19, minVersion=0, maxVersion=7), ApiVersion(apiKey=20, minVersion=0, maxVersion=6), ApiVersion(apiKey=21, minVersion=0, maxVersion=2), ApiVersion(apiKey=22, minVersion=0, maxVersion=4), ApiVersion(apiKey=23, minVersion=0, maxVersion=4), ApiVersion(apiKey=24, minVersion=0, maxVersion=3), ApiVersion(apiKey=25, minVersion=0, maxVersion=3), ApiVersion(apiKey=26, minVersion=0, maxVersion=3), ApiVersion(apiKey=27, minVersion=0, maxVersion=1), ApiVersion(apiKey=28, minVersion=0, maxVersion=3), ApiVersion(apiKey=29, minVersion=0, maxVersion=2), ApiVersion(apiKey=30, minVersion=0, maxVersion=2), ApiVersion(apiKey=31, minVersion=0, maxVersion=2), ApiVersion(apiKey=32, minVersion=0, maxVersion=4), ApiVersion(apiKey=33, minVersion=0, maxVersion=2), ApiVersion(apiKey=34, minVersion=0, maxVersion=2), ApiVersion(apiKey=35, minVersion=0, maxVersion=2), ApiVersion(apiKey=36, minVersion=0, maxVersion=2), ApiVersion(apiKey=37, minVersion=0, maxVersion=3), ApiVersion(apiKey=38, minVersion=0, maxVersion=2), ApiVersion(apiKey=39, minVersion=0, maxVersion=2), ApiVersion(apiKey=40, minVersion=0, maxVersion=2), ApiVersion(apiKey=41, minVersion=0, maxVersion=2), ApiVersion(apiKey=42, minVersion=0, maxVersion=2), ApiVersion(apiKey=43, minVersion=0, maxVersion=2), ApiVersion(apiKey=44, minVersion=0, maxVersion=1), ApiVersion(apiKey=45, minVersion=0, maxVersion=0), ApiVersion(apiKey=46, minVersion=0, maxVersion=0), ApiVersion(apiKey=47, minVersion=0, maxVersion=0), ApiVersion(apiKey=48, minVersion=0, maxVersion=1), ApiVersion(apiKey=49, minVersion=0, maxVersion=1), ApiVersion(apiKey=50, minVersion=0, maxVersion=0), ApiVersion(apiKey=51, minVersion=0, maxVersion=0), ApiVersion(apiKey=56, minVersion=0, maxVersion=0), ApiVersion(apiKey=57, minVersion=0, maxVersion=0), ApiVersion(apiKey=60, minVersion=0, maxVersion=0), ApiVersion(apiKey=61, minVersion=0, maxVersion=0)], throttleTimeMs=0, supportedFeatures=[], finalizedFeaturesEpoch=0, finalizedFeatures=[])
2021-12-30 08:05:53 DEBUG NetworkClient:912 - [AdminClient clientId=adminclient-1] Node 0 has finalized features epoch: 0, finalized features: [], supported features: [], API versions: (Produce(0): 0 to 9 [usable: 9], Fetch(1): 0 to 12 [usable: 12], ListOffsets(2): 0 to 6 [usable: 6], Metadata(3): 0 to 11 [usable: 11], LeaderAndIsr(4): 0 to 5 [usable: 5], StopReplica(5): 0 to 3 [usable: 3], UpdateMetadata(6): 0 to 7 [usable: 7], ControlledShutdown(7): 0 to 3 [usable: 3], OffsetCommit(8): 0 to 8 [usable: 8], OffsetFetch(9): 0 to 7 [usable: 7], FindCoordinator(10): 0 to 3 [usable: 3], JoinGroup(11): 0 to 7 [usable: 7], Heartbeat(12): 0 to 4 [usable: 4], LeaveGroup(13): 0 to 4 [usable: 4], SyncGroup(14): 0 to 5 [usable: 5], DescribeGroups(15): 0 to 5 [usable: 5], ListGroups(16): 0 to 4 [usable: 4], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 3 [usable: 3], CreateTopics(19): 0 to 7 [usable: 7], DeleteTopics(20): 0 to 6 [usable: 6], DeleteRecords(21): 0 to 2 [usable: 2], InitProducerId(22): 0 to 4 [usable: 4], OffsetForLeaderEpoch(23): 0 to 4 [usable: 4], AddPartitionsToTxn(24): 0 to 3 [usable: 3], AddOffsetsToTxn(25): 0 to 3 [usable: 3], EndTxn(26): 0 to 3 [usable: 3], WriteTxnMarkers(27): 0 to 1 [usable: 1], TxnOffsetCommit(28): 0 to 3 [usable: 3], DescribeAcls(29): 0 to 2 [usable: 2], CreateAcls(30): 0 to 2 [usable: 2], DeleteAcls(31): 0 to 2 [usable: 2], DescribeConfigs(32): 0 to 4 [usable: 4], AlterConfigs(33): 0 to 2 [usable: 2], AlterReplicaLogDirs(34): 0 to 2 [usable: 2], DescribeLogDirs(35): 0 to 2 [usable: 2], SaslAuthenticate(36): 0 to 2 [usable: 2], CreatePartitions(37): 0 to 3 [usable: 3], CreateDelegationToken(38): 0 to 2 [usable: 2], RenewDelegationToken(39): 0 to 2 [usable: 2], ExpireDelegationToken(40): 0 to 2 [usable: 2], DescribeDelegationToken(41): 0 to 2 [usable: 2], DeleteGroups(42): 0 to 2 [usable: 2], ElectLeaders(43): 0 to 2 [usable: 2], IncrementalAlterConfigs(44): 0 to 1 [usable: 1], AlterPartitionReassignments(45): 0 [usable: 0], ListPartitionReassignments(46): 0 [usable: 0], OffsetDelete(47): 0 [usable: 0], DescribeClientQuotas(48): 0 to 1 [usable: 1], AlterClientQuotas(49): 0 to 1 [usable: 1], DescribeUserScramCredentials(50): 0 [usable: 0], AlterUserScramCredentials(51): 0 [usable: 0], AlterIsr(56): 0 [usable: 0], UpdateFeatures(57): 0 [usable: 0], DescribeCluster(60): 0 [usable: 0], DescribeProducers(61): 0 [usable: 0], DescribeTransactions(65): UNSUPPORTED, ListTransactions(66): UNSUPPORTED, AllocateProducerIds(67): UNSUPPORTED).
2021-12-30 08:05:53 DEBUG KafkaAdminClient:1162 - [AdminClient clientId=adminclient-1] Sending DescribeClientQuotasRequestData(components=[ComponentData(entityType='user', matchType=0, match='CN=my-user')], strict=false) to kafka-kafka-0.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 0 rack: null). correlationId=5, timeoutMs=9894
2021-12-30 08:05:53 DEBUG NetworkClient:512 - [AdminClient clientId=adminclient-1] Sending DESCRIBE_CLIENT_QUOTAS request with header RequestHeader(apiKey=DESCRIBE_CLIENT_QUOTAS, apiVersion=1, clientId=adminclient-1, correlationId=5) and timeout 9894 to node 0: DescribeClientQuotasRequestData(components=[ComponentData(entityType='user', matchType=0, match='CN=my-user')], strict=false)
2021-12-30 08:05:53 DEBUG NetworkClient:870 - [AdminClient clientId=adminclient-1] Received DESCRIBE_CLIENT_QUOTAS response from node 0 for request with header RequestHeader(apiKey=DESCRIBE_CLIENT_QUOTAS, apiVersion=1, clientId=adminclient-1, correlationId=5): DescribeClientQuotasResponseData(throttleTimeMs=0, errorCode=0, errorMessage='', entries=[])
2021-12-30 08:05:53 DEBUG QuotasOperator:59 - Reconciliation #1(watch) KafkaUser(kafka-mtls/my-user): No expected quotas and no existing quotas -> NoOp
2021-12-30 08:05:53 DEBUG NetworkClient:870 - [AdminClient clientId=adminclient-1] Received ALTER_USER_SCRAM_CREDENTIALS response from node 1 for request with header RequestHeader(apiKey=ALTER_USER_SCRAM_CREDENTIALS, apiVersion=0, clientId=adminclient-1, correlationId=4): AlterUserScramCredentialsResponseData(throttleTimeMs=0, results=[AlterUserScramCredentialsResult(user='my-user', errorCode=91, errorMessage='Attempt to delete a user credential that does not exist')])
2021-12-30 08:05:53 DEBUG ScramCredentialsOperator:66 - Reconciliation #1(watch) KafkaUser(kafka-mtls/my-user): Previously deleted SCRAM credentials for user my-user
2021-12-30 08:05:53 DEBUG KafkaAdminClient:1162 - [AdminClient clientId=adminclient-1] Sending DescribeClientQuotasRequestData(components=[ComponentData(entityType='user', matchType=0, match='my-user')], strict=false) to kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 1 rack: null). correlationId=6, timeoutMs=10000
2021-12-30 08:05:53 DEBUG NetworkClient:512 - [AdminClient clientId=adminclient-1] Sending DESCRIBE_CLIENT_QUOTAS request with header RequestHeader(apiKey=DESCRIBE_CLIENT_QUOTAS, apiVersion=1, clientId=adminclient-1, correlationId=6) and timeout 10000 to node 1: DescribeClientQuotasRequestData(components=[ComponentData(entityType='user', matchType=0, match='my-user')], strict=false)
2021-12-30 08:05:53 DEBUG NetworkClient:870 - [AdminClient clientId=adminclient-1] Received DESCRIBE_CLIENT_QUOTAS response from node 1 for request with header RequestHeader(apiKey=DESCRIBE_CLIENT_QUOTAS, apiVersion=1, clientId=adminclient-1, correlationId=6): DescribeClientQuotasResponseData(throttleTimeMs=0, errorCode=0, errorMessage='', entries=[])
2021-12-30 08:05:53 DEBUG QuotasOperator:59 - Reconciliation #1(watch) KafkaUser(kafka-mtls/my-user): No expected quotas and no existing quotas -> NoOp
2021-12-30 08:05:54 DEBUG AbstractResourceOperator:175 - Reconciliation #1(watch) KafkaUser(kafka-mtls/my-user): Secret kafka-mtls/my-user has been deleted
2021-12-30 08:05:54 DEBUG ResourceSupport:192 - Reconciliation #1(watch) KafkaUser(kafka-mtls/my-user): Satisfied: observe deletion of Secret kafka-mtls/my-user
2021-12-30 08:05:54 DEBUG ResourceSupport:51 - Closing io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@49da2d43
2021-12-30 08:05:54 DEBUG AbstractWatchManager:203 - Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@49da2d43
2021-12-30 08:05:54 DEBUG Watcher:36 - Watcher closed
2021-12-30 08:05:54 DEBUG WatchConnectionManager:45 - Closing websocket okhttp3.internal.ws.RealWebSocket@8d736f8
2021-12-30 08:05:54 DEBUG ResourceSupport:154 - Reconciliation #1(watch) KafkaUser(kafka-mtls/my-user): Completing watch future
2021-12-30 08:05:54 DEBUG WatcherWebSocketListener:109 - Socket closing: 
2021-12-30 08:05:54 DEBUG WatcherWebSocketListener:115 - WebSocket close received. code: 1000, reason: 
2021-12-30 08:05:54 DEBUG AbstractWatchManager:127 - Ignoring error for already closed/closing connection
2021-12-30 08:05:54 DEBUG CustomResource:172 - Calling CustomResource#setKind doesn't do anything because the Kind is computed and shouldn't be changed
2021-12-30 08:05:54 DEBUG StatusDiff:47 - Status differs: {"op":"replace","path":"/","value":{"conditions":[{"type":"Ready","status":"True","lastTransitionTime":"2021-12-30T08:05:54.558118Z"}],"observedGeneration":1,"username":"CN=my-user"}}
2021-12-30 08:05:54 DEBUG StatusDiff:48 - Current Status path / has value 
2021-12-30 08:05:54 DEBUG StatusDiff:49 - Desired Status path / has value 
2021-12-30 08:05:54 DEBUG CustomResource:172 - Calling CustomResource#setKind doesn't do anything because the Kind is computed and shouldn't be changed
2021-12-30 08:05:54 DEBUG CustomResource:172 - Calling CustomResource#setKind doesn't do anything because the Kind is computed and shouldn't be changed
2021-12-30 08:05:54 INFO  CrdOperator:113 - Reconciliation #1(watch) KafkaUser(kafka-mtls/my-user): Status of KafkaUser my-user in namespace kafka-mtls has been updated
2021-12-30 08:05:54 INFO  OperatorWatcher:38 - Reconciliation #2(watch) KafkaUser(kafka-mtls/my-user): KafkaUser my-user in namespace kafka-mtls was MODIFIED
2021-12-30 08:05:54 DEBUG AbstractOperator:323 - Reconciliation #1(watch) KafkaUser(kafka-mtls/my-user): Completed status update
2021-12-30 08:05:54 DEBUG AbstractOperator:366 - Reconciliation #2(watch) KafkaUser(kafka-mtls/my-user): Try to acquire lock lock::kafka-mtls::KafkaUser::my-user
2021-12-30 08:05:54 DEBUG CustomResource:172 - Calling CustomResource#setKind doesn't do anything because the Kind is computed and shouldn't be changed
2021-12-30 08:05:54 DEBUG AbstractOperator:618 - Reconciliation #1(watch) KafkaUser(kafka-mtls/my-user): Updated metric strimzi.resource.state[tag(kind=KafkaUser),tag(name=my-user),tag(reason=none),tag(resource-namespace=kafka-mtls)] = 1
2021-12-30 08:05:54 INFO  AbstractOperator:517 - Reconciliation #1(watch) KafkaUser(kafka-mtls/my-user): reconciled
2021-12-30 08:05:54 DEBUG AbstractOperator:369 - Reconciliation #2(watch) KafkaUser(kafka-mtls/my-user): Lock lock::kafka-mtls::KafkaUser::my-user acquired
2021-12-30 08:05:54 DEBUG CustomResource:172 - Calling CustomResource#setKind doesn't do anything because the Kind is computed and shouldn't be changed
2021-12-30 08:05:54 INFO  AbstractOperator:226 - Reconciliation #2(watch) KafkaUser(kafka-mtls/my-user): KafkaUser my-user will be checked for creation or modification
2021-12-30 08:05:54 DEBUG KafkaUserOperator:120 - Reconciliation #2(watch) KafkaUser(kafka-mtls/my-user): Updating User my-user in namespace kafka-mtls
2021-12-30 08:05:54 DEBUG AbstractOperator:440 - Reconciliation #1(watch) KafkaUser(kafka-mtls/my-user): Lock lock::kafka-mtls::KafkaUser::my-user released
**2021-12-30 08:05:54 DEBUG ScramCredentialsOperator:58 - Reconciliation #2(watch) KafkaUser(kafka-mtls/my-user): Deleting SCRAM credentials for user my-user**
2021-12-30 08:05:54 DEBUG KafkaAdminClient:1419 - [AdminClient clientId=adminclient-1] Queueing Call(callName=alterUserScramCredentials, deadlineMs=1640851594704, tries=0, nextAllowedTryMs=0) with a timeout 10000 ms from now.
2021-12-30 08:05:54 DEBUG KafkaAdminClient:1419 - [AdminClient clientId=adminclient-1] Queueing Call(callName=describeClientQuotas, deadlineMs=1640851594704, tries=0, nextAllowedTryMs=0) with a timeout 10000 ms from now.
2021-12-30 08:05:54 DEBUG KafkaAdminClient:1162 - [AdminClient clientId=adminclient-1] Sending AlterUserScramCredentialsRequestData(deletions=[ScramCredentialDeletion(name='my-user', mechanism=2)], upsertions=[]) to kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 1 rack: null). correlationId=7, timeoutMs=10000
2021-12-30 08:05:54 DEBUG KafkaAdminClient:1419 - [AdminClient clientId=adminclient-1] Queueing Call(callName=describeClientQuotas, deadlineMs=1640851594705, tries=0, nextAllowedTryMs=0) with a timeout 10000 ms from now.
2021-12-30 08:05:54 DEBUG NetworkClient:512 - [AdminClient clientId=adminclient-1] Sending ALTER_USER_SCRAM_CREDENTIALS request with header RequestHeader(apiKey=ALTER_USER_SCRAM_CREDENTIALS, apiVersion=0, clientId=adminclient-1, correlationId=7) and timeout 10000 to node 1: AlterUserScramCredentialsRequestData(deletions=[ScramCredentialDeletion(name='my-user', mechanism=2)], upsertions=[])
2021-12-30 08:05:54 DEBUG KafkaAdminClient:1162 - [AdminClient clientId=adminclient-1] Sending DescribeClientQuotasRequestData(components=[ComponentData(entityType='user', matchType=0, match='CN=my-user')], strict=false) to kafka-kafka-0.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 0 rack: null). correlationId=8, timeoutMs=10000
2021-12-30 08:05:54 DEBUG NetworkClient:512 - [AdminClient clientId=adminclient-1] Sending DESCRIBE_CLIENT_QUOTAS request with header RequestHeader(apiKey=DESCRIBE_CLIENT_QUOTAS, apiVersion=1, clientId=adminclient-1, correlationId=8) and timeout 10000 to node 0: DescribeClientQuotasRequestData(components=[ComponentData(entityType='user', matchType=0, match='CN=my-user')], strict=false)
2021-12-30 08:05:54 DEBUG AbstractResourceOperator:124 - Reconciliation #2(watch) KafkaUser(kafka-mtls/my-user): Secret kafka-mtls/my-user does not exist, noop
2021-12-30 08:05:54 DEBUG NetworkClient:870 - [AdminClient clientId=adminclient-1] Received DESCRIBE_CLIENT_QUOTAS response from node 0 for request with header RequestHeader(apiKey=DESCRIBE_CLIENT_QUOTAS, apiVersion=1, clientId=adminclient-1, correlationId=8): DescribeClientQuotasResponseData(throttleTimeMs=0, errorCode=0, errorMessage='', entries=[])
2021-12-30 08:05:54 DEBUG KafkaAdminClient:1162 - [AdminClient clientId=adminclient-1] Sending DescribeClientQuotasRequestData(components=[ComponentData(entityType='user', matchType=0, match='my-user')], strict=false) to kafka-kafka-0.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 0 rack: null). correlationId=9, timeoutMs=10000
2021-12-30 08:05:54 DEBUG NetworkClient:512 - [AdminClient clientId=adminclient-1] Sending DESCRIBE_CLIENT_QUOTAS request with header RequestHeader(apiKey=DESCRIBE_CLIENT_QUOTAS, apiVersion=1, clientId=adminclient-1, correlationId=9) and timeout 10000 to node 0: DescribeClientQuotasRequestData(components=[ComponentData(entityType='user', matchType=0, match='my-user')], strict=false)
2021-12-30 08:05:54 DEBUG QuotasOperator:59 - Reconciliation #2(watch) KafkaUser(kafka-mtls/my-user): No expected quotas and no existing quotas -> NoOp
2021-12-30 08:05:54 DEBUG NetworkClient:870 - [AdminClient clientId=adminclient-1] Received ALTER_USER_SCRAM_CREDENTIALS response from node 1 for request with header RequestHeader(apiKey=ALTER_USER_SCRAM_CREDENTIALS, apiVersion=0, clientId=adminclient-1, correlationId=7): AlterUserScramCredentialsResponseData(throttleTimeMs=0, results=[AlterUserScramCredentialsResult(user='my-user', errorCode=91, errorMessage='Attempt to delete a user credential that does not exist')])
2021-12-30 08:05:54 DEBUG ScramCredentialsOperator:66 - Reconciliation #2(watch) KafkaUser(kafka-mtls/my-user): Previously deleted SCRAM credentials for user my-user
2021-12-30 08:05:54 DEBUG NetworkClient:870 - [AdminClient clientId=adminclient-1] Received DESCRIBE_CLIENT_QUOTAS response from node 0 for request with header RequestHeader(apiKey=DESCRIBE_CLIENT_QUOTAS, apiVersion=1, clientId=adminclient-1, correlationId=9): DescribeClientQuotasResponseData(throttleTimeMs=0, errorCode=0, errorMessage='', entries=[])
2021-12-30 08:05:54 DEBUG QuotasOperator:59 - Reconciliation #2(watch) KafkaUser(kafka-mtls/my-user): No expected quotas and no existing quotas -> NoOp
2021-12-30 08:05:54 DEBUG CustomResource:172 - Calling CustomResource#setKind doesn't do anything because the Kind is computed and shouldn't be changed
2021-12-30 08:05:54 DEBUG StatusDiff:42 - Ignoring Status diff {"op":"replace","path":"/conditions/0/lastTransitionTime","value":"2021-12-30T08:05:54.739631Z"}
2021-12-30 08:05:54 DEBUG AbstractOperator:330 - Reconciliation #2(watch) KafkaUser(kafka-mtls/my-user): Status did not change
2021-12-30 08:05:54 DEBUG CustomResource:172 - Calling CustomResource#setKind doesn't do anything because the Kind is computed and shouldn't be changed
2021-12-30 08:05:54 DEBUG AbstractOperator:610 - Reconciliation #2(watch) KafkaUser(kafka-mtls/my-user): Removed metric strimzi.resource.statekafka-mtls:KafkaUser/my-user
2021-12-30 08:05:54 DEBUG AbstractOperator:618 - Reconciliation #2(watch) KafkaUser(kafka-mtls/my-user): Updated metric strimzi.resource.state[tag(kind=KafkaUser),tag(name=my-user),tag(reason=none),tag(resource-namespace=kafka-mtls)] = 1
2021-12-30 08:05:54 INFO  AbstractOperator:517 - Reconciliation #2(watch) KafkaUser(kafka-mtls/my-user): reconciled
2021-12-30 08:05:54 DEBUG AbstractOperator:440 - Reconciliation #2(watch) KafkaUser(kafka-mtls/my-user): Lock lock::kafka-mtls::KafkaUser::my-user released
2021-12-30 08:06:06 DEBUG Recycler:78 - -Dio.netty.recycler.maxCapacityPerThread: 4096
2021-12-30 08:06:06 DEBUG Recycler:79 - -Dio.netty.recycler.ratio: 8
2021-12-30 08:06:06 DEBUG Recycler:80 - -Dio.netty.recycler.chunkSize: 32
2021-12-30 08:06:06 DEBUG AbstractByteBuf:63 - -Dio.netty.buffer.checkAccessible: true
2021-12-30 08:06:06 DEBUG AbstractByteBuf:64 - -Dio.netty.buffer.checkBounds: true
2021-12-30 08:06:06 DEBUG ResourceLeakDetectorFactory:196 - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@5e3f8446
2021-12-30 08:06:06 DEBUG ZlibCodecFactory:39 - -Dio.netty.noJdkZlibDecoder: false
2021-12-30 08:06:06 DEBUG ZlibCodecFactory:42 - -Dio.netty.noJdkZlibEncoder: false
2021-12-30 08:06:23 DEBUG KafkaAdminClient:1162 - [AdminClient clientId=adminclient-1] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 1 rack: null). correlationId=10, timeoutMs=10000
2021-12-30 08:06:23 DEBUG NetworkClient:512 - [AdminClient clientId=adminclient-1] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=11, clientId=adminclient-1, correlationId=10) and timeout 10000 to node 1: MetadataRequestData(topics=[], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false)
2021-12-30 08:06:23 DEBUG NetworkClient:870 - [AdminClient clientId=adminclient-1] Received METADATA response from node 1 for request with header RequestHeader(apiKey=METADATA, apiVersion=11, clientId=adminclient-1, correlationId=10): MetadataResponseData(throttleTimeMs=0, brokers=[MetadataResponseBroker(nodeId=0, host='kafka-kafka-0.kafka-kafka-brokers.kafka-mtls.svc', port=9091, rack=null), MetadataResponseBroker(nodeId=2, host='kafka-kafka-2.kafka-kafka-brokers.kafka-mtls.svc', port=9091, rack=null), MetadataResponseBroker(nodeId=1, host='kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc', port=9091, rack=null)], clusterId='EQcrSilvRPyTDEt9b6pawQ', controllerId=1, topics=[], clusterAuthorizedOperations=-2147483648)
2021-12-30 08:06:23 DEBUG AdminMetadataManager:247 - [AdminClient clientId=adminclient-1] Updating cluster metadata to Cluster(id = EQcrSilvRPyTDEt9b6pawQ, nodes = [kafka-kafka-0.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 0 rack: null), kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 1 rack: null), kafka-kafka-2.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 2 rack: null)], partitions = [], controller = kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 1 rack: null))
2021-12-30 08:06:53 DEBUG KafkaAdminClient:1162 - [AdminClient clientId=adminclient-1] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to kafka-kafka-0.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 0 rack: null). correlationId=11, timeoutMs=10000
2021-12-30 08:06:53 DEBUG NetworkClient:512 - [AdminClient clientId=adminclient-1] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=11, clientId=adminclient-1, correlationId=11) and timeout 10000 to node 0: MetadataRequestData(topics=[], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false)
2021-12-30 08:06:53 DEBUG NetworkClient:870 - [AdminClient clientId=adminclient-1] Received METADATA response from node 0 for request with header RequestHeader(apiKey=METADATA, apiVersion=11, clientId=adminclient-1, correlationId=11): MetadataResponseData(throttleTimeMs=0, brokers=[MetadataResponseBroker(nodeId=0, host='kafka-kafka-0.kafka-kafka-brokers.kafka-mtls.svc', port=9091, rack=null), MetadataResponseBroker(nodeId=2, host='kafka-kafka-2.kafka-kafka-brokers.kafka-mtls.svc', port=9091, rack=null), MetadataResponseBroker(nodeId=1, host='kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc', port=9091, rack=null)], clusterId='EQcrSilvRPyTDEt9b6pawQ', controllerId=1, topics=[], clusterAuthorizedOperations=-2147483648)
2021-12-30 08:06:53 DEBUG AdminMetadataManager:247 - [AdminClient clientId=adminclient-1] Updating cluster metadata to Cluster(id = EQcrSilvRPyTDEt9b6pawQ, nodes = [kafka-kafka-2.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 2 rack: null), kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 1 rack: null), kafka-kafka-0.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 0 rack: null)], partitions = [], controller = kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 1 rack: null))
2021-12-30 08:07:23 DEBUG KafkaAdminClient:1162 - [AdminClient clientId=adminclient-1] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 1 rack: null). correlationId=12, timeoutMs=10000
2021-12-30 08:07:23 DEBUG NetworkClient:512 - [AdminClient clientId=adminclient-1] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=11, clientId=adminclient-1, correlationId=12) and timeout 10000 to node 1: MetadataRequestData(topics=[], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false)
2021-12-30 08:07:23 DEBUG NetworkClient:870 - [AdminClient clientId=adminclient-1] Received METADATA response from node 1 for request with header RequestHeader(apiKey=METADATA, apiVersion=11, clientId=adminclient-1, correlationId=12): MetadataResponseData(throttleTimeMs=0, brokers=[MetadataResponseBroker(nodeId=0, host='kafka-kafka-0.kafka-kafka-brokers.kafka-mtls.svc', port=9091, rack=null), MetadataResponseBroker(nodeId=2, host='kafka-kafka-2.kafka-kafka-brokers.kafka-mtls.svc', port=9091, rack=null), MetadataResponseBroker(nodeId=1, host='kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc', port=9091, rack=null)], clusterId='EQcrSilvRPyTDEt9b6pawQ', controllerId=1, topics=[], clusterAuthorizedOperations=-2147483648)
2021-12-30 08:07:23 DEBUG AdminMetadataManager:247 - [AdminClient clientId=adminclient-1] Updating cluster metadata to Cluster(id = EQcrSilvRPyTDEt9b6pawQ, nodes = [kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 1 rack: null), kafka-kafka-0.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 0 rack: null), kafka-kafka-2.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 2 rack: null)], partitions = [], controller = kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 1 rack: null))
2021-12-30 08:07:53 INFO  UserOperator:66 - Triggering periodic reconciliation for namespace kafka-mtls
2021-12-30 08:07:53 DEBUG QuotasOperator:149 - Searching for Users with any quotas
2021-12-30 08:07:53 DEBUG KafkaAdminClient:1419 - [AdminClient clientId=adminclient-1] Queueing Call(callName=describeClientQuotas, deadlineMs=1640851713065, tries=0, nextAllowedTryMs=0) with a timeout 10000 ms from now.
2021-12-30 08:07:53 DEBUG ScramCredentialsOperator:88 - Listing all users with SCRAM credentials
2021-12-30 08:07:53 DEBUG KafkaAdminClient:1162 - [AdminClient clientId=adminclient-1] Sending DescribeClientQuotasRequestData(components=[], strict=false) to kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 1 rack: null). correlationId=13, timeoutMs=10000
2021-12-30 08:07:53 DEBUG KafkaAdminClient:1419 - [AdminClient clientId=adminclient-1] Queueing Call(callName=describeUserScramCredentials, deadlineMs=1640851713066, tries=0, nextAllowedTryMs=0) with a timeout 10000 ms from now.
2021-12-30 08:07:53 DEBUG NetworkClient:512 - [AdminClient clientId=adminclient-1] Sending DESCRIBE_CLIENT_QUOTAS request with header RequestHeader(apiKey=DESCRIBE_CLIENT_QUOTAS, apiVersion=1, clientId=adminclient-1, correlationId=13) and timeout 10000 to node 1: DescribeClientQuotasRequestData(components=[], strict=false)
2021-12-30 08:07:53 DEBUG KafkaAdminClient:1162 - [AdminClient clientId=adminclient-1] Sending DescribeUserScramCredentialsRequestData(users=[]) to kafka-kafka-0.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 0 rack: null). correlationId=14, timeoutMs=10000
2021-12-30 08:07:53 DEBUG NetworkClient:512 - [AdminClient clientId=adminclient-1] Sending DESCRIBE_USER_SCRAM_CREDENTIALS request with header RequestHeader(apiKey=DESCRIBE_USER_SCRAM_CREDENTIALS, apiVersion=0, clientId=adminclient-1, correlationId=14) and timeout 10000 to node 0: DescribeUserScramCredentialsRequestData(users=[])
2021-12-30 08:07:53 DEBUG CustomResource:172 - Calling CustomResource#setKind doesn't do anything because the Kind is computed and shouldn't be changed
2021-12-30 08:07:53 DEBUG NetworkClient:870 - [AdminClient clientId=adminclient-1] Received DESCRIBE_USER_SCRAM_CREDENTIALS response from node 0 for request with header RequestHeader(apiKey=DESCRIBE_USER_SCRAM_CREDENTIALS, apiVersion=0, clientId=adminclient-1, correlationId=14): DescribeUserScramCredentialsResponseData(throttleTimeMs=0, errorCode=0, errorMessage='', results=[])
2021-12-30 08:07:53 DEBUG NetworkClient:870 - [AdminClient clientId=adminclient-1] Received DESCRIBE_CLIENT_QUOTAS response from node 1 for request with header RequestHeader(apiKey=DESCRIBE_CLIENT_QUOTAS, apiVersion=1, clientId=adminclient-1, correlationId=13): DescribeClientQuotasResponseData(throttleTimeMs=0, errorCode=0, errorMessage='', entries=[])
2021-12-30 08:07:53 DEBUG AbstractOperator:366 - Reconciliation #3(timer) KafkaUser(kafka-mtls/my-user): Try to acquire lock lock::kafka-mtls::KafkaUser::my-user
2021-12-30 08:07:53 DEBUG AbstractOperator:369 - Reconciliation #3(timer) KafkaUser(kafka-mtls/my-user): Lock lock::kafka-mtls::KafkaUser::my-user acquired
2021-12-30 08:07:53 DEBUG CustomResource:172 - Calling CustomResource#setKind doesn't do anything because the Kind is computed and shouldn't be changed
2021-12-30 08:07:53 INFO  AbstractOperator:226 - Reconciliation #3(timer) KafkaUser(kafka-mtls/my-user): KafkaUser my-user will be checked for creation or modification
2021-12-30 08:07:53 DEBUG KafkaUserOperator:120 - Reconciliation #3(timer) KafkaUser(kafka-mtls/my-user): Updating User my-user in namespace kafka-mtls
**2021-12-30 08:07:53 DEBUG ScramCredentialsOperator:58 - Reconciliation #3(timer) KafkaUser(kafka-mtls/my-user): Deleting SCRAM credentials for user my-user**
2021-12-30 08:07:53 DEBUG KafkaAdminClient:1419 - [AdminClient clientId=adminclient-1] Queueing Call(callName=alterUserScramCredentials, deadlineMs=1640851713153, tries=0, nextAllowedTryMs=0) with a timeout 10000 ms from now.
2021-12-30 08:07:53 DEBUG KafkaAdminClient:1419 - [AdminClient clientId=adminclient-1] Queueing Call(callName=describeClientQuotas, deadlineMs=1640851713154, tries=0, nextAllowedTryMs=0) with a timeout 10000 ms from now.
2021-12-30 08:07:53 DEBUG KafkaAdminClient:1419 - [AdminClient clientId=adminclient-1] Queueing Call(callName=describeClientQuotas, deadlineMs=1640851713154, tries=0, nextAllowedTryMs=0) with a timeout 10000 ms from now.
2021-12-30 08:07:53 DEBUG KafkaAdminClient:1162 - [AdminClient clientId=adminclient-1] Sending AlterUserScramCredentialsRequestData(deletions=[ScramCredentialDeletion(name='my-user', mechanism=2)], upsertions=[]) to kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 1 rack: null). correlationId=15, timeoutMs=10000
2021-12-30 08:07:53 DEBUG NetworkClient:512 - [AdminClient clientId=adminclient-1] Sending ALTER_USER_SCRAM_CREDENTIALS request with header RequestHeader(apiKey=ALTER_USER_SCRAM_CREDENTIALS, apiVersion=0, clientId=adminclient-1, correlationId=15) and timeout 10000 to node 1: AlterUserScramCredentialsRequestData(deletions=[ScramCredentialDeletion(name='my-user', mechanism=2)], upsertions=[])
2021-12-30 08:07:53 DEBUG KafkaAdminClient:1162 - [AdminClient clientId=adminclient-1] Sending DescribeClientQuotasRequestData(components=[ComponentData(entityType='user', matchType=0, match='CN=my-user')], strict=false) to kafka-kafka-0.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 0 rack: null). correlationId=16, timeoutMs=10000
2021-12-30 08:07:53 DEBUG NetworkClient:512 - [AdminClient clientId=adminclient-1] Sending DESCRIBE_CLIENT_QUOTAS request with header RequestHeader(apiKey=DESCRIBE_CLIENT_QUOTAS, apiVersion=1, clientId=adminclient-1, correlationId=16) and timeout 10000 to node 0: DescribeClientQuotasRequestData(components=[ComponentData(entityType='user', matchType=0, match='CN=my-user')], strict=false)
**2021-12-30 08:07:53 DEBUG AbstractResourceOperator:121 - Reconciliation #3(timer) KafkaUser(kafka-mtls/my-user): Secret kafka-mtls/my-user exist, deleting it**
2021-12-30 08:07:53 DEBUG AbstractWatchManager:195 - Watching https://10.222.0.1/api/v1/namespaces/kafka-mtls/secrets?fieldSelector=metadata.name%3Dmy-user&watch=true...
2021-12-30 08:07:53 DEBUG NetworkClient:870 - [AdminClient clientId=adminclient-1] Received DESCRIBE_CLIENT_QUOTAS response from node 0 for request with header RequestHeader(apiKey=DESCRIBE_CLIENT_QUOTAS, apiVersion=1, clientId=adminclient-1, correlationId=16): DescribeClientQuotasResponseData(throttleTimeMs=0, errorCode=0, errorMessage='', entries=[])
2021-12-30 08:07:53 DEBUG KafkaAdminClient:1162 - [AdminClient clientId=adminclient-1] Sending DescribeClientQuotasRequestData(components=[ComponentData(entityType='user', matchType=0, match='my-user')], strict=false) to kafka-kafka-0.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 0 rack: null). correlationId=17, timeoutMs=10000
2021-12-30 08:07:53 DEBUG QuotasOperator:59 - Reconciliation #3(timer) KafkaUser(kafka-mtls/my-user): No expected quotas and no existing quotas -> NoOp
2021-12-30 08:07:53 DEBUG NetworkClient:512 - [AdminClient clientId=adminclient-1] Sending DESCRIBE_CLIENT_QUOTAS request with header RequestHeader(apiKey=DESCRIBE_CLIENT_QUOTAS, apiVersion=1, clientId=adminclient-1, correlationId=17) and timeout 10000 to node 0: DescribeClientQuotasRequestData(components=[ComponentData(entityType='user', matchType=0, match='my-user')], strict=false)
2021-12-30 08:07:53 DEBUG NetworkClient:870 - [AdminClient clientId=adminclient-1] Received ALTER_USER_SCRAM_CREDENTIALS response from node 1 for request with header RequestHeader(apiKey=ALTER_USER_SCRAM_CREDENTIALS, apiVersion=0, clientId=adminclient-1, correlationId=15): AlterUserScramCredentialsResponseData(throttleTimeMs=0, results=[AlterUserScramCredentialsResult(user='my-user', errorCode=91, errorMessage='Attempt to delete a user credential that does not exist')])
2021-12-30 08:07:53 DEBUG ScramCredentialsOperator:66 - Reconciliation #3(timer) KafkaUser(kafka-mtls/my-user): Previously deleted SCRAM credentials for user my-user
2021-12-30 08:07:53 DEBUG NetworkClient:870 - [AdminClient clientId=adminclient-1] Received DESCRIBE_CLIENT_QUOTAS response from node 0 for request with header RequestHeader(apiKey=DESCRIBE_CLIENT_QUOTAS, apiVersion=1, clientId=adminclient-1, correlationId=17): DescribeClientQuotasResponseData(throttleTimeMs=0, errorCode=0, errorMessage='', entries=[])
2021-12-30 08:07:53 DEBUG QuotasOperator:59 - Reconciliation #3(timer) KafkaUser(kafka-mtls/my-user): No expected quotas and no existing quotas -> NoOp
2021-12-30 08:07:53 DEBUG WatcherWebSocketListener:50 - WebSocket successfully opened
2021-12-30 08:07:53 DEBUG ResourceSupport:166 - Reconciliation #3(timer) KafkaUser(kafka-mtls/my-user): Opened watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@51a6c9b0 for evaluation of observe deletion of Secret kafka-mtls/my-user
2021-12-30 08:07:53 DEBUG ResourceSupport:196 - Reconciliation #3(timer) KafkaUser(kafka-mtls/my-user): Not yet satisfied: observe deletion of Secret kafka-mtls/my-user
2021-12-30 08:07:53 DEBUG ResourceSupport:176 - Reconciliation #3(timer) KafkaUser(kafka-mtls/my-user): Pre-check is not complete yet, let's wait for the watch: observe deletion of Secret kafka-mtls/my-user
2021-12-30 08:07:53 DEBUG ResourceSupport:196 - Reconciliation #3(timer) KafkaUser(kafka-mtls/my-user): Not yet satisfied: observe deletion of Secret kafka-mtls/my-user
2021-12-30 08:07:53 DEBUG KafkaAdminClient:1162 - [AdminClient clientId=adminclient-1] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 1 rack: null). correlationId=18, timeoutMs=10000
2021-12-30 08:07:53 DEBUG NetworkClient:512 - [AdminClient clientId=adminclient-1] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=11, clientId=adminclient-1, correlationId=18) and timeout 10000 to node 1: MetadataRequestData(topics=[], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false)
2021-12-30 08:07:53 DEBUG NetworkClient:870 - [AdminClient clientId=adminclient-1] Received METADATA response from node 1 for request with header RequestHeader(apiKey=METADATA, apiVersion=11, clientId=adminclient-1, correlationId=18): MetadataResponseData(throttleTimeMs=0, brokers=[MetadataResponseBroker(nodeId=0, host='kafka-kafka-0.kafka-kafka-brokers.kafka-mtls.svc', port=9091, rack=null), MetadataResponseBroker(nodeId=2, host='kafka-kafka-2.kafka-kafka-brokers.kafka-mtls.svc', port=9091, rack=null), MetadataResponseBroker(nodeId=1, host='kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc', port=9091, rack=null)], clusterId='EQcrSilvRPyTDEt9b6pawQ', controllerId=1, topics=[], clusterAuthorizedOperations=-2147483648)
2021-12-30 08:07:53 DEBUG AdminMetadataManager:247 - [AdminClient clientId=adminclient-1] Updating cluster metadata to Cluster(id = EQcrSilvRPyTDEt9b6pawQ, nodes = [kafka-kafka-0.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 0 rack: null), kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 1 rack: null), kafka-kafka-2.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 2 rack: null)], partitions = [], controller = kafka-kafka-1.kafka-kafka-brokers.kafka-mtls.svc:9091 (id: 1 rack: null))
2021-12-30 08:07:54 DEBUG AbstractResourceOperator:175 - Reconciliation #3(timer) KafkaUser(kafka-mtls/my-user): Secret kafka-mtls/my-user has been deleted
2021-12-30 08:07:54 DEBUG ResourceSupport:192 - Reconciliation #3(timer) KafkaUser(kafka-mtls/my-user): Satisfied: observe deletion of Secret kafka-mtls/my-user
2021-12-30 08:07:54 DEBUG ResourceSupport:51 - Closing io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@51a6c9b0
2021-12-30 08:07:54 DEBUG AbstractWatchManager:203 - Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@51a6c9b0
2021-12-30 08:07:54 DEBUG Watcher:36 - Watcher closed
2021-12-30 08:07:54 DEBUG WatchConnectionManager:45 - Closing websocket okhttp3.internal.ws.RealWebSocket@72a54d30
2021-12-30 08:07:54 DEBUG ResourceSupport:154 - Reconciliation #3(timer) KafkaUser(kafka-mtls/my-user): Completing watch future
2021-12-30 08:07:54 DEBUG WatcherWebSocketListener:109 - Socket closing: 
2021-12-30 08:07:54 DEBUG WatcherWebSocketListener:115 - WebSocket close received. code: 1000, reason: 
2021-12-30 08:07:54 DEBUG AbstractWatchManager:127 - Ignoring error for already closed/closing connection
2021-12-30 08:07:54 DEBUG CustomResource:172 - Calling CustomResource#setKind doesn't do anything because the Kind is computed and shouldn't be changed
2021-12-30 08:07:54 DEBUG StatusDiff:42 - Ignoring Status diff {"op":"replace","path":"/conditions/0/lastTransitionTime","value":"2021-12-30T08:07:54.362032Z"}
2021-12-30 08:07:54 DEBUG AbstractOperator:330 - Reconciliation #3(timer) KafkaUser(kafka-mtls/my-user): Status did not change
2021-12-30 08:07:54 DEBUG CustomResource:172 - Calling CustomResource#setKind doesn't do anything because the Kind is computed and shouldn't be changed
2021-12-30 08:07:54 DEBUG AbstractOperator:610 - Reconciliation #3(timer) KafkaUser(kafka-mtls/my-user): Removed metric strimzi.resource.statekafka-mtls:KafkaUser/my-user
2021-12-30 08:07:54 DEBUG AbstractOperator:618 - Reconciliation #3(timer) KafkaUser(kafka-mtls/my-user): Updated metric strimzi.resource.state[tag(kind=KafkaUser),tag(name=my-user),tag(reason=none),tag(resource-namespace=kafka-mtls)] = 1
2021-12-30 08:07:54 INFO  AbstractOperator:517 - Reconciliation #3(timer) KafkaUser(kafka-mtls/my-user): reconciled
2021-12-30 08:07:54 DEBUG AbstractOperator:440 - Reconciliation #3(timer) KafkaUser(kafka-mtls/my-user): Lock lock::kafka-mtls::KafkaUser::my-user released

2 replies

scholzj Dec 30, 2021
Maintainer

The tls-external type does not expect you to create any secret. You create the certificate with whatever way you want and that is it. You do not pass it in any way to the User Operator.

hari819 Dec 30, 2021
Author

ok thankyou @scholzj , i will try running my tests directly using the user certs i created ,

hari819 · 2022-01-03T06:47:39Z

hari819
Jan 3, 2022
Author

@scholzj , i think there is some issue with my user certificate i have generated ,

i get the below error while trying to connect to the broker using ingress bootstrap URL from kafka , kafka-console-producer.bat --bootstrap-server

[2022-01-03 12:07:45,729] ERROR [Producer clientId=console-producer] Connection to node -1 (bootstrap.kafka.kafka-mtls.../...*:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)

when i enabled ssl debug on both kafka and zk , i see the below error ,

javax.net.ssl|DEBUG|48|data-plane-kafka-network-thread-2-ListenerName(EXTERNAL-9094)-SSL-8|2022-01-03 06:35:23.214 GMT|CertificateMessage.java:372|Consuming client Certificate handshake message (
"Certificates": [
"certificate" : {
"version" : "v3",
"serial number" : "0C A3 17 3F C3 29 D6 F6 A4 F2 19 EF 6E 70 81 B9 A9 5E C5 2B",
"signature algorithm": "SHA256withRSA",
"issuer" : "CN=ClientsCA, O=XXXX",
"not before" : "2021-12-29 16:37:00.000 GMT",
"not after" : "2031-12-27 16:37:00.000 GMT",
"subject" : "CN=my-user, O=XXXX",
"subject public key" : "RSA",
"extensions" : [
{
ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: F6 F3 79 F3 20 E0 B2 6F 95 37 87 CE DF D9 11 CD ..y. ..o.7......
0010: DE 67 07 CE .g..
]
]
},
{
ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
CA:false
PathLen: undefined
]
},
{
ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
emailProtection
]
},
{
ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
]
},
{
ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 44 E2 C0 85 92 A1 FC BF 68 D1 50 20 7D FE 0E 6B D.......h.P ...k
0010: 47 53 15 60 GS.`
]
]
}
]}
]
)
javax.net.ssl|ERROR|48|data-plane-kafka-network-thread-2-ListenerName(EXTERNAL-9094)-SSL-8|2022-01-03 06:35:23.217 GMT|TransportContext.java:341|Fatal (CERTIFICATE_UNKNOWN): Extended key usage does not permit use for TLS client authentication (
"throwable" : {
sun.security.validator.ValidatorException: Extended key usage does not permit use for TLS client authentication
at java.base/sun.security.validator.EndEntityChecker.checkTLSClient(EndEntityChecker.java:245)
at java.base/sun.security.validator.EndEntityChecker.check(EndEntityChecker.java:146)
at java.base/sun.security.validator.Validator.validate(Validator.java:277)
at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:135)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkClientCerts(CertificateMessage.java:682)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:411)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:375)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)
at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430)
at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514)
at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368)
at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at kafka.network.Processor.poll(SocketServer.scala:989)
at kafka.network.Processor.run(SocketServer.scala:892)
at java.base/java.lang.Thread.run(Thread.java:829)}
)
2022-01-03 06:35:23,218 INFO [SocketServer listenerType=ZK_BROKER, nodeId=2] Failed authentication with /10.32.0.1 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-2-ListenerName(EXTERNAL-9094)-SSL-8]
javax.net.ssl|WARNING|48|data-plane-kafka-network-thread-2-ListenerName(EXTERNAL-9094)-SSL-8|2022-01-03 06:35:23.218 GMT|SSLEngineOutputRecord.java:168|outbound has closed, ignore outbound application data,

@scholzj , could you please help me with the command to generate the user certficates, I used like this using cfssl based on your example repo ,

#user cert for mtls
cfssl genkey my-user.json | cfssljson -bare my-user
cfssl sign -config config.json -profile userCA -ca clients.pem -ca-key clients-key.pem my-user.csr my-user.json | cfssljson -bare my-user

after that i just converted them to p12 format using the generated crt and key files,
this was all working when i was using the certificates/secrets generated by strimzi if i leave the type as "tls" , only change is i am generating all the certificates using the custom certificates repo and generating the user cert after that ,

Thanks,

0 replies

hari819 · 2022-01-03T07:22:29Z

hari819
Jan 3, 2022
Author

I see that it is becasue of using the same config.json file which has the below extendedUsages ,

  	"server": {
  		"usages": ["server auth", "client auth", "signing", "key encipherment"],
  		"expiry": "87600h"
  	},

and in the log i see the cert cannot be used for clientAuth ,

 {
    ObjectId: 2.5.29.37 Criticality=false
    ExtendedKeyUsages [
      emailProtection
    ]

  },
  {
    ObjectId: 2.5.29.15 Criticality=true
    KeyUsage [
      DigitalSignature
    ]
  },

it only lists emailProtection as usage , i will use a different config and generate a new user certificate,

0 replies

hari819 · 2022-01-03T08:36:54Z

hari819
Jan 3, 2022
Author

@scholzj , i could connect easily when i generated the certs using openssl ,

thankyou

0 replies

Strimzi

tls-external type not available in 0.27 version of the operator? #6117

Uh oh!

hari819 Dec 28, 2021

Replies: 6 comments · 6 replies

Uh oh!

hari819 Dec 28, 2021 Author

Uh oh!

hari819 Dec 28, 2021 Author

Uh oh!

scholzj Dec 28, 2021 Maintainer

Uh oh!

hari819 Dec 28, 2021 Author

Uh oh!

scholzj Dec 28, 2021 Maintainer

Uh oh!

hari819 Dec 28, 2021 Author

Uh oh!

hari819 Dec 30, 2021 Author

Uh oh!

scholzj Dec 30, 2021 Maintainer

Uh oh!

hari819 Dec 30, 2021 Author

Uh oh!

hari819 Jan 3, 2022 Author

Uh oh!

Uh oh!

hari819 Jan 3, 2022 Author

Uh oh!

hari819 Jan 3, 2022 Author

hari819
Dec 28, 2021

Replies: 6 comments 6 replies

hari819
Dec 28, 2021
Author

hari819
Dec 28, 2021
Author

scholzj Dec 28, 2021
Maintainer

hari819 Dec 28, 2021
Author

scholzj Dec 28, 2021
Maintainer

hari819 Dec 28, 2021
Author

hari819
Dec 30, 2021
Author

scholzj Dec 30, 2021
Maintainer

hari819 Dec 30, 2021
Author

hari819
Jan 3, 2022
Author

hari819
Jan 3, 2022
Author

hari819
Jan 3, 2022
Author