Failed Authentication messages in kafka broker logs

[sgv0007]

Hi Team,

I am getting lot of failed authentication messages in kafka broker logs from 3 ip's . These are not error message and my broker works fine along with kafka connect . I have faced no issues so far but I need to stop these messages as they are filling up logs. When I checked those ip's are from pod running in kube-system namespace.

kafka broker logs:

2022-11-21 18:29:00,377 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /172.22.199.5 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(LBINTERNAL-9093)-SASL_SSL-7]
2022-11-21 18:29:00,722 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /172.22.199.36 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(LBINTERNAL-9093)-SASL_SSL-8]
2022-11-21 18:29:01,061 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /172.22.199.67 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(LBINTERNAL-9093)-SASL_SSL-9]

kubectl get pods -o wide -n kube-system:

azure-ip-masq-agent-nj7bj 1/1 Running 0 20d 172.22.199.36 aks-agentpool-26742881-vmss000001
azure-ip-masq-agent-w2pqd 1/1 Running 0 20d 172.22.199.5 aks-agentpool-26742881-vmss000000
azure-ip-masq-agent-x8mgf 1/1 Running 0 20d 172.22.199.67 aks-agentpool-26742881-vmss000002
azure-policy-555b999fc9-59k7d 1/1 Running 1 (104d ago) 104d 172.22.199.27 aks-agentpool-26742881-vmss000000
azure-policy-webhook-8cb5c6df6-kj865 1/1 Running 0 104d 172.22.199.11 aks-agentpool-26742881-vmss000000
cloud-node-manager-7xmdt 1/1 Running 0 20d 172.22.199.67 aks-agentpool-26742881-vmss000002
cloud-node-manager-h4bpd 1/1 Running 0 20d 172.22.199.5 aks-agentpool-26742881-vmss000000
cloud-node-manager-jfrkt 1/1 Running 0 20d 172.22.199.36 aks-agentpool-26742881-vmss000001
coredns-57dd64d696-pkjxb 1/1 Running 0 45d 172.22.199.13 aks-agentpool-26742881-vmss000000
coredns-57dd64d696-sdc6w 1/1 Running 0 33d 172.22.199.75 aks-agentpool-26742881-vmss000002
coredns-autoscaler-549489f84f-r7jdf 1/1 Running 0 33d 172.22.199.78 aks-agentpool-26742881-vmss000002
csi-azuredisk-node-5t5gz 3/3 Running 0 30d 172.22.199.67 aks-agentpool-26742881-vmss000002
csi-azuredisk-node-8cfvx 3/3 Running 0 30d 172.22.199.36 aks-agentpool-26742881-vmss000001
csi-azuredisk-node-w94g5 3/3 Running 0 30d 172.22.199.5 aks-agentpool-26742881-vmss000000
csi-azurefile-node-2x7vp 3/3 Running 0 30d 172.22.199.67 aks-agentpool-26742881-vmss000002
csi-azurefile-node-cmzr7 3/3 Running 0 30d 172.22.199.36 aks-agentpool-26742881-vmss000001
csi-azurefile-node-fmqlb 3/3 Running 0 30d 172.22.199.5 aks-agentpool-26742881-vmss000000
konnectivity-agent-7bb7f89795-2bb6z 1/1 Running 0 15d 172.22.199.34 aks-agentpool-26742881-vmss000000
konnectivity-agent-7bb7f89795-rbh7p 1/1 Running 0 15d 172.22.199.56 aks-agentpool-26742881-vmss000001
kube-proxy-dbz5p 1/1 Running 1 (33d ago) 56d 172.22.199.36 aks-agentpool-26742881-vmss000001
kube-proxy-jpgkj 1/1 Running 0 56d 172.22.199.67 aks-agentpool-26742881-vmss000002
kube-proxy-s5swj 1/1 Running 0 56d 172.22.199.5 aks-agentpool-26742881-vmss000000

kafka yaml:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: eesb-cluster
spec:
  clientsCa:
    generateCertificateAuthority: false
  kafka:
    version: 3.1.0
    replicas: 3
    logging: 
      type: inline
      loggers:
        kafka.root.logger.level: "INFO"
    resources: 
      requests:
        memory: 2Gi
        cpu: "1"
      limits:
        memory: 4Gi
        cpu: "2"
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
      periodSeconds: 10
      successThreshold: 1
      failureThreshold: 3
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
      periodSeconds: 10
      successThreshold: 1
      failureThreshold: 3
    template:
      pod:
        imagePullSecrets:
        - name: artifactory-kafka
# Kafka JVM      
jvmOptions:
  gcLoggingEnabled: false
  -Xms: 512m
  -Xmx: 1g
  -XX:
    UseG1GC: true
    MaxGCPauseMillis: 20
    InitiatingHeapOccupancyPercent: 35
    ExplicitGCInvokesConcurrent: true
listeners:
  - name: plain
    port: 9092
    type: internal
    tls: false
  - name: lbinternal
    port: 9093
    type: loadbalancer
    tls: true
    authentication:
     type: oauth
     validIssuerUri: https://sts.windows.net/13ef78-5f82-xxxx-xxxx/
     jwksEndpointUri: https://login.microsoftonline.com/13ef78-5f82-xxxx-xxxx/discovery/keys
     #userNameClaim: preferred_username
     maxSecondsWithoutReauthentication: 3600    
     disableTlsHostnameVerification: true
     jwksExpirySeconds: 360
     jwksRefreshSeconds: 300
     jwksMinRefreshPauseSeconds: 1
     checkAccessTokenType: false
     accessTokenIsJwt: true
     
    configuration:
      bootstrap:
        loadBalancerIP: 172.22.199.99
        annotations:
          service.beta.kubernetes.io/azure-load-balancer-internal: "true"
          external-dns.alpha.kubernetes.io/hostname: eesb.dach-uat.euro.com
        alternativeNames: 
        - eesb.dach-uat.euro.com            
      brokers:
      - broker: 0
        loadBalancerIP: 172.22.199.100
        annotations:
          service.beta.kubernetes.io/azure-load-balancer-internal: "true"
          external-dns.alpha.kubernetes.io/hostname: kafka-0.eesb.dach-uat.euro.com
        advertisedHost: kafka-0.eesb.dach-uat.eurofins.com
      - broker: 1
        loadBalancerIP: 172.22.199.101
        annotations:
          service.beta.kubernetes.io/azure-load-balancer-internal: "true"
          external-dns.alpha.kubernetes.io/hostname: kafka-1.eesb.dach-uat.euro.com
        advertisedHost: kafka-1.eesb.dach-uat.euro.com
      - broker: 2
        loadBalancerIP: 172.22.199.102
        annotations:
          service.beta.kubernetes.io/azure-load-balancer-internal: "true"
          external-dns.alpha.kubernetes.io/hostname: kafka-2.eesb.dach-uat.euro.com
        advertisedHost: kafka-2.eesb.dach-uat.euro.com
      brokerCertChainAndKey:
        secretName: esb-uat-kafka-listener-cert
        certificate: tls.crt
        key: tls.key
config:
  num.partitions: 1
  offsets.topic.replication.factor: 3
  log.retention.hours: 24
  log.segment.bytes: 1073741824
  log.roll.hours: 24
  log.retention.check.interval.ms: 300000
  num.network.threads: 3
  num.io.threads: 8
  transaction.state.log.replication.factor: 3
  transaction.state.log.min.isr: 2
  default.replication.factor: 3
  min.insync.replicas: 2
  num.recovery.threads.per.data.dir: 1
  inter.broker.protocol.version: "3.1"
  auto.create.topics.enable: false
  delete.topic.enable: true
# Rack awareness
 # replica.selector.class: org.apache.kafka.common.replica.RackAwareReplicaSelector 
 # broker.rack:
#rack:
 # topologyKey: topology.kubernetes.io/region=
 # topologyKey: topology.kubernetes.io/zone= 
storage:
  type: jbod
  volumes:
  - deleteClaim: false
    id: 0
    size: 80Gi
    type: persistent-claim
  
# Kafka Prometheus JMX Exporter
#metricsConfig:
  #type: jmxPrometheusExporter
  #valueFrom:
    #configMapKeyRef:
      #key: kafka-metrics-config.yml
      #name: kafka-zk-metrics
#metrics:
  #lowercaseOutputName: true
#kafkaExporter:
  #groupRegex: .*
  #topicRegex: .*
# Kafka JMX metrics
#jmxOptions:
  #authentication:
    #type: "password"


  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 10Gi
      deleteClaim: false
    # Zookeeper loggers and log level  
    #logging:
      #type: external
      #name: kafka-logging-configmap
    logging: 
      type: inline
      loggers:
        zookeeper.root.logger: "INFO"
    resources:
      requests:
        memory: 1Gi
        cpu: "1"
      limits:
        memory: 2Gi
        cpu: "2"
    # Zookeeper JVM      
    jvmOptions:
      gcLoggingEnabled: false
      -Xms: 512m
      -Xmx: 512m
      -XX:
        UseG1GC: true
        MaxGCPauseMillis: 20
        InitiatingHeapOccupancyPercent: 35
        ExplicitGCInvokesConcurrent: true

please help.

Thanks,
Shiva

[scholzj]

These messages come from misconfigured clients or possible from some software which is not Kafka client and just tests the TCP connection. You can get rid of them in two ways -> you change the logging configuration, but no idea how many important messages you loose. Or you stop the clients or the software doing this. The pods in the kube-system namespace ar epossibly just forwarding the connections -> so you would need to investigate what pods they are and where are they forwarding the connection from.