Unable to collect the kafka cluster logs using openshift cluster logforwarder

[L040639]

Using cluster logforwarder we are pushing namespace pod logs to the loki and using grafana we use query but the kafka broker logs are not exporting even i have added the kafka cluster namespace correctly in the log forwarder , some of the pods in the same kafka cluster namespce logs are correctly exporting but kafka broker logs are not exporting , is there any configuration i missed ??

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-cluster
  namespace: <XXX>
spec:
  kafka:
    authorization:
      type: opa
      url: https://<XXX>/v1/data/kafka/allow
      allowOnError: false
      enableMetrics: true
      tlsTrustedCertificates:
        - secretName: lk-ca-root-bundle
          certificate: ca.crt
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: '3.9'
      connections.max.idle.ms: 1800000
    storage:
      type: persistent-claim
      size: 500Gi
      class: gp3
    listeners:
      - authentication:
          checkAccessTokenType: false
          clientId: <XX>
          clientSecret:
            key: secret
            secretName: entra-id-client
          jwksEndpointUri: '<XX>'
          maxSecondsWithoutReauthentication: 3600
          enableMetrics: true
          type: oauth
          validIssuerUri: '<XXX>'
          userNameClaim: azp
        configuration:
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: kafka-streams-prd-cert-secret
        name: oauth
        port: 9093
        tls: true
        type: internal
      - authentication:
          checkAccessTokenType: false
          clientId: <XXX>
          clientSecret:
            key: secret
            secretName: entra-id-client
          jwksEndpointUri: '<XXX>'
          maxSecondsWithoutReauthentication: 3600
          enableMetrics: true
          type: oauth
          validIssuerUri: '<XXX>'
          userNameClaim: azp
        configuration:
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: kafka-streams-prd-cert-secret
          bootstrap:
            alternativeNames:
            - <XX>
            annotations:
              external-dns.alpha.kubernetes.io/hostname: <XX>.
              external-dns.alpha.kubernetes.io/access: "private"
          brokers:
          - broker: 0
            advertisedHost: <XXX>
            annotations:
              external-dns.alpha.kubernetes.io/hostname: <XXXX>.
              external-dns.alpha.kubernetes.io/access: "private"
          - broker: 1
            advertisedHost: <XXX>
            annotations:
              external-dns.alpha.kubernetes.io/hostname: <XXX>.
              external-dns.alpha.kubernetes.io/access: "private"
          - broker: 2
            advertisedHost: <XXX>
            annotations:
              external-dns.alpha.kubernetes.io/hostname: <XXX>.
              external-dns.alpha.kubernetes.io/access: "private"
        name: external
        port: 9094
        tls: true
        type: loadbalancer
    version: 3.9.0
    replicas: 3
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: kafka-metrics-config.yml
          name: kafka-streams-prd-metrics
    template:
      externalBootstrapService:
        metadata:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-type: nlb
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
            service.beta.kubernetes.io/aws-load-balancer-internal: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout: "60"
            service.beta.kubernetes.io/aws-load-balancer-listener-attributes.TCP-9094: tcp.idle_timeout.seconds=5000
      perPodService:
        metadata:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-type: nlb
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-internal: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout: "60"
            service.beta.kubernetes.io/aws-load-balancer-listener-attributes.TCP-9094: tcp.idle_timeout.seconds=5000
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: <XX>   ##dedicated nodegroup for kafka
                      operator: Exists
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/cluster
                      operator: In
                      values:
                        - kafka-cluster
                    - key: strimzi.io/component-type
                      operator: In
                      values:
                        - kafka
                topologyKey: kubernetes.io/hostname
        tolerations:
          - key: "<XX>"
            operator: "Exists"
            effect: "NoSchedule"
        topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: topology.kubernetes.io/zone
            whenUnsatisfiable: ScheduleAnyway
            labelSelector:
              matchLabels:
                strimzi.io/cluster: kafka-cluster
                strimzi.io/component-type: kafka
  entityOperator:
    topicOperator: {}
    userOperator: {}
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: <XXX>
                      operator: Exists
        tolerations:
          - key: "<XX>"
            operator: "Exists"
            effect: "NoSchedule"
  zookeeper:
    storage:
      type: persistent-claim
      class: gp3
      size: 50Gi
    replicas: 3
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-streams-prd-metrics
          key: zookeeper-metrics-config.yml
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: <XX>  ##dedicated nodegroup for kafka
                      operator: Exists
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/cluster
                      operator: In
                      values:
                        - kafka-cluster
                    - key: strimzi.io/component-type
                      operator: In
                      values:
                        - zookeeper
                topologyKey: kubernetes.io/hostname
        tolerations:
          - key: "<XX>"
            operator: "Exists"
            effect: "NoSchedule"
        topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: topology.kubernetes.io/zone
            whenUnsatisfiable: ScheduleAnyway
            labelSelector:
              matchLabels:
                strimzi.io/cluster: kafka-cluster
                strimzi.io/component-type: zookeeper
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: node-role.kubernetes.io/worker
                      operator: Exists
        tolerations:
          - key: "<XXX>"
            operator: "Exists"
            effect: "NoSchedule"

apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: collector
  namespace: <XX>
spec:
  serviceAccount:
    name: collector
  inputs:
    - name: default-logstore
      type: application
      application:
        includes:
          - namespace: default
          - namespace: <XXX>
    - name: splunk-dev-app-logs
      type: application
      application:
        includes:
          - namespace: default
          - namespace: <XXX>
    - name: loki-app-logs
      type: application
      application:
        includes:
          - namespace: kafka-streams
    - name: loki-infra-logs
      type: infrastructure
      infrastructure:
        sources:
          - container
          - node
  outputs:
    - name: default-lokistack
      type: lokiStack
      lokiStack:
        target:
          name: logging-loki
          namespace: <XXX>
        authentication:
          token:
            from: serviceAccount
      tls:
        ca:
          key: service-ca.crt
          configMapName: openshift-service-ca.crt
    - name: splunk-spe-dev
      type: splunk
      splunk:
        authentication:
          token:
            secretName: spe-splunk-secret
            key: hecToken
        url: "<XXXX>"
    - name: loki-receiver
      type: loki
      loki:
        url: >-
          <XXX>
        tenantKey: '{.kubernetes.namespace_name||"default"}'
        labelKeys:
          - kubernetes.namespace_name
          - kubernetes.pod_name
          - kubernetes.container_name
          - kubernetes.labels.app
        tuning:
          compression: gzip
      tls:
        ca:
          key: XX
          secretName: XX
    - name: loki-infra-receiver
      type: loki
      loki:
        url: >-
          <XXX>
        tenantKey: infrastructure
        labelKeys:
          - kubernetes.namespace_name
          - kubernetes.pod_name
          - kubernetes.container_name
          - log_type
        tuning:
          compression: gzip
      tls:
        ca:
          key: XXX
          secretName: XX
    - name: loki-admin-apps-receiver
      type: loki
      loki:
        url: >-
          <XXX>
        tenantKey: application
        labelKeys:
          - kubernetes.namespace_name
          - kubernetes.pod_name
          - kubernetes.container_name
          - kubernetes.labels.app
          - log_type
        tuning:
          compression: gzip
      tls:
        ca:
          key: XXX
          secretName: XX
    - name: loki-audit-receiver
      type: loki
      loki:
        url: >-
         <XXX>
        tenantKey: audit
        labelKeys:
          - kubernetes.namespace_name
          - kubernetes.pod_name
          - kubernetes.container_name
          - log_type
        tuning:
          compression: gzip
      tls:
        ca:
          key: ca.crt
          secretName: loki-secret
  pipelines:
    - name: default-loki-s3
      inputRefs:
        - default-logstore
      outputRefs:
        - default-lokistack
    - name: splunk-spe-dev-logging
      inputRefs:
        - splunk-dev-app-logs
      outputRefs:
        - splunk-spe-dev
    - name: loki-logging
      inputRefs:
        - loki-app-logs
      outputRefs:
        - loki-receiver
    - name: loki-infrastructure-logging
      inputRefs:
        - loki-infra-logs
      outputRefs:
        - loki-infra-receiver
    - name: loki-admin-apps-logging
      inputRefs:
        - loki-app-logs
      outputRefs:
        - loki-admin-apps-receiver
    - name: loki-audit-logging
      inputRefs:
        - audit
      outputRefs:
        - loki-audit-receiver

[scholzj]

Strimzi just logs into standard output and the rest is up to your infrastrure (OpenShift, CRI-O, etc.). Strimzi does not really export the logs anywhere. Do you see the logs at least with kubectl logs?

In any case, I think you will need to check with your OpenShift support.