Configmap ref are not seen in exporter pod nor in any of the pods.

[Manidoit]

I am using this file strimzi-kafka-operator/examples/metrics/kafka-metrics.yaml and it is creating all the objects. But while describing the exporter pod, I found

valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml

is not there. Due to that the metrics which are mentioned are not seen in prometheus/Grafana. Am I missing something. Below is the describe pod command.

`kubectl get pod kafka-cluster-kafka-exporter-5897cdc8d8-trhbv -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    strimzi.io/cluster-ca-cert-generation: "0"
  creationTimestamp: "2022-06-22T12:04:06Z"
  generateName: kafka-cluster-kafka-exporter-5897cdc8d8-
  labels:
    app: kafka-cluster
    app.kubernetes.io/instance: kafka-cluster
    app.kubernetes.io/managed-by: strimzi-cluster-operator
    app.kubernetes.io/name: kafka-exporter
    app.kubernetes.io/part-of: strimzi-kafka-cluster
    pod-template-hash: 5897cdc8d8
    strimzi.io/cluster: kafka-cluster
    strimzi.io/kind: Kafka
    strimzi.io/name: kafka-cluster-kafka-exporter
  name: kafka-cluster-kafka-exporter-5897cdc8d8-trhbv
  namespace: kafka
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: kafka-cluster-kafka-exporter-5897cdc8d8
    uid: 72b6974c-7aac-46a6-b25c-ed2c305d8d89
  resourceVersion: "22426529"
  uid: f6e5f99e-8c34-48c3-bccb-8b5ac85e3e61
spec:
  containers:
  - command:
    - /opt/kafka-exporter/kafka_exporter_run.sh
    env:
    - name: KAFKA_EXPORTER_LOGGING
      value: "0"
    - name: KAFKA_EXPORTER_KAFKA_VERSION
      value: 3.2.0
    - name: KAFKA_EXPORTER_GROUP_REGEX
      value: .*
    - name: KAFKA_EXPORTER_TOPIC_REGEX
      value: .*
    - name: KAFKA_EXPORTER_KAFKA_SERVER
      value: kafka-cluster-kafka-bootstrap:9091
    - name: KAFKA_EXPORTER_ENABLE_SARAMA
      value: "false"
    image: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /healthz
        port: tcp-prometheus
        scheme: HTTP
      initialDelaySeconds: 15
      periodSeconds: 30
      successThreshold: 1
      timeoutSeconds: 15
    name: kafka-cluster-kafka-exporter
    ports:
    - containerPort: 9404
      name: tcp-prometheus
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /healthz
        port: tcp-prometheus
        scheme: HTTP
      initialDelaySeconds: 15
      periodSeconds: 30
      successThreshold: 1
      timeoutSeconds: 15
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /tmp
      name: strimzi-tmp
    - mountPath: /etc/kafka-exporter/kafka-exporter-certs/
      name: kafka-exporter-certs
    - mountPath: /etc/kafka-exporter/cluster-ca-certs/
      name: cluster-ca-certs
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-bgbz4
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: gke-dsp-main-us-east-nap-e2-standard--4446d7a3-699t
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: kafka-cluster-kafka-exporter
  serviceAccountName: kafka-cluster-kafka-exporter
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - emptyDir:
      medium: Memory
      sizeLimit: 5Mi
    name: strimzi-tmp
  - name: kafka-exporter-certs
    secret:
      defaultMode: 292
      secretName: kafka-cluster-kafka-exporter-certs
  - name: cluster-ca-certs
    secret:
      defaultMode: 292
      secretName: kafka-cluster-cluster-ca-cert
  - name: kube-api-access-bgbz4
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2022-06-22T12:04:06Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2022-06-22T12:04:36Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2022-06-22T12:04:36Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2022-06-22T12:04:06Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://cdeb99d12c5d21db6e4d231b82740d6ce74f40d4ce9a70493a487a699ca51980
    image: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
    imageID: quay.io/strimzi/kafka@sha256:9dfc09273ad8fc4bdc75cbe93b4a061a05a5defb5884a221ae1e178ba602787f
    lastState: {}
    name: kafka-cluster-kafka-exporter
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2022-06-22T12:04:09Z"
  hostIP: 10.60.0.147
  phase: Running
  podIP: 10.61.4.176
  podIPs:
  - ip: 10.61.4.176
  qosClass: BestEffort
  startTime: "2022-06-22T12:04:06Z"`

[scholzj]

There are two different things:

• Kafka Exporter is a tool for exporting the consumer lag from Kafka as a Prometheus metric. It connects to the Kafka API, collects the information about the committed offsets by consumers and latest offsets from partitions, calculates the lag and provides it as a metric. This is configured with the .spec.kafkaExporter section of the Kafka custom resource. This is a separate tool because Kafka does not provide this as a metric on its own.
• The JMX Exporter (configured with the config map for example in .spec.kafka.metrics) runs inside as the Kafka or ZooKeeper pods as an agent and exposes the metrics which Kafka provides directly as Prometheus metrics. So you need to scrape them from the Kafka (or ZooKeeper) pods directly on port 9404. Even there, you will not see your ConfigMap mounted => the values are copied into an internal config map first and only this one is mounted into the pods.

[Manidoit]

Thanks for the reply @scholzj But we are not able to see any metrics in prometheus which are mentioned in the kafka-metrics.yaml

lowercaseOutputName: true
    rules:
    # Special cases and very specific rules
    - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
      name: kafka_server_$1_$2
      type: GAUGE
      labels:
       clientId: "$3"
       topic: "$4"
       partition: "$5"
    - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
      name: kafka_server_$1_$2
      type: GAUGE
      labels:
       clientId: "$3"
       broker: "$4:$5"
    - pattern: kafka.server<type=(.+), cipher=(.+), protocol=(.+), listener=(.+), networkProcessor=(.+)><>connections
      name: kafka_server_$1_connections_tls_info
      type: GAUGE
      labels:
        cipher: "$2"
        protocol: "$3"
        listener: "$4"
        networkProcessor: "$5"
    - pattern: kafka.server<type=(.+), clientSoftwareName=(.+), clientSoftwareVersion=(.+), listener=(.+), networkProcessor=(.+)><>connections
      name: kafka_server_$1_connections_software
      type: GAUGE
      labels:
        clientSoftwareName: "$2"
        clientSoftwareVersion: "$3"
        listener: "$4"
        networkProcessor: "$5" ```

[scholzj]

Well, as discussed in some of the other threads. You can:

• Check for example with curl whether the metrics are exposed in the pods on the port 9404
• If they are, you will need to make sure your Prometheus is properly configured to scrape the metrics from them. This might be different and depend on your Prometheus installation.