Kubernetes service for Kafka exporter is missing

[scottc-git]

It seams Strimzi (at least the version that I am using - 0.27.1) does not create a Kubernetes service for the Prometheus exporter. As a result, the Prometheus does not discover the service and of course the Grafana dashboard is empty.

• The exporter is actually created and it is collecting metrics (I port-forwarded to hit the metrics URL to verify this)
• I am using plain Prometheus helm chart (no Operator and no ServiceMonitor)
• I know Strimzi 0.17.0 created the service in question and my Prometheus scraped them.

What would've changed? I noticed that metrics: became metricsConfig: and I believe the CRD implementation has changed since then. How can I have the service back and the necessary labels for Prometheus like prometheus.io/scrape: true

kafka.yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: strimzi
spec:
  kafka:
    version: 2.8.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls
      - name: external
        port: 9094
        type: ingress
        tls: true
        authentication:
          type: tls
        configuration:
          bootstrap:
            host: bootstrap.override.me
            annotations:
              kubernetes.io/ingress.class: nginx
              nginx.ingress.kubernetes.io/ssl-passthrough: "true"
              nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
          brokers:
            - broker: 0
              host: broker-0.override.me
              annotations:
                kubernetes.io/ingress.class: nginx
                nginx.ingress.kubernetes.io/ssl-passthrough: "true"
                nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
            - broker: 1
              host: broker-1.override.me
              annotations:
                kubernetes.io/ingress.class: nginx
                nginx.ingress.kubernetes.io/ssl-passthrough: "true"
                nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
            - broker: 2
              host: broker-2.override.me
              annotations:
                kubernetes.io/ingress.class: nginx
                nginx.ingress.kubernetes.io/ssl-passthrough: "true"
                nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    authorization:
      type: simple
      superUsers:
        - CN=admin-user
        - CN=mm2-user
        - CN=sr-user
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: "2.8"
      inter.broker.protocol.version: "2.8"
      allow.everyone.if.no.acl.found: "true"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml
    template:
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/name
                      operator: In
                      values:
                        - strimzi-kafka
                topologyKey: "kubernetes.io/hostname"
    rack:
      topologyKey: topology.kubernetes.io/zone
  zookeeper: # TODO consider removing this for kafka 2.8 or later
    replicas: 3
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: zookeeper-metrics-config.yml
    template:
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/name
                      operator: In
                      values:
                        - strimzi-zookeeper
                topologyKey: "kubernetes.io/hostname"
  entityOperator:
    topicOperator:
      logging:
        type: inline
        loggers:
          # The default INFO logging is very verbose, outputs many GB/day
          rootLogger.level: WARN
    userOperator: {}
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"
    logging: debug
    # Allow `commonLabels` and `commonAnnotations` to be inserted by Kustomize.
    # Empty maps must be specified here because kafkaExporter is not created by default.
    template:
      deployment: { metadata: { labels: {}, annotations: {} } }
      pod: { metadata: { labels: {}, annotations: {} } }
      service: { metadata: { labels: {}, annotations: {} } }

[scholzj]

The Service was removed long time ago (0.20.0). You can scrape the metrics directly from the Pod without any service. If you really need a service for your Prometheus, you can create it your self. The same applies for the Prometheus metrics in other resources (e.g. Kafka brokers). They are also not anymore exposed on the services (through the services exist there as they are used also for something else).

[scottc-git]

Good to know and thanks for the asnwer @scholzj However, the exporter pod does not have the Prometheus labels. My Prometheus will not discover the pod unless I provide special config which I want to avoid.

[scottc-git]

Ah, you indicated I have to create the service. I can do that but it would've been nice if the service existed. Accessing pod directly by Prometheus seems weird. @scholzj

[scholzj]

As I said, the service (as well as any labels) were removed intentionally since they often create conflicts between the different configurations. Unfortunately, Prometheus is used with many different configurations so it is hard to make it out of the box for everyone. So we had to go the minimal way -> that gives the users the options to add services or labels when needed. If you need any labels on the pods, you can add them through the template sections in the Kafka custom resource.

[scottc-git]

Good to know about the template option which I already use for anti-affinity. Thanks again @scholzj