Monitoring Questions

[doronl]

Hi, using version 0.29, I didn't find any notes on documenmtation for this...

Anyway, I configued monitoring according to examples, (JMX on Kafka and ZK as well as Kafka exporter), after reconciliation I see only the Kafka exporter pod running... where are the JMX Exporters? did find any sidecar containers or related pods

I a deploying to a cluser with existing prometheus and Grafana, I added the new dashboards but I can only see the operators metrics...

[scholzj]

You should probably share your Kafka custom resource to make it clear what examples you used. From your description it is not clear if you are trying to use JMX access or Prometheus metrics.

[scholzj]

I'm not really expert on Prometheus and its configuration. I don't think you need app: strimzi label on the Kafka CR. The Kafka pods should be picked up with this selector from the PodMonitor:

  selector:
    matchExpressions:
      - key: "strimzi.io/kind"
        operator: In
        values: ["Kafka", "KafkaConnect"]

You just need to make sure your Prometheus (Prometheus Operator) sees the PodMonitors and can connect to the brokers (things such as network isolation between namespaces etc. might cause issues for example)

[doronl]

I see your proint, selection is by the resource type... curl works for me inside the pods with localhost:9404/metrics however only the operator metrics are arriving to Prometheus, not the kafka exporter nore the kafka JMX or ZK JMX are arriving there...

[scholzj]

As I said, I'm afraid I don't know much about Prometheus :-/. I know the usual advice is to check the targets in the Prometheus UI. That should tell you if Prometheus knows it should scrape them or not. But I'm afraid I don't know much what next.

selection is by the resource type

This is actually a label which is on the pod. It mirrors the Kind of the resource which triggered the creation, but it has the form of the label on the pods here.

[scholzj]

One think I noticed is that your Cluster Operator Pod Monitor does not have these labels:

    prometheus: kube-prometheus
    release: kube-prometheus-stack

All other PodMonitors above have them. Could that be related somehow? Possibly, also the relabeling rules are different on the PodMonitor for the Kafka metrics. So another possible things which can cause some differences.

[doronl]

OK, thanks, I will continue digging into this, will update if there is anything interesting

[doronl]

I had some improvement but still not functioning as expected...

I used a ServiceMonitor instead of a PodMonitor, it needs to work with a service though... I tried creating a service that will give me all pods with:

spec:
  ports:
    - name: http
      protocol: TCP
      port: 9404
      targetPort: 9404
  selector:
    strimzi.io/kind: Kafka
  type: ClusterIP

But it picked up the entity-operator as it has label strimzi.io/kind: Kafka as well and it is a failing target on prometheus...
At the same time I still have some problems with metrics and dashboards, questions:

1. Is there a minimal requirement for prometheus and grafana versions for the metrics integration to work?
2. Is the label strimzi.io/kind: Kafka on the entity-operator correct? Can I remove it somehow?
3. Can I add a custom label to zookeeper and kafka-exporter pods?

[scholzj]

I used a ServiceMonitor instead of a PodMonitor, it needs to work with a service though...

It is just examples, so you are free to customize it anyway you want. But if one PodMonitor works for you, I'm not sure you need to switch to Service Monitor to get the other working.

Is the label strimzi.io/kind: Kafka on the entity-operator correct? Can I remove it somehow?

It is correct as it was deployed through a Kafka custom resource. Strimzi uses it, you cannot remove it.

Can I add a custom label to zookeeper and kafka-exporter pods?

Every compoent has its own template section where you can add additional labels. So feel free to use it. Check the docs for more details -> look for template as the keyword.

[doronl]

Just a short update, was able to make it work using a custom label on Kafka, Zookeeper and The Kafka Exporter but I did it using the PodMonitor (not the ServiceMonitor) because I needed podMetricsEndpoints that ServiceMonitor don't support, this is the reason dashbaords didn't work for me even if metrics were scraped with ServiceMonitor... thanks for now, next week will work on zone awareness

[lenin-joseph]

@scholzj , I have a requirement of adding a custom label to JMX metrics, ex: kafka_server_ReplicaManager_Value{app_kubernetes_io_managed_by="AAAA"}.

Where can I add this? I see that you mentioned about template in previous comments, could you point to the specific files please?

[scholzj]

I guess you can do it in Prometheus when scraping the metrics. Or maybe in the JMX Prometheus exporter configuration config map. But I never tried it, so don't know anything more detailed.