(Cluster) Operator metrics?

[armisz]

Hi, There is a sample Grafana dashboard available for Operators: https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/grafana-dashboards/strimzi-operators.json

Which pod is providing those metrics used in the dashboard? Is is perhaps the Cluster Operator? How can I enable the publication of the metrics?

I am using Strimzi 0.29.0 and have already Grafana dashboards for Kafka, ZK and Kafka Exporter (i.e. a running Prometheus scraping the pods in question).

Thanks!
Michael

[scholzj]

It's all the operators - Cluster operator, User operator, and Topic operator. The metrics are always published. You just have to scrape them. They are IIRC on ports 8080 for Cluster and Topic operators and 8081 for the User Operator. The path is always /metrics.

[armisz]

Thanks! May be worth mentioning this in the documentation. I assumed that the port will be 9404 - similar to Kafka and ZK.

I also noticed that some of the expressions do not work. The JVM metrics
https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/grafana-dashboards/strimzi-operators.json#L1448
https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/grafana-dashboards/strimzi-operators.json#L1538
https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/grafana-dashboards/strimzi-operators.json#L1628
do not have a container selector and will return no data.

[scholzj]

The ports are in the examples. Feel free to open a PR for the docs if you want to improve it ;-)

The dashboards work fine for me. I do not have any errors in these charts. I do not understand what you mean by selector. There is no selector here.

[armisz]

"expr": "sum(jvm_memory_used_bytes{container=~\"user-operator|topic-operator|strimzi-cluster-operator\"}) by (container)"

returns entries like

jvm_memory_used_bytes{app="strimzi", area="heap", avq_app="my-strimzi-kafka", id="Eden Space", instance="10.254.106.137:8080", job="kubernetes-pods", kubernetes_namespace="dev", kubernetes_pod_name="my-strimzi-kafka-entity-operator-54bcd4f68-jjdvl", strimzi_io_cluster="my-strimzi-kafka"}

There is no data about container which can be used to narrow down the hits and group.

[scholzj]

I guess in that case you need to either fix your labeling of the metrics to match our examples or adjust the dashboard to somehow match the labels you have there.