Metric not present in prometheus

[fsouza-bux]

Hi guys,

I have deployed the chart example here (strimzi-kafka.json) to grafana, querying our prometheus.

I couldn't see a few metrics used in the chart above, one example is the kafka_controller_kafkacontroller_offlinepartitionscount metric.

Digging a bit into it, I found a metric called kafka_controller_kafkacontroller_value, that has a label name, with one the values being OfflinePartitionsCount.

We do have a different JMX pattern then the one used in the example kafka-metrics.yaml, but i could not see any pattern rule that could generate the kafka_controller_kafkacontroller_offlinepartitionscount metric, only a few patterns with count, but putting a "*_count" at the end of the metric name, and the metric above has no "_" before the count.

Another example of metrics i could not find is the kafka_server_replicamanager_leadercount, used in some charts and filters in the same chart, and at the same time i can find another metric called "_value", like kafka_server_replicafetchermanager_value, but at this time i could not find any metric with a label name with "LeaderCount" not nothing.

So my questions are:

1. Are all the metrics in the dashboard example to prometheus included in the example pattern above or do we need to include more patterns in the JMX exporter configmap?
2. If the answer above is yes, where do we find the complete set of patterns to set in the configmap to have the grafana dashboard working properly?
3. Could those metrics above be missing because of inactivity of the cluster? It's for now a DEV / Evaluation cluster only, so we are not pushing / pulling messages there all the time, only when testing a few things.

Cheers,

Fernando

[scholzj]

The Dashboard is written to work with the metrics example (you should always take both from the same Strimzi version - preferably from the version you use). So if you configure the Prometheus export as in our examples, all metrics the dashboard is using should be exposed. If you have your own configuration of the JMX Exporter, it is possible that some of the metrics will not work because of different labels, names etc.

I think it is also possible that some metrics which do not have a value do not exist => for example when there are no topics, no clients etc. But TBH I'm not 100% about it ... I would need to try it with the example YAMLs to know for sure.

[fsouza-bux]

Hi Jakub, thanks for replying! Could you point me which example metric configuration i should be using to get all metrics used in the grafana dashboard? I can probably just add those patterns to my own configmap for the JMX metrics. Would it be the https://github.com/strimzi/strimzi-kafka-operator/blob/master/examples/metrics/kafka-metrics.yaml?

[scholzj]

Yes, that is the matching example for the dashboard.

[fsouza-bux]

Thanks, I'll give it a try and check the metrics.

[fsouza-bux]

Hi Jakub, just checked in minikube with the configmap and metrics were there. Thanks!