limit kafka exporter metrics

[zarafree]

We wanted to gather only limited metrics from kafka exporter (Not JMX kafka metrics). For example, we need only kafka_consumergroup_current_offset and kafka_consumergroup_lag metrics from kafka exporter. We are getting OOMkilled for kafka exporter. We tried to increase pod limit but no luck so wanted to filter metrics so it will improve performance. any help / suggestions?

[scholzj]

Have you check the docs? That shouls the available options. But I don't think it supports only some metric types

[zarafree]

Thanks for reply. I have checked that documentation but not found any specific settings/configuration for kafka exporter.

[scholzj]

The available options will be for sure described in the API reference n the Configuration guide if nowhere else.

[zarafree]

We have two types filters available -

1. topic regex
2. consumer group regex
   What we want is to filter on metric type? I don't think that is available as per below link.
   Reference: https://github.com/danielqsj/kafka_exporter#metrics
   Our use case is only get required metrics for monitoring but due to above limitation it is capturing all the metrics and making exporter pod OOMkilled. We have large cluster with around 9000+ topics and we set resource limit as 12GB for exporter pod.
   Could you please guide us on this?

[scholzj]

I cannot guide you on this. As I said, the configuration options available are what you can use. And as you said, the Kafka Exporter does not support anything else. So I think your only options are to give it more memory, use less topics and stop using Kafka Exporter.

9000+ topics are a lot so I wonder if you are using Kafka the right way. But there might be much you can do about it anyway.