Set additional flags for kafka_exporter

[davidpiendl]

Hi,
is it somehow easy possible to configure additional properties of the kafka_exporter? In the CRD there are only a few available (i.e groupRegex, topicRegex ...).

Best regards

[scholzj]

No, I don't think that is possible. Not without modyfing the container image and the startup scripts inside. Why? What would you want to configure?

[davidpiendl]

We want to test a new setting for the used sarama client in the exporter. We run our cluster with enabled autocreation of topics. For consumers and metadata fetches we hope that this setting will fix our experience with recreated topics after deletion with no explicit producer/consumer running. This is most likely caused by the exporter.

[davidpiendl]

Thanks for your quick responses. As far as I've read, simple metadata requests are also enough to trigger topic creation.

[scholzj]

So, do you know what topics does it auto-create? The flag defaults to true -> I really wonder why does it actually want to default to true. What topics does it auto-create?

[scholzj]

Thanks for your quick responses. As far as I've read, simple metadata requests are also enough to trigger topic creation.

In that case, the topic auto-creation is even much more evil then I expected :-o

[scholzj]

So, apparently you are right, a metadata request triggers it. I guess you present a good case why it should be disabled. It would be good to understand what would be the case to have it enabled. If there is one, a new option can be added to disable it. If there isn't one, we can just disable it by default in the container image without changing the API which would be easier.

[davidpiendl]

We enabled it due to the fact we have an asynchronous environment where it can be the case, that topics are deleted, but there are still events in our producing streaming pipeline, which require the topic to exist. Any (Admin)Api call in this streaming pipeline to check if topics exist, or even always trying to create the topic explicitely (even if it still exists), would impact our performance. Therefore we rely on the autocreation feature, but sadly this feature does to much :(.