Changing jmx prometheus exporter metrics config from legacy to new format does not update kafka statefulset

[gregwhorley]

Using strimzi-kafka-operator 0.21.1, we had the jmx prometheus exporter configuration set since 0.20.0 in what is now referred to as the legacy format:

spec:
  kafka:
    metrics:
      rules:
        lowercaseOutputName: true
        - pattern: ...

Which, from what I can tell, adds a volume mount to the kafka statefulset that points at a kafka-kafka-config configmap keyed on metrics-config.yml whose value is the json representation of the settings and rules underneath metrics:. At first, I was trying to update the existing ruleset and add new rules within the kafka helm template, which was applied via helm terraform provider without a problem. Getting a manifest of the strimzi helm release afterwards shows the new rule set as expected, but the kafka-kafka-config configmap was not updated.

After looking at issues in the strimzi-kafka-operator repo, I noticed that we were using an old format and thought I should try changing to the new one:

spec:
  kafka:
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yaml

With a separate helm template file that defines the configmap with the rules I wanted:

kind: ConfigMap
apiVersion: v1
metadata:
  name: kafka-metrics
  labels:
    app: strimzi
data:
  kafka-metrics-config.yaml: |
    lowercaseOutputName: true
    rules:
    - pattern ...

Which also applied without a problem and the new Helm release revision manifest shows the new configmap and metrics config. Also, the configmap was created successfully with the key and value I expected. However, the kafka statefulset still was not updated and had the old volume mount pointing at the old configmap and key.

What am I missing?

[scholzj]

The way the metrics are implemented is that the operator copies the metrics configuration into another ConfigMap and that one is mounted into the Kafka pods. This was done so originally when the metrics were included in the Kafka CR directly. Now it is still done when mounting the metrics from Config Map. I guess with only ConfigMaps being supported from 0.23 on, we might get to change this and mount the config map directly, not sure TBH, the separation has some advantages sometimes.

Also, The JMX Exporter is reloading the rules periodically from files, so when the rules change, there is no need to restart the pods. We just wait for it to update automatically.

So if you were expecting some restart of the pods, that does not happen right now, but it is expected to not happen and you do not need to be worried about it.

[gregwhorley]

Thank you for the explanation of the mechanics. One thing I did not do was exec into a kafka pod and look at the metrics config, which I just did. I see the updated rules in /opt/kafka/custom-config/metrics-config.yml.