jmxOptions and credentials to the endpoint

[fsouza-bux]

Hi guys,

We pushed the following configuration to our strimzi cluster:

      jmxOptions:
        authentication:
          type: password

Not sure if I misunderstood the docs, but i was expecting for the http call to the endpoint to ask for credentials as described in the documentation here:

JMX options to open JMX port 9999 to obtain JMX metrics, which is configured for password protection in this example. You can open the port without protection using jmxOptions: {}.

[kafka@kafka-kafka-0 kafka]$ curl http://localhost:9404/metrics
# HELP jvm_classes_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_loaded gauge
jvm_classes_loaded 8511.0
# HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
# TYPE jvm_classes_loaded_total counter
jvm_classes_loaded_total 8712.0
# HELP jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution
# TYPE jvm_classes_unloaded_total counter
jvm_classes_unloaded_total 201.0
....

So calls using localhost, or even from another the pod/namespace using the headless service, they work without asking for any user / password.

It's not a big issue for us as the ports are not going to exposed externally, only prometheus service discovery will connect to those ports, but just to understand if it's really having any effect in the endpoint.

The secret for the JMX option was anyway created, with the user / password in it.

[scholzj]

AFAIK JMX is not based on HTTP. The jmxOptions configure the access to the broker using the Java JMX protocol and that is where the password should be needed.

The metrics endpoint you are accessing there with the curl http://localhost:9404/metrics is the JMX Exporter whihc takes the JMX metrics and exports them in Prometheus format as HTTP. That has no password protection. This is controlled by the metrics / metricsConfig fields.

[fsouza-bux]

thanks a lot for clarifying it! Was imagining that the whole configuration was just for the exporter, and not for the jmx extractor tied to the jvm!

[ibyrktr]

@scholzj I have been exploring the same issue. I can go inside kafka-exporter pod and issue a curl command curl http://localhost:9404/metrics and it works but kafka-exporter does not expose the same metrics with the jmx exporters in the kafka brokers.

here it suggests that it is possible to access the metrics both http://localhost:9999/metrics and via java jmx . How can we reach at the kafka broker jmx export endpoint via curl inside the kafka container? Could it that no http port is mapped in the strimzi config and thus simply http://localhost:9999/metrics does not work.

[scholzj]

I'm not sure this really relates to this question. Kafka Exporter, JMX Exporter, and JMX are completely different things.

1. Kafka Exporter is a separate tool that generates some additional metrics which Kafka itself does not provide such as consumer lag.
2. JMX Exporter will be used in the Kafka pods directly when you enable the metrics in .spec.kafka.metricsConfig in the Kafka custom resource. See https://strimzi.io/docs/operators/latest/full/deploying.html#assembly-metrics-str for more details. But essentially, once oyu enable it, it will expose the metrics on port 9404 on each broker pod.
3. JMX (jmxOptions field) gives you access to JMX. You can connect there with JMX tools and use JMX to collect the metrics instead of Prometheus.

[ibyrktr]

Yeah. Sorry I did not realize zookeeper, kafka, etc. components were also using 9404 to expose the metrics. Thanks.