JMX Metrics Enabled on Zookeeper/Datadog

[tferfce]

Hello,

So I'm trying to use jmx metrics that are exposed from zookeeper and collect them with zk integration for datadog. I have also configured the the openmetrics integration with prometheus exporter which is supported from Kafka crd and it works like charm. When i try to use this though it doesn't collect and zookeeper specific metrics.

Works:
{
"openmetrics": {
"init_config": {
"service": "Zookeeper"
},
"instances": [
{
"openmetrics_endpoint": "http://%%host%%:9404/metrics",
"namespace": "zookeeper",
"metrics": [
".*"
]
}
]
}
}
Doesn't Work:
{
"zk": {
"init_config": {
"is_jmx": true,
"collect_default_metrics": true
},
"instances": [
{
"host": "%%host%%",
"port": "2181",
"timeout": 120,
"name": "JMX Zookeeper"
}
]
}
}

The only metrics im able to collect are these.
zookeeper.instances
zookeeper.instances.inactive

My question is which metrics are exposed from this configuration??

Thanks,
Thomas

[scholzj]

The JMX access definitely won't be on port 2181 where you seem to point in your JSON but on port 9999.

[scholzj]

Also, JMX does not expose some metrics. It opens a port where clients can connect with the JMX client and talk with the server. So it is up to your metrics server what it does with the JMX interface, what metrics it connects, whether it uses it to configure things or shutdown things etc. (as JMX is not only a monitoring but a management solution).

[tferfce]

I already tried the 9999 port and i get this error

And just to note that this happens only with zk integration. With jmx integration where i jvm metrics are collected have no issues! On port 2181 i wasn't getting these errors.

[tferfce]

Datadog supports zk integration which is documented in here ZK Integration. The jmx integration is a different one that supports the port you mentioned and its actually working for zookeeper JMX Integration. The reason i was trying to connect on 2181 is the zk integration. As for the mTLS you might be right cause cause it is mentioned to use SSL and to configure the DD agent to use SSL to connect to zookeeper. Prometheus metrics with openmetrics Openmetrics Integration is working for 9404.

Prometheus: 9094 works
JMX: 9999 works on jmx but not on zk (error)
ZK: 8121 doesn't collect

[scholzj]

Ok, well, as I said, Strimzi does not support any third party applications connecting directly to ZooKeeper. So I do not think that will work for you.

[tferfce]

Do you think that enabling the SSL communication with DD and Zookeeper might have something to do with it and worth the try or the Prometheus metrics is the only way in my case?

[scholzj]

I think the Prometheus metrics are all that is really needed. Also, keep in mind that you are not missing enabling TLS communication. As I said, this is not supported in Strimzi. You would need to hack mTLS security - e.g. steal and reuse the certificates of the Kafka brokers or similar component in order to pass the authentication. And if you decide to do it, you should properly understand what will DataDog do with the ZooKeeper cluster as it can do basically anything it wants with it as it is not just a metrics export.

[tferfce]

Hmm i understand. It carries a security risk on that! Very much appreciate the support! Thanks!