Expose Zookeeper JMX port for Datadog agent.

[raj-saxena]

Hi,
We use Datadog for monitoring & the agent can auto-discover integrations based on annotations. Details

So far, I have been successful in enabling the Jmx port for kafka brokers and correctly marking the pods so that Datadog can identify & pull metrics. ✅

I want to do the same for Zookeeper metrics and have added similar marker annotations to the Zookeeper pods. However, the Datadog check reports error possibly because the port is not open/reachable.

      Traceback (most recent call last):
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/zk/zk.py", line 248, in _send_command
          with closing(socket.create_connection((self.host, self.port))) as sock:
        File "/opt/datadog-agent/embedded/lib/python3.8/socket.py", line 808, in create_connection
          raise err
        File "/opt/datadog-agent/embedded/lib/python3.8/socket.py", line 796, in create_connection
          sock.connect(sa)
      TimeoutError: [Errno 110] Connection timed out

Unlike KafkaClusterSpec, there doesn't seem to be a JmxOptions for Zookeeper. Please advise.
Gist of our Kafka.yaml - https://gist.github.com/raj-saxena/72d326ae7b73409a703908b500572ed2

[scholzj]

The JMX access is currently supported only in Kafka brokers (and in 0.22.0 it will be added to Kafka Connect and Mirror Maker 2 as well). Historically, this has been always contributed by users - as maintainers we prefer to use the Prometheus endpoint for monitoring. So I think the JMX access to Zookeeper is waiting for someone to contribute it. If you would want to look into it, these are the PRs which added the KAfka and Kafka Connect JMS access:

• Kafka connect: add jmxOptions #4096
• Feat: enable internal JMX Connections on a secure or insecure JMX port on Kafka Brokers #2268

[raj-saxena]

Update: Datadog supports Prometheus & OpenMetrics metrics collection - https://docs.datadoghq.com/agent/kubernetes/prometheus/

I added the following annotations to configure it:

...
  kafkaExporter:
      # Annotations for Datadog JMX auto-discovery
    template:
      pod:
        metadata:
          annotations: 
            ad.datadoghq.com/my-cluster-kafka-exporter.check_names: '["openmetrics"]'
            ad.datadoghq.com/my-cluster-kafka-exporter.init_configs: '[{}]'
            ad.datadoghq.com/my-cluster-kafka-exporter.instances: |
              [
                {
                  "prometheus_url": "http://%%host%%:%%port%%/metrics",
                  "namespace": "my_kafka",
                  "metrics": ["go_gc_duration_seconds","go_goroutines","go_memstats_alloc_bytes","go_memstats_alloc_bytes_total","go_memstats_buck_hash_sys_bytes","go_memstats_frees_total","go_memstats_gc_sys_bytes","go_memstats_heap_alloc_bytes","go_memstats_heap_idle_bytes","go_memstats_heap_inuse_bytes","go_memstats_heap_objects","go_memstats_heap_released_bytes_total","go_memstats_heap_sys_bytes","go_memstats_last_gc_time_seconds","go_memstats_lookups_total","go_memstats_mallocs_total","go_memstats_mcache_inuse_bytes","go_memstats_mcache_sys_bytes","go_memstats_mspan_inuse_bytes","go_memstats_mspan_sys_bytes","go_memstats_next_gc_bytes","go_memstats_other_sys_bytes","go_memstats_stack_inuse_bytes","go_memstats_stack_sys_bytes","go_memstats_sys_bytes","kafka_brokers","kafka_consumergroup_current_offset","kafka_consumergroup_lag","kafka_exporter_build_info","kafka_topic_partition_current_offset","kafka_topic_partition_in_sync_replica","kafka_topic_partition_leader","kafka_topic_partition_leader_is_preferred","kafka_topic_partition_oldest_offset","kafka_topic_partition_replicas","kafka_topic_partition_under_replicated_partition","kafka_topic_partitions","process_cpu_seconds_total","process_max_fds","process_open_fds","process_resident_memory_bytes","process_start_time_seconds","process_virtual_memory_bytes"]
                }
              ]   

Soon, the metrics appeared in Datadog 🎉

[scholzj]

@helloimowen You need to decide if you want to use Prometheus or JMX. JMX is not an HTTP based protocol. You cannot use it using curl. The Prometheus metrics ae HTTP based and can be used with curl. But you configure them differently and are on port 9404 (check the examples for the configuration in examples/metrics). You need to decide what exactly you want.

[helloimowen]

Got it. - thanks for the quick response! Ideally we would use JMX and datadog's default kafka integration, but we have been unable to see metrics in datadog as of yet. Do you have any advice on verifying that port 9999 is configured to send metrics without curl?

[scholzj]

I have no experience with Datadog. So you need to ask them I guess, or maybe someone from the other Datadog users here can help. JMX is a Java specific protocol based on TCP - Datadog needs to connect to it to get the metrics.

[iMajna]

Do you have any advice on verifying that port 9999 is configured to send metrics without curl?

You can always get answers on that one via downloading jmxterm and hook it up to Kafka. If you cannot reach it out from your local machine then I suggest to play a bit. One quick "solution" can be copy/pasting this .jar to one of the kafka pods (copy/paste it in /home/kafka directory as you will have access only there.) and try to connect to port 9999 and follow maybe this blog-post

Whether port is even exposed you can see it via running:
k get pod <name-of-kafka-pod` -oyaml

Least but not last, this annotation is working example for us:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: <name_here>
spec:
  kafka:
    logging:
      type: inline
      loggers:
        kafka.root.logger.level: "INFO"
    jmxOptions: {}
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: kafka-metrics-config.yml
          name: kafka-zk-metrics
...
    template:
      pod:
        metadata:
          annotations:
            ad.datadoghq.com/kafka.check_names: '["kafka"]'
            ad.datadoghq.com/kafka.init_configs: |
              [
                {
                  "is_jmx": true,
                  "collect_default_metrics": true,
                  "collect_default_jvm_metrics": true,
                  "conf": [
                    {
                      "include": {
                        "domain": "kafka.log",
                        "bean_regex": "kafka\\.log:type=Log,name=Size,topic=(.*?),partition=(.*?)(?:,|$)",
                        "tags": { "topic": "$1", "partition": "$2" },
                        "attribute": {
                          "Value": {
                            "alias": "kafka.log.partition.size",
                            "metric_type": "gauge"
                          }
                        }
                      }
                    },
                    {
                      "include": {
                        "domain": "kafka.cluster",
                        "bean_regex": "kafka\\.cluster:type=Partition,name=UnderReplicated,topic=(.*?),partition=(.*?)(?:,|$)",
                        "tags": { "topic": "$1", "partition": "$2" },
                        "attribute": {
                          "Value": {
                            "alias": "kafka.cluster.partition.underreplicated",
                            "metric_type": "gauge"
                          }
                        }
                      }
                    },
                    {
                      "include": {
                        "domain": "kafka.controller",
                        "bean_regex": "kafka\\.controller:type=KafkaController,name=PreferredReplicaImbalanceCount",
                        "attribute": {
                          "Value": {
                            "alias": "kafka.controller.kafkacontroller.preferredreplicaimbalancecount",
                            "metric_type": "count"
                          }
                        }
                      }
                    },
                    {
                      "include": {
                        "domain": "kafka.controller",
                        "bean_regex": "kafka\\.controller:type=KafkaController,name=ActiveBrokerCount",
                        "attribute": {
                          "Value": {
                            "alias": "kafka.controller.kafkacontroller.activebrokercount",
                            "metric_type": "count"
                          }
                        }
                      }
                    },
                    {
                      "include": {
                        "domain": "kafka.server",
                        "bean": "kafka.server:type=app-info",
                        "attribute": {
                          "version": {
                            "alias": "kafka.server.cluster.version",
                            "metric_type": "gauge",
                            "values": {
                              "default": 1
                            }
                          }
                        }
                      }
                    },
                    {
                      "include": {
                        "domain": "kafka.controller",
                        "bean_regex": "kafka\\.controller:type=KafkaController,name=FencedBrokerCount",
                        "attribute": {
                          "Value": {
                            "alias": "kafka.controller.kafkacontroller.fencedbrokercount",
                            "metric_type": "count"
                          }
                        }
                      }
                    }
                  ]
                }
              ]
            ad.datadoghq.com/kafka.instances: '[{"host": "%%host%%","port":"9999","max_returned_metrics": 1000}]'
            ad.datadoghq.com/kafka.logs: '[{"source":"kafka","service":"kafka"}]'
...

Hope it helps!

[helloimowen]

Thanks so much for the help! My actual issue here was unrelated - was not using the specific -jmx tagged datadog agent image.

[romankydybets]

this is how we did

and datadog docs about openmetrics
https://docs.datadoghq.com/containers/kubernetes/prometheus/?tab=kubernetesadv2

also keep we allow our datadog agent take this metrics from strimzi namespace

here is the snipped or kafka-exporter from

kafkaExporter:
    template:
      pod:
        # taking this from this thread
        # https://github.com/orgs/strimzi/discussions/6258
        metadata:
            annotations: 
              ad.datadoghq.com/<kafka-exporter-container-name>.checks: |
                {
                  "openmetrics": {
                    "init_config": {},
                    "instances": [
                      {
                        "openmetrics_endpoint": "http://%%host%%:9404/metrics",
                        "namespace": "<your namespace>",
                        "metrics": [
                          {"go_gc_duration_seconds":"go_gc_duration_seconds"},
                          {"go_goroutines":"go_goroutines"},
                          {"go_memstats_alloc_bytes":"go_memstats_alloc_bytes"},
                          {"go_memstats_alloc_bytes_total":"go_memstats_alloc_bytes_total"},
                          {"go_memstats_buck_hash_sys_bytes":"go_memstats_buck_hash_sys_bytes"},
                          {"go_memstats_frees_total":"go_memstats_frees_total"},
                          {"go_memstats_gc_sys_bytes":"go_memstats_gc_sys_bytes"},
                          {"go_memstats_heap_alloc_bytes":"go_memstats_heap_alloc_bytes"},
                          {"go_memstats_heap_idle_bytes":"go_memstats_heap_idle_bytes"},
                          {"go_memstats_heap_inuse_bytes":"go_memstats_heap_inuse_bytes"},
                          {"go_memstats_heap_objects":"go_memstats_heap_objects"},
                          {"go_memstats_heap_released_bytes_total":"go_memstats_heap_released_bytes_total"},
                          {"go_memstats_heap_sys_bytes":"go_memstats_heap_sys_bytes"},
                          {"go_memstats_last_gc_time_seconds":"go_memstats_last_gc_time_seconds"},
                          {"go_memstats_lookups_total":"go_memstats_lookups_total"},
                          {"go_memstats_mallocs_total":"go_memstats_mallocs_total"},
                          {"go_memstats_mcache_inuse_bytes":"go_memstats_mcache_inuse_bytes"},
                          {"go_memstats_mcache_sys_bytes":"go_memstats_mcache_sys_bytes"},
                          {"go_memstats_mspan_inuse_bytes":"go_memstats_mspan_inuse_bytes"},
                          {"go_memstats_mspan_sys_bytes":"go_memstats_mspan_sys_bytes"},
                          {"go_memstats_next_gc_bytes":"go_memstats_next_gc_bytes"},
                          {"go_memstats_other_sys_bytes":"go_memstats_other_sys_bytes"},
                          {"go_memstats_stack_inuse_bytes":"go_memstats_stack_inuse_bytes"},
                          {"go_memstats_stack_sys_bytes":"go_memstats_stack_sys_bytes"},
                          {"go_memstats_sys_bytes":"go_memstats_sys_bytes"},
                          {"kafka_brokers":"kafka_brokers"},
                          {"kafka_consumergroup_current_offset":"kafka_consumergroup_current_offset"},
                          {"kafka_consumergroup_lag":"kafka_consumergroup_lag"},
                          {"kafka_exporter_build_info":"kafka_exporter_build_info"},
                          {"kafka_topic_partition_in_sync_replica":"kafka_topic_partition_in_sync_replica"},
                          {"kafka_topic_partition_current_offset":"kafka_topic_partition_current_offset"},
                          {"kafka_topic_partition_leader":"kafka_topic_partition_leader"},
                          {"kafka_topic_partition_leader_is_preferred":"kafka_topic_partition_leader_is_preferred"},
                          {"kafka_topic_partition_oldest_offset":"kafka_topic_partition_oldest_offset"},
                          {"kafka_topic_partition_replicas":"kafka_topic_partition_replicas"},
                          {"kafka_topic_partition_under_replicated_partition":"kafka_topic_partition_under_replicated_partition"},
                          {"kafka_topic_partitions":"kafka_topic_partitions"},
                          {"process_cpu_seconds_total":"process_cpu_seconds_total"},
                          {"process_max_fds":"process_max_fds"},
                          {"process_open_fds":"process_open_fds"},
                          {"process_resident_memory_bytes":"process_resident_memory_bytes"},
                          {"process_start_time_seconds":"process_start_time_seconds"},
                          {"process_virtual_memory_bytes":"process_virtual_memory_bytes"}
                        ]
                      }
                    ]
                  }
                }