Unable to fetch JMX metrics

[Shubhamb7]

Hi,
I am facing issue fetching the jmx metrics. Can someone help me with this? Thankyou.

I am running Kafka Broker with jmx_exporter. Below is the service file for the same.

[Unit]
Description=Apache Kafka Server
Requires=network-online.target
[Service]
User=root
Group=root
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64"
Environment="KAFKA_OPTS=-javaagent:/opt/monitoring/jmx_prometheus_javaagent-0.18.0.jar=7071:/opt/monitoring/kafka-jmx.yml"
ExecStart=/bin/sh -c '/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties > /opt/kafkadata/systemlogs/kafka.log 2>&1'
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal
[Install]
WantedBy=multi-user.target

Below is the yaml file for kafka ui

logging:
level:
root: INFO
com.provectus: DEBUG
#org.springframework.http.codec.json.Jackson2JsonEncoder: DEBUG
#org.springframework.http.codec.json.Jackson2JsonDecoder: DEBUG
reactor.netty.http.server.AccessLog: INFO
kafka:
clusters:
- name: Cluster-01
bootstrapServers: 10.25.1.233:9092,10.25.1.220:9092,10.25.1.56:9092
zookeeper: 10.25.1.185:2181,10.25.1.217:2181,10.25.1.9:2181
kafkaConnect:
- name: Connect
address: http://10.25.1.23:8083
metrics:
port: 7071
type: JMX

But when I run the kafka-ui jar, it is not able to fetch the jmx metrics

http://:8080/api/clusters/Cluster-01/metrics -> {"items":[]}

Below is the error log

2023-03-27 08:39:00,870 ERROR [boundedElastic-3] c.p.k.u.s.m.JmxMetricsRetriever: Error connecting to service:jmx:rmi:///jndi/rmi://10.25.1.56:7071/jmxrmi
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
java.net.SocketTimeoutException: Read timed out]
at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:370)
at com.provectus.kafka.ui.service.metrics.JmxMetricsRetriever.withJmxConnector(JmxMetricsRetriever.java:79)
at com.provectus.kafka.ui.service.metrics.JmxMetricsRetriever.retrieveSync(JmxMetricsRetriever.java:66)
at com.provectus.kafka.ui.service.metrics.JmxMetricsRetriever.lambda$retrieve$0(JmxMetricsRetriever.java:52)
at reactor.core.publisher.MonoSupplier.call(MonoSupplier.java:86)
at reactor.core.publisher.FluxSubscribeOnCallable$CallableSubscribeOnSubscription.run(FluxSubscribeOnCallable.java:227)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: javax.naming.CommunicationException: null
at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:138)
at java.naming/com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:220)
at java.naming/javax.naming.InitialContext.lookup(InitialContext.java:409)
at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1839)
at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1813)
at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:302)
... 12 common frames omitted
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
java.net.SocketTimeoutException: Read timed out
at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:308)
at java.rmi/sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:204)
at java.rmi/sun.rmi.server.UnicastRef.newCall(UnicastRef.java:344)
at java.rmi/sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:116)
at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:134)
... 17 common frames omitted
Caused by: java.net.SocketTimeoutException: Read timed out
at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:283)
at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:309)
at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)
at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)
at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966)
at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:263)
at java.base/java.io.DataInputStream.readByte(DataInputStream.java:271)
at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:247)
... 21 common frames omitted

[Haarolean]

Hi, please share your kafka-ui configs

[Shubhamb7]

Hi, below is the yml file I'm using while running the kafka-ui jar

logging:
  level:
    root: INFO
    com.provectus: DEBUG
    #org.springframework.http.codec.json.Jackson2JsonEncoder: DEBUG
    #org.springframework.http.codec.json.Jackson2JsonDecoder: DEBUG
    reactor.netty.http.server.AccessLog: INFO

kafka:
  clusters:
    - name: Cluster-01
      bootstrapServers: 10.25.1.104:9092,10.25.1.185:9092,10.25.1.183:9092
      zookeeper: 10.25.1.123:2181,10.25.1.68:2181,10.25.1.21:2181
      kafkaConnect:
        - name: Connect
          address: http://10.25.1.113:8083
      metrics:
        port: 7071
        type: JMX

This is how I'm running the jar

java -Dspring.config.additional-location=provectus-app.yml -jar kafka-ui-api-v0.6.0.jar

When I tried to fetch the metrics

ubuntu@kafka-ui:~$ curl http://localhost:8080/api/clusters/
[{"name":"Cluster-01","defaultCluster":null,"status":"online","lastError":null,"brokerCount":3,"onlinePartitionCount":54,"topicCount":3,"bytesInPerSec":null,"bytesOutPerSec":null,"readOnly":false,"version":"3.3-IV3","features":["KAFKA_CONNECT","TOPIC_DELETION"]}]

ubuntu@kafka-ui:~$ curl http://localhost:8080/api/clusters/Cluster-01/metrics
{"items":[]}

[Haarolean]

You're using prometheus rather than JMX.
Please set KAFKA_CLUSTERS_0_METRICS_TYPE: PROMETHEUS property.

[Shubhamb7]

Okay I replaced the JMX with PROMETHEUS and it worked. Thankyou!!

[Haarolean]

@Shubhamb7 you're welcome :)