Micrometer not providing memory or cpu metrics when compiled to native image

[fage88]

I am trying to implement observability on my application using MicroMeter and OTLP.

When I run locally and look at the available metrics I see all kinds of great CPU and memory metrics, but when I compile to native and deploy I don't see any memory metrics at all and I see NAN for the system cpu.

Is there a config I need to turn on to get cpu and memory metrics for a GraalVm image?

Currently I have

quarkus.native.enable-vm-inspection                                                            = true
quarkus.micrometer.binder.jvm                                                                  = true
quarkus.micrometer.binder.system                                                               = true
quarkus.micrometer.binder.vertx.enabled                                                        = true
quarkus.micrometer.binder.http-client.enabled                                                  = true
quarkus.micrometer.binder.http-server.enabled                                                  = true

and that renders output like this at /q/metrics

# HELP agroal_acquire_count_total Number of times an acquire operation succeeded.
# TYPE agroal_acquire_count_total counter
agroal_acquire_count_total{datasource="default",} 274.0
# HELP agroal_flush_count_total Number of connections removed from the pool, not counting invalid / idle.
# TYPE agroal_flush_count_total counter
agroal_flush_count_total{datasource="default",} 0.0
# HELP agroal_destroy_count_total Number of destroyed connections.
# TYPE agroal_destroy_count_total counter
agroal_destroy_count_total{datasource="default",} 0.0
# HELP agroal_creation_time_total_milliseconds Total time waiting for connections to be created.
# TYPE agroal_creation_time_total_milliseconds gauge
agroal_creation_time_total_milliseconds{datasource="default",} 16.0
# HELP http_server_requests_seconds  
# TYPE http_server_requests_seconds summary
http_server_requests_seconds_count{method="GET",outcome="SUCCESS",status="200",uri="/rest/platform/v1/site/{platform}/{name}/{status}",} 16.0
http_server_requests_seconds_sum{method="GET",outcome="SUCCESS",status="200",uri="/rest/platform/v1/site/{platform}/{name}/{status}",} 2.255190471
# HELP http_server_requests_seconds_max  
# TYPE http_server_requests_seconds_max gauge
http_server_requests_seconds_max{method="GET",outcome="SUCCESS",status="200",uri="/rest/platform/v1/site/{platform}/{name}/{status}",} 0.368301545
# HELP jvm_threads_peak_threads The peak live thread count since the Java virtual machine started or peak was reset
# TYPE jvm_threads_peak_threads gauge
jvm_threads_peak_threads 17.0
# HELP jvm_gc_overhead_percent An approximation of the percent of CPU time used by GC activities over the last lookback period or since monitoring began, whichever is shorter, in the range [0..1]
# TYPE jvm_gc_overhead_percent gauge
jvm_gc_overhead_percent 0.0
# HELP process_files_max_files The maximum file descriptor count
# TYPE process_files_max_files gauge
process_files_max_files 1048576.0
# HELP agroal_creation_time_max_milliseconds Maximum time for a connection to be created.
# TYPE agroal_creation_time_max_milliseconds gauge
agroal_creation_time_max_milliseconds{datasource="default",} 16.0
# HELP http_server_bytes_written_max  
# TYPE http_server_bytes_written_max gauge
http_server_bytes_written_max 8192.0
# HELP http_server_bytes_written  
# TYPE http_server_bytes_written summary
http_server_bytes_written_count 1226.0
http_server_bytes_written_sum 9652578.0
# HELP agroal_active_count Number of active connections. These connections are in use and not available to be acquired.
# TYPE agroal_active_count gauge
agroal_active_count{datasource="default",} 1.0
# HELP agroal_max_used_count Maximum number of connections active simultaneously.
# TYPE agroal_max_used_count gauge
agroal_max_used_count{datasource="default",} 1.0
# HELP process_uptime_seconds The uptime of the Java virtual machine
# TYPE process_uptime_seconds gauge
process_uptime_seconds 96.242
# HELP agroal_blocking_time_total_milliseconds Total time applications waited to acquire a connection.
# TYPE agroal_blocking_time_total_milliseconds gauge
agroal_blocking_time_total_milliseconds{datasource="default",} 17.0
# HELP agroal_invalid_count_total Number of connections removed from the pool for being idle.
# TYPE agroal_invalid_count_total counter
agroal_invalid_count_total{datasource="default",} 0.0
# HELP agroal_blocking_time_average_milliseconds Average time an application waited to acquire a connection.
# TYPE agroal_blocking_time_average_milliseconds gauge
agroal_blocking_time_average_milliseconds{datasource="default",} 0.0
# HELP agroal_available_count Number of idle connections in the pool, available to be acquired.
# TYPE agroal_available_count gauge
agroal_available_count{datasource="default",} 0.0
# HELP agroal_creation_count_total Number of created connections.
# TYPE agroal_creation_count_total counter
agroal_creation_count_total{datasource="default",} 1.0
# HELP jvm_threads_live_threads The current number of live threads including both daemon and non-daemon threads
# TYPE jvm_threads_live_threads gauge
jvm_threads_live_threads 15.0
# HELP jvm_info_total JVM version info
# TYPE jvm_info_total counter
jvm_info_total{runtime="unknown",vendor="Oracle Corporation",version="unknown",} 1.0
# HELP system_cpu_count The number of processors available to the Java virtual machine
# TYPE system_cpu_count gauge
system_cpu_count 1.0
# HELP jvm_threads_daemon_threads The current number of live daemon threads
# TYPE jvm_threads_daemon_threads gauge
jvm_threads_daemon_threads 11.0
# HELP http_server_connections_seconds_max  
# TYPE http_server_connections_seconds_max gauge
http_server_connections_seconds_max 2.627015749
# HELP http_server_connections_seconds  
# TYPE http_server_connections_seconds summary
http_server_connections_seconds_active_count 3.0
http_server_connections_seconds_duration_sum 5.239655883
# HELP system_cpu_usage The "recent cpu usage" of the system the application is running in
# TYPE system_cpu_usage gauge
system_cpu_usage NaN
# HELP agroal_blocking_time_max_milliseconds Maximum time an application waited to acquire a connection.
# TYPE agroal_blocking_time_max_milliseconds gauge
agroal_blocking_time_max_milliseconds{datasource="default",} 16.0
# HELP jvm_classes_loaded_classes The number of classes that are currently loaded in the Java virtual machine
# TYPE jvm_classes_loaded_classes gauge
jvm_classes_loaded_classes 0.0
# HELP jvm_classes_unloaded_classes_total The total number of classes unloaded since the Java virtual machine has started execution
# TYPE jvm_classes_unloaded_classes_total counter
jvm_classes_unloaded_classes_total 0.0
# HELP agroal_awaiting_count Approximate number of threads blocked, waiting to acquire a connection.
# TYPE agroal_awaiting_count gauge
agroal_awaiting_count{datasource="default",} 0.0
# HELP process_start_time_seconds Start time of the process since unix epoch.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.658868771795E9
# HELP agroal_leak_detection_count_total Number of times a leak was detected. A single connection can be detected multiple times.
# TYPE agroal_leak_detection_count_total counter
agroal_leak_detection_count_total{datasource="default",} 0.0
# HELP process_cpu_usage The "recent cpu usage" for the Java Virtual Machine process
# TYPE process_cpu_usage gauge
process_cpu_usage NaN
# HELP agroal_reap_count_total Number of connections removed from the pool for being idle.
# TYPE agroal_reap_count_total counter
agroal_reap_count_total{datasource="default",} 0.0
# HELP process_files_open_files The open file descriptor count
# TYPE process_files_open_files gauge
process_files_open_files 29.0
# HELP agroal_creation_time_average_milliseconds Average time for a connection to be created.
# TYPE agroal_creation_time_average_milliseconds gauge
agroal_creation_time_average_milliseconds{datasource="default",} 16.0

I know graalvm is not really using a JVM so I was expecting some difference when running as a native image, but I was surprised to see no CPU or Memory metrics at all.
Is there another extension or config I should be using?

[quarkus-bot[bot]]

/cc @ebullient, @jmartisk

[ebullient]

Substrate VM does not (to my knowledge) support MemoryPoolMXBeans (oracle/graal#4510)

Quarkus avoids loading this binder:

quarkus/extensions/micrometer/runtime/src/main/java/io/quarkus/micrometer/runtime/MicrometerRecorder.java

Line 116 in 314c240

if (!ImageInfo.inImageCode()) {

Which would no-op itself at bind time because the MemoryPoolMxBeans aren't present):
https://github.com/micrometer-metrics/micrometer/blob/fd775bf43c15b4e0d25d8eea18112ef1510069c8/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/jvm/JvmGcMetrics.java#L250

So any metrics emitted by that binder won't be present in native image. Others might not fail, but may not be populated.

See also spring-attic/spring-native#1006 (comment)

[fage88]

Hi @ebullient

Thank you for the fast response. That is too bad, but obviously out of the Quarkus teams control.

-Jeff