jmx add jvm metrics yaml

instrumentation/jmx-metrics/library/jvm.md

@@ -0,0 +1,34 @@
+# JVM Metrics
+
+Here is the list of metrics based on MBeans exposed by the JVM and that are defined in [`jvm.yaml`](./src/main/resources/jmx/rules/jvm.yaml).
+
+Those metrics are defined in the [JVM runtime metrics semantic conventions](https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/).
+
+| Metric Name                                                                                                                           | semconv maturity | Type          | Attributes                            | Description                                        |
+|---------------------------------------------------------------------------------------------------------------------------------------|:-----------------|---------------|---------------------------------------|----------------------------------------------------|
+| [jvm.memory.used](https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/#metric-jvmmemoryused)                              | stable           | UpDownCounter | jvm.memory.pool.name, jvm.memory.type | Used memory                                        |
+| [jvm.memory.committed](https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/#metric-jvmmemorycommitted)                    | stable           | UpDownCounter | jvm.memory.pool.name, jvm.memory.type | Committed memory                                   |
+| [jvm.memory.limit](https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/#metric-jvmmemorylimit)                            | stable           | UpDownCounter | jvm.memory.pool.name, jvm.memory.type | Max obtainable memory                              |
+| [jvm.memory.init](https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/#metric-jvmmemoryinit)                              | experimental     | UpDownCounter | jvm.memory.pool.name, jvm.memory.type | Initial memory requested                           |
+| [jvm.memory.used_after_last_gc](https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/#metric-jvmmemoryused_after_last_gc)  | stable           | UpDownCounter | jvm.memory.pool.name, jvm.memory.type | Memory used after latest GC                        |
+| [jvm.thread.count](https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/#metric-jvmthreadcount)                            | stable           | UpDownCounter | [^1]                                  | Threads count                                      |
+| [jvm.class.loaded](https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/#metric-jvmclassloaded)                            | stable           | Counter       |                                       | Classes loaded since JVM start                     |
+| [jvm.class.unloaded](https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/#metric-jvmclassunloaded)                        | stable           | Counter       |                                       | Classes unloaded since JVM start                   |
+| [jvm.class.count](https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/#metric-jvmclasscount)                              | stable           | UpDownCounter |                                       | Classes currently loaded count                     |
+| [jvm.cpu.count](https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/#metric-jvmcpucount)                                  | stable           | UpDownCounter |                                       | Number of CPUs available                           |
+| [jvm.cpu.recent_utilization](https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/#metric-jvmcpurecent_utilization)        | stable           | Gauge         |                                       | Recent CPU utilization for process reported by JVM |
+| [jvm.system.cpu.load_1m](https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/#metric-jvmsystemcpuload_1m)                 | experimental     | Gauge         |                                       | Average CPU load reported by JVM                   |
+| [jvm.system.cpu.recent_utilization](https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/#metric-jvmcpurecent_utilization) | experimental     | Gauge         |                                       | Recent CPU utilization reported by JVM             |
+| [jvm.buffer.memory.used](https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/#metric-jvmbuffermemoryused)                 | experimental     | UpDownCounter | jvm.buffer.pool.name                  | Memory used by buffers                             |
+| [jvm.buffer.memory.limit](https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/#metric-jvmbuffermemorylimit)               | experimental     | UpDownCounter | jvm.buffer.pool.name                  | Maximum memory usage for buffers                   |
+| [jvm.buffer.memory.count](https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/#metric-jvmbuffermemorycount)               | experimental     | UpDownCounter | jvm.buffer.pool.name                  | Buffers count                                      |
+
+## Limitations and unsupported metrics
+
+There are a few limitations to the JVM metrics that are captured through the JMX interface with declarative YAML.
+Using the [runtime-telemetry](../../runtime-telemetry) modules with instrumentation allow to capture metrics without those limitations.
+
+[^1]: `jvm.thread.daemon` and `jvm.thread.state` attributes are not supported.
+
+- [jvm.gc.duration](https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/#metric-jvmgcduration) metric is not supported as it is only exposed through JMX notifications which are not supported with YAML.
+- [jvm.cpu.time](https://opentelemetry.io/docs/specs/semconv/runtime/jvm-metrics/#metric-jvmcputime) metric is not supported yet due to lack of unit conversion, see [#13369](https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/13369) for details.

instrumentation/jmx-metrics/library/src/main/resources/jmx/rules/jvm.yaml

@@ -0,0 +1,120 @@
+---
+
+rules:
+
+  - bean: java.lang:type=MemoryPool,name=*
+    prefix: jvm.memory.
+    type: updowncounter
+    unit: By
+    metricAttribute:
+      jvm.memory.pool.name: param(name)
+      jvm.memory.type: lowercase(beanattr(Type))
+    mapping:
+      # jvm.memory.used
+      Usage.used:
+        metric: used
+        desc: Measure of memory used.
+      # jvm.memory.committed
+      Usage.committed:
+        metric: committed
+        desc: Measure of memory committed.
+      # jvm.memory.limit
+      Usage.max:
+        metric: limit
+        desc: Measure of max obtainable memory.
+      # jvm.memory.init (experimental)
+      Usage.init:
+        metric: init
+        desc: Measure of initial memory requested.
+      # jvm.memory.used_after_last_gc
+      # note: metric attribute "jvm.memory.type" will always be "heap" as GC only manages heap
+      CollectionUsage.used:
+        metric: used_after_last_gc
+        desc: Measure of memory used, as measured after the most recent garbage collection event on this pool.
+
+  - bean: java.lang:type=Threading
+    prefix: jvm.thread.
+    mapping:
+      # jvm.thread.count
+      # limitation: 'jvm.thread.daemon' and 'jvm.thread.state' metric attributes are not provided
+      ThreadCount:
+        metric: count
+        type: updowncounter
+        unit: "{thread}"
+        desc: Number of executing platform threads.
+
+  - bean: java.lang:type=ClassLoading
+    prefix: jvm.class.
+    type: updowncounter
+    unit: "{class}"
+    mapping:
+      # jvm.class.loaded
+      TotalLoadedClassCount:
+        metric: loaded
+        desc: Number of classes loaded since JVM start.
+      # jvm.class.unloaded
+      UnloadedClassCount:
+        metric: unloaded
+        desc: Number of classes unloaded since JVM start.
+      # jvm.class.count
+      LoadedClassCount:
+        metric: count
+        desc: Number of classes currently loaded.
+
+  - bean: java.lang:type=OperatingSystem
+    prefix: jvm.
+    dropNegativeValues: true
+    mapping:
+      # jvm.cpu.count
+      AvailableProcessors:
+        metric: cpu.count
+        type: updowncounter
+        unit: "{cpu}"
+        desc: Number of processors available to the Java virtual machine.
+      # jvm.cpu.time
+      ProcessCpuTime:
+        metric: cpu.time
+        type: counter
+        sourceUnit: ns
+        unit: s
+        desc: CPU time used by the process as reported by the JVM.
+      # jvm.cpu.recent_utilization
+      ProcessCpuLoad:
+        metric: cpu.recent_utilization
+        type: gauge
+        unit: '1'
+        desc: Recent CPU utilization for the process as reported by the JVM.
+      # jvm.system.cpu.load_1m (experimental)
+      SystemLoadAverage:
+        metric: system.cpu.load_1m
+        type: gauge
+        unit: "{run_queue_item}"
+        desc: Average CPU load of the whole system for the last minute as reported by the JVM.
+      # jvm.system.cpu.utilization (experimental)
+      SystemCpuLoad:
+        metric: system.cpu.utilization
+        type: gauge
+        unit: '1'
+        desc: Recent CPU utilization for the whole system as reported by the JVM.
+
+  - bean: java.nio:name=*,type=BufferPool
+    prefix: jvm.buffer.
+    type: updowncounter
+    metricAttribute:
+      jvm.buffer.pool.name: param(name)
+    mapping:
+      # jvm.buffer.memory.used (experimental)
+      MemoryUsed:
+        metric: memory.used
+        unit: By
+        desc: Measure of memory used by buffers.
+      # jvm.buffer.memory.limit (experimental)
+      TotalCapacity:
+        metric: memory.limit
+        unit: By
+        desc: Measure of total memory capacity of buffers.
+      # jvm.buffer.count (experimental)
+      Count:
+        metric: count
+        unit: "{buffer}"
+        desc: Number of buffers in the pool.

instrumentation/jmx-metrics/library/src/test/java/io/opentelemetry/instrumentation/jmx/rules/JvmTargetSystemTest.java

@@ -0,0 +1,208 @@
+/*
+ * Copyright The OpenTelemetry Authors
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+package io.opentelemetry.instrumentation.jmx.rules;
+
+import static io.opentelemetry.instrumentation.jmx.rules.assertions.DataPointAttributes.attribute;
+import static io.opentelemetry.instrumentation.jmx.rules.assertions.DataPointAttributes.attributeGroup;
+import static io.opentelemetry.instrumentation.jmx.rules.assertions.DataPointAttributes.attributeWithAnyValue;
+
+import io.opentelemetry.instrumentation.jmx.rules.assertions.AttributeMatcher;
+import io.opentelemetry.instrumentation.jmx.rules.assertions.AttributeMatcherGroup;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.wait.strategy.Wait;
+
+class JvmTargetSystemTest extends TargetSystemTest {
+
+  @ParameterizedTest
+  @ValueSource(
+      strings = {
+        // openj9 image that might have slight differences for JVM metrics
+        "tomcat:jdk8-adoptopenjdk-openj9",
+        // basic tomcat image with standard hotspot jdk
+        "tomcat:9.0"
+      })
+  void testJvmMetrics(String image) {
+    List<String> yamlFiles = Collections.singletonList("jvm.yaml");
+
+    yamlFiles.forEach(this::validateYamlSyntax);
+
+    List<String> jvmArgs = new ArrayList<>();
+    jvmArgs.add(javaAgentJvmArgument());
+    jvmArgs.addAll(javaPropertiesToJvmArgs(otelConfigProperties(yamlFiles)));
+
+    // testing with a basic tomcat image as test application to capture JVM metrics
+    GenericContainer<?> target =
+        new GenericContainer<>(image)
+            .withEnv("CATALINA_OPTS", String.join(" ", jvmArgs))
+            .withStartupTimeout(Duration.ofMinutes(2))
+            .withExposedPorts(8080)
+            .waitingFor(Wait.forListeningPorts(8080));
+
+    copyFilesToTarget(target, yamlFiles);
+
+    startTarget(target);
+
+    AttributeMatcher poolNameAttribute = attributeWithAnyValue("jvm.memory.pool.name");
+
+    AttributeMatcherGroup heapPoolAttributes =
+        attributeGroup(attribute("jvm.memory.type", "heap"), poolNameAttribute);
+
+    AttributeMatcherGroup nonHeapPoolAttributes =
+        attributeGroup(attribute("jvm.memory.type", "non_heap"), poolNameAttribute);
+
+    AttributeMatcher bufferPoolName = attributeWithAnyValue("jvm.buffer.pool.name");
+    verifyMetrics(
+        MetricsVerifier.create()
+            .add(
+                "jvm.memory.used",
+                metric ->
+                    metric
+                        .hasDescription("Measure of memory used.")
+                        .hasUnit("By")
+                        .isUpDownCounter()
+                        .hasDataPointsWithAttributes(heapPoolAttributes, nonHeapPoolAttributes))
+            .add(
+                "jvm.memory.committed",
+                metric ->
+                    metric
+                        .hasDescription("Measure of memory committed.")
+                        .hasUnit("By")
+                        .isUpDownCounter()
+                        .hasDataPointsWithAttributes(heapPoolAttributes, nonHeapPoolAttributes))
+            .add(
+                "jvm.memory.limit",
+                metric ->
+                    metric
+                        .hasDescription("Measure of max obtainable memory.")
+                        .hasUnit("By")
+                        .isUpDownCounter()
+                        .hasDataPointsWithAttributes(heapPoolAttributes, nonHeapPoolAttributes))
+            .add(
+                "jvm.memory.init",
+                metric ->
+                    metric
+                        .hasDescription("Measure of initial memory requested.")
+                        .hasUnit("By")
+                        .isUpDownCounter()
+                        .hasDataPointsWithAttributes(heapPoolAttributes, nonHeapPoolAttributes))
+            .add(
+                "jvm.memory.used_after_last_gc",
+                metric ->
+                    metric
+                        .hasDescription(
+                            "Measure of memory used, as measured after the most recent garbage collection event on this pool.")
+                        .hasUnit("By")
+                        .isUpDownCounter()
+                        // note: there is no GC for non-heap memory
+                        .hasDataPointsWithAttributes(heapPoolAttributes))
+            .add(
+                "jvm.thread.count",
+                metric ->
+                    metric
+                        .hasDescription("Number of executing platform threads.")
+                        .hasUnit("{thread}")
+                        .isUpDownCounter()
+                        .hasDataPointsWithoutAttributes())
+            .add(
+                "jvm.class.loaded",
+                metric ->
+                    metric
+                        .hasDescription("Number of classes loaded since JVM start.")
+                        .hasUnit("{class}")
+                        .isUpDownCounter()
+                        .hasDataPointsWithoutAttributes())
+            .add(
+                "jvm.class.unloaded",
+                metric ->
+                    metric
+                        .hasDescription("Number of classes unloaded since JVM start.")
+                        .hasUnit("{class}")
+                        .isUpDownCounter()
+                        .hasDataPointsWithoutAttributes())
+            .add(
+                "jvm.class.count",
+                metric ->
+                    metric
+                        .hasDescription("Number of classes currently loaded.")
+                        .hasUnit("{class}")
+                        .isUpDownCounter()
+                        .hasDataPointsWithoutAttributes())
+            .add(
+                "jvm.cpu.count",
+                metric ->
+                    metric
+                        .hasDescription(
+                            "Number of processors available to the Java virtual machine.")
+                        .hasUnit("{cpu}")
+                        .isUpDownCounter()
+                        .hasDataPointsWithoutAttributes())
+            .add(
+                "jvm.cpu.time",
+                metric ->
+                    metric
+                        .hasDescription("CPU time used by the process as reported by the JVM.")
+                        .hasUnit("s")
+                        .isCounter()
+                        .hasDataPointsWithoutAttributes())
+            .add(
+                "jvm.cpu.recent_utilization",
+                metric ->
+                    metric
+                        .hasDescription(
+                            "Recent CPU utilization for the process as reported by the JVM.")
+                        .hasUnit("1")
+                        .isGauge()
+                        .hasDataPointsWithoutAttributes())
+            .add(
+                "jvm.system.cpu.load_1m",
+                metric ->
+                    metric
+                        .hasDescription(
+                            "Average CPU load of the whole system for the last minute as reported by the JVM.")
+                        .hasUnit("{run_queue_item}")
+                        .isGauge()
+                        .hasDataPointsWithoutAttributes())
+            .add(
+                "jvm.system.cpu.utilization",
+                metric ->
+                    metric
+                        .hasDescription(
+                            "Recent CPU utilization for the whole system as reported by the JVM.")
+                        .hasUnit("1")
+                        .isGauge()
+                        .hasDataPointsWithoutAttributes())
+            .add(
+                "jvm.buffer.memory.used",
+                metric ->
+                    metric
+                        .hasDescription("Measure of memory used by buffers.")
+                        .hasUnit("By")
+                        .isUpDownCounter()
+                        .hasDataPointsWithOneAttribute(bufferPoolName))
+            .add(
+                "jvm.buffer.memory.limit",
+                metric ->
+                    metric
+                        .hasDescription("Measure of total memory capacity of buffers.")
+                        .hasUnit("By")
+                        .isUpDownCounter()
+                        .hasDataPointsWithOneAttribute(bufferPoolName))
+            .add(
+                "jvm.buffer.count",
+                metric ->
+                    metric
+                        .hasDescription("Number of buffers in the pool.")
+                        .hasUnit("{buffer}")
+                        .isUpDownCounter()
+                        .hasDataPointsWithOneAttribute(bufferPoolName)));
+  }
+}