State Metrics for JMX MBeans not generated with OpenTelemetry Java Agent 2.17.0

[jadolgl]

Describe the bug

When using the OpenTelemetry Java Agent version 2.17.0, state metrics configured via JMX rules are no longer being generated. This issue was not present in version 2.16.0, where the same configuration successfully produced the expected state metrics.

Specifically, for a Tomcat server using the provided tomcat.yaml JMX rules, the http.server.tomcat.serverInfo state metric is not created. In version 2.16.0, a "Created" log message would appear for this metric, which is absent in 2.17.0.

Steps to reproduce

1. Configure a Tomcat server with the following JMX rules (tomcat.yaml):

tomcat.yaml

rules:
  - bean: Catalina:type=Server
    unit: '1'
    prefix: http.server.tomcat.
    mapping:
      serverInfo:
        type: state
        metric: serverInfo
        metricAttribute:
          version: beanattr(serverInfo)
          running:
            yes: '*'
  - bean: java.lang:type=Runtime
    unit: '1'
    prefix: http.server.tomcat.jvm.
    mapping:
      VmInfo:
        type: state
        metric: vminfo
        metricAttribute:
          version: beanattr(VmVersion)
          vendor: beanattr(VmVendor)
          running:
            yes: '*'
  - bean: Catalina:type=Loader,host=localhost,context=*
    unit: '1'
    prefix: http.server.tomcat.
    mapping:
      stateName:
        type: state
        metric: context
        metricAttribute:
          context: param(context)
          state:
            ok: STARTED

2. Start Tomcat with following Java Agent arguments
   -Dotel.jmx.config=<path_to_tomcat.yaml> -Dotel.exporter.otlp.endpoint=http://localhost:10087 -Dotel.jmx.target.system=tomcat

Expected behavior

State metrics, particularly http.server.tomcat.serverInfo, should be created and exported as observed in version 2.16.0. A log message indicating the creation of these metrics (e.g., related to Created serverInfo metric for bean 'Catalina:type=Server') should appear in the debug output.

Actual behavior

No state metrics are generated. The expected "Created" log messages for state metrics (like serverInfo) are absent from the debug output. The agent starts without any explicit errors related to JMX metric generation, but the metrics simply do not appear.

Javaagent or library instrumentation version

opentelemetry-javaagent - version: 2.17.0

Environment

JDK: Running on Java 17.0.15. JVM OpenJDK 64-Bit Server VM - Azul Systems, Inc. - 17.0.15+6-LTS
OS: SLES 15SP5

Additional context

No response

[laurit]

cc @SylvainJuge

[SylvainJuge]

Thanks @jadolgl , you've found a regression in 2.17.0 for JMX state metrics.

I have opened #14194 to fix it, the only work-around is to use 2.16.0 for now.

Also, starting with 2.17.0, the metrics captured when otel.jmx.target.system=tomcat have changed so I would recommend to:

• get a copy of tomcat.yaml found here to use newer definitions
• use this copy of tomcat.yaml with the otel.jmx.config option
• modify tomcat.yaml to add your own metrics (or use a separate yaml file)
• remove tomcat value in otel.jmx.target.system configuration

In addition, regarding the metrics definitions that you provided as an example, I would suggest to modify them like this:

  - bean: Catalina:type=Server
    # explicit unit can be removed with 2.17.0 or later
    unit: ''
    mapping:
      stateName:
        type: state
        # use 'tomcat.*' namespace to align with 2.17.0 metrics
        metric: tomcat.server
        metricAttribute:
          # also use 'tomcat.*' namespace for metric attributes for consistency with 2.17.0 metrics
          tomcat.version: beanattr(serverInfo)
          tomcat.running:
            # using state metrics is more relevant when we have at least two states
            # as each state will be mapped to a metric with a distinct metric attribute value
            yes: 'STARTED'
            no: '*'

Also, the VmInfo MBean attribute does not seem to exist for the java.lang:type=Runtime MBean, if you need to capture the JVM version it should already be captured in resource attributes.