Define JMX metrics "conventions"

[SylvainJuge]

The goal of this issue is to describe (in comments) all the challenges/compromises found while trying to align the JMX yaml definitions in instrumentation/jmx-metrics/library.

The current migration effort involves the following steps for each supported system:

• move the existing yaml file into library
• include metrics that are part of jmx-scraper/jmx-gatherer if needed
• try to make the metrics fit the semantic conventions
  • metric names lowercase and split with dots
  • namespace metrics per system
  • namespace metrics attributes or reuse existing semconv attributes
  • units of time duration should be in seconds
  • existing semconv attributes should be reused whenever possible

[SylvainJuge]

JVM runtime metrics #13392

• metrics are defined in semantic conventions so there wasn't any discussion needed on the metrics to capture and/or their definitions.
• some of the metrics can't be captured through JMX metrics in YAML as they rely on JMX notifications which require dedicated code (histograms). For example: jvm.gc.duration.
• some of the metrics attributes can't be captured through JMX metrics in YAML, so the metrics do not have all the recommended attributes, which is quite similar to the java8/java17 runtime versions that do not produce identical metric attributes. For example jvm.thread.count metric do not have jvm.thread.daemon or jvm.thread.state metric attributes.
• experimental metrics are included in the YAML definition, there is no config option to enable/disable them unlike the ones provides in runtime instrumentation. For example jvm.memory.init

[SylvainJuge]

Tomcat metrics: #13650

• pre-aggregated metric tomcat.request.duration.max :
  • .max as suffix
  • gauge metric type, even if it never decreases, as the value can't be aggregated
• pre-aggregated metric to represent total duration tomcat.request.duration
  • no .total suffix to fit "do not use total" semconv rule.
  • This potentially prevents later providing an histogram of each request duration with the same metric name, if we pick an .total suffix we don't have this issue.
  • This probably need further discussion.
• use distinct metrics for tomcat.thread.count and tomcat.thread.busy.count when breakdown by a metric attribute is not possible (because there is no direct reporting of the "idle thread count".

[SylvainJuge]

Jetty metrics: #14067

• aggregation over multiple MBean instances document & test jmx metric implicit aggregation #14111
• some metrics had to be removed because we can't capture them reliably when more than 1 MBean instance is present without mapping all the MBean name parameters to metric attributes (which are technical IDs)

[SylvainJuge]

Wildfly metrics: #14208

• using wildfly.db.client.connection. namespace to reuse db.client.connection from semconv

[SylvainJuge]

I'll close this issue as we handle most of the changes in the JMX metrics on a case-by-case basis and it's quite hard/impossible to define generic rules that cover all of them. Also, most the choices regarding metric names and their mapping are dependent on each of the supported systems.