[jmx-gatherer] Transformation Closure not working

[john-hyun-eb]

Component(s)

jmx-metrics

What happened?

Description

The transformation closure feature, and thus the ability to utilize non-numeric metrics, is not working. The feature was added via !960 but I believe some interaction with !949 is causing the problem. Not sure if this is the cause but InstrumentHelper.groovy is invoking the static MBeanHelper.getBeanAttribute which ignores transformations.

Steps to Reproduce

Run a simple Java program that waits forever and exposes a JMX port.

Run the jmx gatherer with the following groovy collection script:

`def jvmMetrics = otel.mbean("java.lang:type=Runtime",
["my.mbean.jvm.vendor": { mbean -> mbean.getProperty("VmVendor") == "Eclipse Adoptium" ? 1 : 0 }]
)

otel.instrument(jvmMetrics,
"my.metric.jvm.vendor",
"my.mbean.jvm.vendor",
otel.&longValueCallback)
`

Expected Result

If run on a Termurin JDK it should report a my.metric.jvm.vendor value of 1

Actual Result

The logs show:

WARNING: Expected attribute my.mbean.jvm.vendor not found in mbean java.lang:type=Runtime

Component version

1.45.0

Log output

No response

Additional context

No response

[SylvainJuge]

Hi @john-hyun-eb,

From your description you would like to capture a metric that has the value of 0 when the vendor is not Eclipse Adoptium and has a value of 1 otherwise.

The example provided in the documentation here refers to the mapping of a metric attribute, not the metric value.

As you are using a custom groovy script, I would strongly suggest to attempt using jmx-scraper (which relies on instrumentation jmx-metrics module) as an alternative as it provides the ability to define metrics in YAML (simpler to write and maintain, also more secure as it's not executable like groovy) and it also includes the ability to capture "state metrics" which seems to be what you are trying to capture here.

As an alternative, you might as well attempt to capture a custom metric value with jmx-gatherer in groovy, but I am not very familiar with this part of the API so you might have to experiment a bit on your own here.

[john-hyun-eb]

Sorry, the VmVendor example was just a simplistic example someone could easily repeat on their own. I'm actually trying to capture whether a Kafka Connect task (not Kafka or Kafka consumers, brokers - but Connect) is failed or not. The actual mbean (pattern) is "kafka.connect:type=connector-task-metrics,connector=”{connector}”,task=”{task}”" and the attribute is "status" which can have the values unassigned, running, paused, failed, or destroyed. I want the value failed to map to 1 and the rest mapped to 0 so I know when a task is failed. Is this not the use case the documentation is describing?

As for the jmx-scraper I'd like to use it but the documentation says it is a "work-in-progress component not ready to be used in production". I am currently using the gatherer in production. But yes the "state metrics" is exactly what I'm trying to do.

As for "you might as well attempt to capture a custom metric value with jmx-gatherer in groovy" I thought this was what the transformation closure was for. I've been trying various different things and there does not seem to be an execution path that results in the transformation being invoked. The instrumentation helper references the static getBeanAttribute which ignores the transformation and any custom metric that might be produced from it.

[breedx-splk]

As for the jmx-scraper I'd like to use it but the documentation says it is a "work-in-progress component not ready to be used in production". I am currently using the gatherer in production. But yes the "state metrics" is exactly what I'm trying to do.

Hey @john-hyun-eb just so you know -- the gatherer is also not marked "stable" (it's alpha), and our plan in the near future is to remove the jmx-metrics (gatherer) from the contrib repo completely and only have the jmx-scraper. With that, please have a look at it.

[SylvainJuge]

One major difference between JMX scraper and JMX gatherer is that JMX scraper relies on the instrumentation/jmx-metric implementation that is used in the instrumentation agent, the latter should be considered "production ready" as it has been there for a while. As a side note, I don't know if there is an official per-feature maturity status in the instrumentation agent.

We have reached a stage where jmx-scraper allows to capture metrics that are equivalent to the ones currently captured by JMX gatherer (using the otel.jmx.target.system configuration option), so maybe it is the right time to change this misleading wording. Also maybe it could be a good time to write a proper migration guide from JMX Gatherer to ensure smooth transition.

So, if you agree, I would suggest to do the following:

• remove the "do not use in production" in the readme
• document migration from JMX Gatherer to JMX Scraper

[john-hyun-eb]

I guess I'll just go the Scraper route then. It would be nice to know whether the Transformation Closure actually works since I spent some time trying to figure it out, but if the Scraper is the intended long term solution I'll switch over to that. Thanks!

[SylvainJuge]

In case you have any issue with jmx-scraper, don't hesitate to open issues or discussions as feedback is always welcome, I'd be happy to help if needed here.