App insights not exporting jmx metrics that are available later

[numirov]

We have confluent kafka connect to which we are attaching app insights java agent. We want to expose connector metrics through jmx. We added them as described in the app insights doc, confluent doc:
{ "jmxMetrics": [ { "name": "Connector JMX - type", "objectName": "kafka.connect:type=connector-metrics,connector=connector-name", "attribute": "connector-type" }, { "name": "Test Connector JMX - version", "objectName": "kafka.connect:type=connector-metrics,connector=connector-name", "attribute": "connector-version" }, { "name": "Test Connector JMX - class", "objectName": "kafka.connect:type=connector-metrics,connector=connector-name", "attribute": "connector-class" }, { "name": "Test Connector JMX - status(sandbox)", "objectName": "kafka.connect:type=connector-metrics,connector=connector-name", "attribute": "status" }, { "name": "Test java iometrics - count", "objectName": "java.nio:name=mapped,type=BufferPool", "attribute": "Count" }, { "name": "test connector - failed startup", "objectName": "kafka.connect:type=connect-worker-metrics", "attribute": "connector-startup-failure-total" }, { "name": "test connector - success startup", "objectName": "kafka.connect:type=connect-worker-metrics", "attribute": "connector-startup-success-total" } ], "selfDiagnostics": { "level": "DEBUG" } }
So from above only kafka.connect:type are not appearing in the portal.
We checked confluent connect using kafka tools and it is showing that above mentioned connect specific metrics are being exposed. We observed app insights logs and could see that connect metrics are not available straight away(can't see it here: INFO c.m.a.a.i.p.AvailableJmxMetricLogger - available jmx metrics:) but rather later when it spins them up one by one( can see them here INFO c.m.a.a.i.p.AvailableJmxMetricLogger - newly available jmx metrics since last output:). So we have a feeling that jmx metrics which are available later are not being picked up? Cause we can't see them in the portal.

Expected behavior

Confluent connect metrics are available

Actual behavior

Confluent connect metrics are not available

To Reproduce

Submitting a PR with an example reproducing the issue in this repository would make it easier for the Application Insight maintainers to help you. Before doing this, you have to fork this repository.

System information

Please provide the following information:

• SDK Version: eclipse temurin jdk 17
• OS type and version: App insights 3.6.0
• Application Server type and version (if applicable):
• Using spring-boot? no
• Additional relevant libraries (with version, if applicable):

Logs

Turn on SDK logs and attach/paste them to the issue. If using an application server, also attach any relevant server logs.

⚠ BE SURE TO REMOVE ANY PRIVATE INFORMATION FROM THE LOGS BEFORE POSTING ⚠

Screenshots

If applicable, add screenshots to help explain your problem.

[jeanbisutti]

@numirov Do you mean that you can't see these Kafka JMX metrics as soon as the application starts but can see them later in the portal?

Some JMX metrics can take time to be available.

[numirov]

hi @jeanbisutti , no, metrics are not visible in the portal later. Even though we saw log form app insights that those metrics are available some time after initial startup they don't appear in app insights ui as described in the doc. Although we can see other metrics.

[jeanbisutti]

@numirov Can you please send the Application Insights logs at the debug level to jeanbisutti@microsoft.com?

[numirov]

@jeanbisutti I have emailed it

[jeanbisutti]

@numirov Thank you for the logs. I have replied you by email.

It seems an issue with ObjectName configuration. I have suggested to use wildcard (https://docs.oracle.com/javase/8/docs/api/javax/management/ObjectName.html) to ease ObjectName configuration.