Skip to content

Conversation

@mitali-salvi
Copy link
Contributor

@mitali-salvi mitali-salvi commented Oct 3, 2024

Issue #, if available:

Description of changes:

  1. Implemented custom provider for OtelJMX metrics insights
  2. Is enabled when AWS_JMX_ENABLED is set to true
  3. Sets up a metrics exporter over 4314 port only over HTTP protocol

Testing
Tested as part of amazon-cloudwatch-observability helm-chart

  1. Annotate a pod deploying a sprint boot project with instrumentation.opentelemetry.io/inject-java: "true" cloudwatch.aws.amazon.com/inject-jmx-jvm: "true" and deploy on the cluster
  2. Deploy the helm-chart with a custom amazon-cloudwatch-operator build to inject the following environment vars
OTEL_EXPORTER_OTLP_PROTOCOL
OTEL_METRICS_EXPORTER
OTEL_LOGS_EXPORTER
AWS_JMX_EXPORTER_METRICS_ENDPOINT
OTEL_JMX_TARGET_SYSTEM
  1. Verified AwsJMX exporter is enabled in pod logs
 INFO software.amazon.opentelemetry.javaagent.providers.AwsJMXMetricsCustomizerProvider - AWS JMX metric collection enabled

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@mitali-salvi mitali-salvi requested a review from a team as a code owner October 3, 2024 14:54
@mitali-salvi mitali-salvi changed the base branch from runtime-feature to main October 4, 2024 21:19
Copy link
Member

@bjrara bjrara left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to verify how well works with Application Signals runtime metrics.

@bjrara
Copy link
Member

bjrara commented Oct 11, 2024

Please update the PR with the verification result of the following test cases as there's no UT added for those changes:

  1. OTEL_METRICS_EXPORTER is none, OTEL_AWS_JMX_EXPORTER_METRICS_ENDPOINT is not set -> no metrics are exported.
  2. OTEL_METRICS_EXPORTER is none, OTEL_AWS_JMX_EXPORTER_METRICS_ENDPOINT is set, OTEL_JMX_TARGET_SYSTEM is jvm -> only JMX JVM metrics are exported to AWS JMX EP, no other metrics are exported.
  3. OTEL_METRICS_EXPORTER is not none (e.g. logging), OTEL_AWS_JMX_EXPORTER_METRICS_ENDPOINT is set, OTEL_JMX_TARGET_SYSTEM is jvm -> only JMX JVM metrics are exported to AWS JMX EP, all metrics are exported to the default exporter, JMX JVM is dual published.
  4. OTEL_METRICS_EXPORTER is not none (e.g. logging), OTEL_AWS_JMX_EXPORTER_METRICS_ENDPOINT is not set -> no metrics are exported to AWS JMX EP, all metrics are exported to the default exporter.
  5. OTEL_METRICS_EXPORTER is not none (e.g. logging), otel.jmx.enabled is disabled, OTEL_AWS_JMX_EXPORTER_METRICS_ENDPOINT is set, OTEL_JMX_TARGET_SYSTEM is jvm -> same as 4, except JMX JVM metrics are not exported to the default exporter.

@musa-asad
Copy link
Contributor

  1. Tested and it didn't emit any metrics as expected.
- env:
        - name: OTEL_METRICS_EXPORTER
          value: none
  1. JVM metrics received:
    Screenshot 2024-10-11 at 6 33 24 PM
- env:
        - name: OTEL_METRICS_EXPORTER
          value: none
        - name: OTEL_AWS_JMX_EXPORTER_METRICS_ENDPOINT
          value: http://cloudwatch-agent.amazon-cloudwatch:4314/v1/metrics
        - name: OTEL_JMX_TARGET_SYSTEM
          value: jvm
  1. JVM metrics received:
    Same as above.

We got errors in the logging to show it was trying to send metrics to the default OTLP exporter endpoint.
Screenshot 2024-10-11 at 6 34 36 PM

- env:
        - name: OTEL_METRICS_EXPORTER
          value: otlp
        - name: OTEL_AWS_JMX_EXPORTER_METRICS_ENDPOINT
          value: http://cloudwatch-agent.amazon-cloudwatch:4314/v1/metrics
        - name: OTEL_JMX_TARGET_SYSTEM
          value: jvm
  1. Same error and no metrics.
- env:
        - name: OTEL_METRICS_EXPORTER
          value: otlp
        - name: OTEL_JMX_TARGET_SYSTEM
          value: jvm
  1. Same error and no metrics like above.
- env:
        - name: OTEL_METRICS_EXPORTER
          value: otlp
        - name: OTEL_JMX_ENABLED
          value: "false"
        - name: OTEL_AWS_JMX_EXPORTER_METRICS_ENDPOINT
          value: http://cloudwatch-agent.amazon-cloudwatch:4314/v1/metrics
        - name: OTEL_JMX_TARGET_SYSTEM
          value: jvm

@bjrara bjrara merged commit 81e799b into aws-observability:main Oct 11, 2024
4 checks passed
@mitali-salvi mitali-salvi deleted the jmx-feature branch October 14, 2024 10:57
@jefchien jefchien mentioned this pull request Oct 23, 2024
bjrara pushed a commit that referenced this pull request Nov 1, 2024
*Description of changes:* Adds contract tests to verify the expected
behavior of the following changes:
-
#817
-
#898
-
#901

Starts up sample apps with the instrumented SDK and verifies that
metrics are received for each of the supported JMX target systems.

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants