Skip to content

Commit e144053

Browse files
musa-asadbjrara
andauthored
Add missing JMX metrics for ContainerInsights. (#898)
### Description of changes * Added missing metrics from the ContainerInsights [dashboard](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-metrics.html#ContainerInsights-Prometheus-metrics-jmx) not included in the [agent](https://github.com/aws-observability/aws-otel-java-instrumentation/blob/c39b76830931899a678f72d502dafe33329adde5/instrumentation/jmx-metrics/src/main/resources/jmx/rules/jvm.yaml)'s solution for JMX. * Compared the current spec files for JVM and Tomcat with the dashboard to see what was missing. * Set up an EC2 cluster, a Tomcat application, and JMXTerm to find the bean and metric names for the missing metrics. * Added these to the spec file based on appropriate types and the naming convention from [OTel](https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/jmx-metrics/javaagent/README.md). * Then, the ADOT Jar image was built and tested with the `amazon-cloudwatch-agent-operator` and `helm-charts` on an EKS cluster with a custom Tomcat deployment, which correctly produced the correct metrics: * `./scripts/local_patch.sh && ./gradlew build` * `docker build --platform linux/amd64 -t adot-autoinstrumentation-java .` * `[add docker image to ECR]` ### Manual testing For testing, I decided to use this built image with the CloudWatch Agent to see if these metrics are able to be emitted and tracked on the CW console. #### Configuration ``` { "metrics": { "namespace": "tomcat", "metrics_collected": { "jmx": { "jvm": { "measurement": [ "jvm.system.swap.space.total", "jvm.system.swap.space.free", "jvm.system.physical.memory.total", "jvm.system.physical.memory.free", "jvm.system.available.processors", "jvm.system.cpu.utilization", "jvm.open_file_descriptor.count", "jvm.daemon_threads.count", "jvm.threads.count" ] }, "tomcat": { "measurement": [ "tomcat.rejected_sessions", "tomcat.sessions" ] } } } } } ``` #### CW Console <img width="949" alt="Screenshot 2024-10-04 at 11 08 51 AM" src="https://github.com/user-attachments/assets/e9f3869b-f17d-4d17-81b5-0c0b97272012"> By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: Mengyi Zhou (bjrara) <[email protected]>
1 parent 51098f9 commit e144053

File tree

3 files changed

+48
-10
lines changed

3 files changed

+48
-10
lines changed

instrumentation/jmx-metrics/src/main/resources/README.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,4 +10,7 @@ OTEL_EXPERIMENTAL_METRICS_VIEW_CONFIG: classpath:/jmx/view.yaml
1010

1111
### rules/*.yaml
1212
The rules are a translation of the JMX Metric Gatherer's [target systems](https://github.com/open-telemetry/opentelemetry-java-contrib/tree/main/jmx-metrics/src/main/resources/target-systems)
13-
based on the [JMX metric rule YAML schema](https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/jmx-metrics/javaagent/README.md#basic-syntax).
13+
based on the [JMX metric rule YAML schema](https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/jmx-metrics/javaagent/README.md#basic-syntax).
14+
15+
### SystemCpuLoad
16+
The `SystemCpuLoad` metric is deprecated and Java versions 14+ now use `CpuLoad`. However, to avoid emitting double metrics, we stick to using `SystemCpuLoad` as it still works on newer versions.

instrumentation/jmx-metrics/src/main/resources/jmx/rules/jvm.yaml

Lines changed: 34 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -75,21 +75,50 @@ rules:
7575
desc: The maximum amount of memory can be used for the memory pool
7676
- bean: java.lang:type=Threading
7777
unit: "1"
78-
prefix: jvm.threads.
7978
type: gauge
8079
mapping:
8180
ThreadCount:
82-
metric: count
81+
metric: jvm.threads.count
8382
desc: Number of threads
83+
DaemonThreadCount:
84+
metric: jvm.daemon_threads.count
85+
desc: Number of daemon threads
8486
- bean: java.lang:type=OperatingSystem
85-
prefix: jvm.cpu.
8687
type: gauge
8788
mapping:
89+
TotalSwapSpaceSize:
90+
metric: jvm.system.swap.space.total
91+
desc: The host swap memory size in bytes
92+
unit: by
93+
FreeSwapSpaceSize:
94+
metric: jvm.system.swap.space.free
95+
desc: The amount of available swap memory in bytes
96+
unit: by
97+
TotalPhysicalMemorySize:
98+
metric: jvm.system.physical.memory.total
99+
desc: The total physical memory size in host
100+
unit: by
101+
FreePhysicalMemorySize:
102+
metric: jvm.system.physical.memory.free
103+
desc: The amount of free physical memory in host
104+
unit: by
105+
AvailableProcessors:
106+
metric: jvm.system.available.processors
107+
desc: The number of available processors
108+
unit: "1"
109+
SystemCpuLoad:
110+
metric: jvm.system.cpu.utilization
111+
desc: The current load of CPU in host
112+
unit: "1"
88113
ProcessCpuTime:
89-
metric: time
114+
metric: jvm.cpu.time
90115
unit: ns
91116
desc: CPU time used
92117
ProcessCpuLoad:
93-
metric: recent_utilization
118+
metric: jvm.cpu.recent_utilization
94119
unit: "1"
95120
desc: Recent CPU utilization for the process
121+
OpenFileDescriptorCount:
122+
metric: jvm.open_file_descriptor.count
123+
desc: The number of opened file descriptors
124+
unit: "1"

instrumentation/jmx-metrics/src/main/resources/jmx/rules/tomcat.yaml

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,15 @@ rules:
33
- bean: Catalina:type=Manager,host=localhost,context=*
44
metricAttribute:
55
context: param(context)
6+
unit: sessions
7+
type: gauge
68
mapping:
79
activeSessions:
810
metric: tomcat.sessions
9-
type: gauge
10-
unit: sessions
1111
desc: The number of active sessions.
12+
rejectedSessions:
13+
metric: tomcat.rejected_sessions
14+
desc: The number of rejected sessions.
1215
- bean: Catalina:type=GlobalRequestProcessor,name=*
1316
metricAttribute:
1417
name: param(name)
@@ -68,12 +71,15 @@ rules:
6871
- bean: Tomcat:type=Manager,host=localhost,context=*
6972
metricAttribute:
7073
context: param(context)
74+
unit: sessions
75+
type: gauge
7176
mapping:
7277
activeSessions:
7378
metric: tomcat.sessions
74-
type: gauge
75-
unit: sessions
7679
desc: The number of active sessions.
80+
rejectedSessions:
81+
metric: tomcat.rejected_sessions
82+
desc: The number of rejected sessions.
7783
- bean: Tomcat:type=GlobalRequestProcessor,name=*
7884
metricAttribute:
7985
name: param(name)

0 commit comments

Comments
 (0)