Skip to content

Conversation

robsunday
Copy link
Contributor

@robsunday robsunday commented Aug 12, 2025

Fixes #14274
Includes:

  • Hadoop JMX metrics semconv alignment
  • Moved yaml with JMX metrics mapping to the library
  • Integration tests for Hadoop 2.x and Hadoop 3.x


| Metric Name | Type | Attributes | Description |
|---------------------------------|---------------|-------------------------------------|--------------------------------------------------------|
| hadoop.dfs.capacity | UpDownCounter | hadoop.node.name | Current raw capacity of data nodes. |
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[for reviewer] Naming prefix of metrics has been change to utilize a metric context (dfs).
Context is described in official docs: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/Metrics.html

@otelbot-java-instrumentation
Copy link
Contributor

🔧 The result from spotlessApply was committed to the PR branch.

@robsunday robsunday changed the title Hadoop jmx metics semconv alignment [JMX Insight] Hadoop jmx metics semconv alignment Aug 13, 2025
@robsunday robsunday marked this pull request as ready for review August 14, 2025 13:39
@robsunday robsunday requested a review from a team as a code owner August 14, 2025 13:39

| Metric Name | Type | Attributes | Description |
|---------------------------------|---------------|-------------------------------------|--------------------------------------------------------|
| hadoop.dfs.capacity | UpDownCounter | hadoop.node.name | Current raw capacity of data nodes. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The capacity is in bytes, so maybe we should start adding a dedicated column for the units (then we can do the same for other metrics as follow-up).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also not sure if we should have a metric that is a prefix of another, maybe capacity.raw could work here to replicate the wording in the docs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not find any official recommendation regarding metric name being prefix of another metric, and in my opinion in this case hadoop.dfs.capacity.used is indeed a part of total hadoop.dfs.capacity.
I'd take it to SIG meeting, if this is confusing for others as well then I'll change hadoop.dfs.capacity to hadoop.dfs.capacity.raw.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On SIG meeting we talked about that. Indeed we should avoid using metric name as a prefix for another metric.
Also we decided to add .limit suffix, which is aligned with semconv recommendation.

@SylvainJuge SylvainJuge self-assigned this Aug 21, 2025
Comment on lines 57 to 68
# hadoop.dfs.data_node.live
NumLiveDataNodes:
metric: data_node.live
type: updowncounter
unit: "{node}"
desc: Number of data nodes which are currently live.
# hadoop.dfs.data_node.dead
NumDeadDataNodes:
metric: data_node.dead
type: updowncounter
unit: "{node}"
desc: Number of data nodes which are currently dead.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The datanode and namenode wording is quite widespread in the hadoop docs, so I think we could get rid of the _ (but this is probably my personal preference here), also those are metrics that relate to the cluster and the nodes, and not related to the dfs part, even though they are captured on the same MBean object as the others.

So I would suggest to simplify those to:

  • hadoop.datanode.live instead of hadoop.dfs.data_node.live
  • hadoop.datanode.dead instead of hadoop.dfs.data_node.dead

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hadoop docs are quite inconsistent here. I saw DataNode and NameNode pascal case notation in many places, so I converted them to snake case. This follows this recommendation (see bullet starting with "For each multi-word dot-delimited component") and to match other metric names, like jvm.file_descriptor.count, http.client.open_connections, etc...
I agree that lowercase is easier to read, so I'd gladly change it. The question is what others think about it...

I'll get rid of dfs segment - good observation.

@laurit laurit added this to the v2.21.0 milestone Sep 19, 2025
@trask trask merged commit 3df64da into open-telemetry:main Sep 19, 2025
89 checks passed
mznet pushed a commit to mznet/opentelemetry-java-instrumentation that referenced this pull request Sep 26, 2025
Co-authored-by: otelbot <[email protected]>
Co-authored-by: SylvainJuge <[email protected]>
Co-authored-by: Trask Stalnaker <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

jmx hadoop metrics update and align with semconv

4 participants