Grpc Server metrics rpc_server_duration contains high cardinality labels of network_peer_port

[jianwu]

Describe the bug

GRPC Server metrics: rpc_server_duration contains high cardinality labels cardinality labels network_peer_port. As port number can change for every request. The total time series will be huge which makes the metrics not very useful.

Steps to reproduce

• Start a standard grpc application with OTel Java Agent enabled and enabled prometheus metrics server at port 9464.
• Issue grpc request continuously, (In our case, the Kubernetes just issue health check every few seconds).
• Look at the metrics page: "http://localhost:9464"
• Observe the page size is huge with tens thousands metrics line looks like:

rpc_server_duration_milliseconds_bucket{network_peer_address="127.0.0.1",network_peer_port="32924",network_type="ipv4",otel_scope_name="io.opentelemetry.grpc-1.6",otel_scope_version="2.4.0-alpha",rpc_grpc_status_code="0",rpc_method="Check",rpc_service="grpc.health.v1.Health",rpc_system="grpc",server_address="",server_port="50051",le="0.0"} 0

The reason of huge number of entries is because it contains high cardinality labels like network_peer_port

Expected behavior

For prometheus metrics, we should avoid high cardinality labels.

Actual behavior

There are high cardinality labels.

Javaagent or library instrumentation version

1.38.0

Environment

JDK: 17
OS: linux

Additional context

No response

[laurit]

@jianwu could you provide a minimal sample application that reproduces the issue along with any necessary instructions.

[heyams]

@laurit is there a way to use system property to remove certain attributes from the metric instrumentation, something like "Dotel.instrumentation.metrics.experimental", that would be a cool feature?

[laurit]

@laurit is there a way to use system property to remove certain attributes from the metric instrumentation, something like "Dotel.instrumentation.metrics.experimental", that would be a cool feature?

As far as I know there isn't. There are metrics views that can be used to drop attributes but they don't use system properties. It is probably possible to build system property based metric dropping on top of the metrics view configuration. That being said I don't think that this would help here. We use metric advice to limit the set of attributes added to the metric, having the peer port show up is unexpected. I find it unlikely that this would happen with just using the agent, there is probably something else that is needed to run into this.

[github-actions]

This has been automatically marked as stale because it has been marked as needing author feedback and has not had any activity for 7 days. It will be closed automatically if there is no response from the author within 7 additional days from this comment.

[t-peoples]

I'm seeing this problem as well:

WARNING: Instrument rpc.server.duration has exceeded the maximum allowed cardinality

This is mainly due to huge cardinality on the network_peer_port attribute, but the network_peer_address attribute can easily grow to a very large cardinality as well for a service with a large number of clients.

@laurit I don't have permission to reopen this issue, but I think these two attributes should be opt-in, perhaps similar to captureProtocolAttributes and captureLocalSocketAttributes?

[jaydeluca]

@t-peoples the best way to get attention to your problem is to open a new issue, and you could reference this one if you believe it is related

[laurit]

This is mainly due to huge cardinality on the network_peer_port attribute, but the network_peer_address attribute can easily grow to a very large cardinality as well for a service with a large number of clients.

These should not be added by default. Please create a new issue, include details needed to reproduce the issue.