Reduce cardinality on built-in metrics

[NatMarchand]

Hey there !
We're currently developing a multi-tenant app (up to 1k tenants per app instance). Each tenant is bound to a dns (tenantX.ourapp.com). Therefore, as we use built-in metrics from httpclient, we see a high cardinality as we have a lot of server.address values.
Is there a way to aggregate at the source ?
Ideally, we'd like to replace the values tenant1.ourapp.com, tenant2.ourapp.com, etc. by a placeholded value : {tenant}.ourapp.com

[martincostello]

Here's an example of a potential approach you could use from something I added to a personal application of mine: code.

In my case I'm adding a cosmetic "display name" with peer.service based on the value of server.address, but I think you could use the same principle to do what you propose with a placeholder and edit whichever metric it is that you wish to reduce the cardinality of by adjusting the tags.

The core functionality is to configure HttpClientTraceInstrumentationOptions.EnrichWithHttpRequestMessage with a custom callback, and then change the tags on the activity.

[NatMarchand]

Unless I'm missing something, I don't think server.adress tag of a metric is coming from the activity/trace span, am I right ?
https://github.com/dotnet/runtime/blob/179473d3c8a1012b036ad732d02804b062923e8d/src/libraries/System.Net.Http/src/System/Net/Http/Metrics/MetricsHandler.cs#L144C14-L149C14

[martincostello]

You're right that what I'm doing is for Traces, not Metrics, sorry.

Looking at the code you've linked to, looks like if there are any callbacks registered then they're called before the metric is recorded.

Unfortunately it doesn't look like the TagList (which has add/remove functionality) is publicly exposed for editing, there's only an AddCustomTag() method, which I don't think will overwrite an existing tag.

I'm not immediately aware of any other way there might be a way to customise the metrics at the point they're generated/recorded - maybe that could be a feature request to the .NET runtime for .NET 10?

I don't think sampling would help you either, as that would reduce the volume but not the cardinality.

Let me have a dig around to see if there's a similar hook in the OpenTelemetry libraries to edit the tags for metrics.

[martincostello]

Yeah, I couldn't immediately find a hook anywhere to edit the tags for the metrics.

[NatMarchand]

Yeah I tried this way, I used reflection to get the tags list, little bit hacky but why not ?
However, it seems that it is not taken in account. I will end up dropping the tag for now.

[cijothomas]

You can use Views to drop a tag completely, but there isn't anything that allows injecting the custom logic you want.

There is some work in OTel spec to add a concept of "measurement-processor", which would likely allow a hook to customize the tag/attribute values.
open-telemetry/opentelemetry-specification#4318

Sorry, no better solution as of now. If you end up finding some way around, do post back.

[martincostello]

Maybe I can try to open a PR on dotnet repo ?

New APIs for .NET itself should start with an API Suggestion. There might still be time to get it into .NET 10 if there's support for it and it gets approved.

[NatMarchand]

Well, after reviewing the spec from opentelemetry, it seems that it fits our needs. Not sure if needed to open a suggestion. What do you think ?

[martincostello]

PRs to add new APIs to .NET won't be merged without an associated approved new API Suggestion - it's their process.

[NatMarchand]

Sorry, I wasn't clear. I meant, as an issue is open in otel spec, the work will be done in the future in the otel repo. It might not be useful to do the work also on the dotnet side.

[cijothomas]

MeasurementProcessor PR in the spec, if approved, will likely be a pure OTel repo implementation without requiring any change in DotNet repo.