Support a publishing filter for PushMeterRegistry

[allnightlong]

Please describe the feature request.

Introduce the filter, that could include/exclude meters from being published.

Rationale
Currently, all the meters registered in MeterRegistry are published on each step.
But for some meters (counters) with big tags cardinality the delta between steps is usually zero. It means, that registry is publishing lots of zero-values. This process consumes CPU, network, storage on the backend without adding any value.

Additional context
In my situation, I'm using newrelic as a backend, and about 80-90% of published events are zero-based.

[jonatan-ivanov]

Thank you for the issue!
Publishing zeros are by design. There is a semantical difference between not having a time-series and having it but not being updated. For example: an application is down (and does not publish metrics) or it is up but it does not receive traffic. That's why I would argue that it does not add any value, I think it does and users are using this feature.

If you have high cardinality, I think that's a different issue that needs a different solution, please see: https://develotters.com/posts/high-cardinality/

If you don't want to go that way, here are a few things you can do:

• You can filter those Meters out using a MeterFilter (see docs about MeterFilter)
• You can remove meters using MeterRegistry#remove
• In general, you can extend the MeterRegistry and drop what you don't need in publish, in New Relic's case, you can provide your own NewRelicClientProvider that does the filtering
• Or if you use the OTLP registry, you can implement your own MetricsSender that does the filtering

[allnightlong]

Hi, @jonatan-ivanov, thank you for the answer.

Sorry for not being specific about "adding value", I meant that it doesn't add much value for my specific app. And yes, I'm aware, that high-cardinality sometimes considers as an anti-pattern, but I'm working with a big legacy application, that is not easy to quickly refactor.

I also tried the suggested approaches, but unfortunately neither of them work in my case:

Meter Filters
Filters work on registration step only, not on publish step. My goal is to have counters in registry, but publish data only if for incremented counters during last "tick-window". MeterFilters can't do that, they don't even have access to meter's value (rather to MeterId only). And yes, ideally, I would like to have some kind of Filter to decide, if meter should be included into published data according to it's state.

MeterRegistry#remove
I've tried to cleanup registry before publish data (poor's man garbage collector for unupdated meters). But this approach has a significant flow: the counter produced by OtlpMeterRegistry relies on StepCounter (which uses StepDouble inherited from StepValue internally). The way StepValue works is it has previous value from last "tick" and current value in DoubleAdder. And micrometer only updates previous value with current once the "tick" happens (time-window shifts by the step). It means, that even if counter.count() returns 0, it still could contains non-zero value in current variable, and by removing this counter, we would lose metrics data.
And I couldn't find a way to workaround this problem apart from very dirty reflection hacks.

override MeterRegistry
I use not NewRelicMeterRegistry, but rather vendor-agnostic OtlpMeterRegistry. And I've got an impression, that it's intentionally was made hard to override:

• It has private, not protected constructor

private OtlpMeterRegistry(OtlpConfig config, Clock clock, ThreadFactory threadFactory, OtlpMetricsSender metricsSender) 

so my overriding class can't call super(), when I need to provide custom send (and I need to do it - see #7009).

• It has Builder to set meterSender, but this builder only returns OtlpMeterRegistry - so no overriding here as well.
• Even if I manage to create my custom overriding class, I would need to use OtlpMetricConverter in my publish() method, but this class has package-level visibility, so I would need to copy-paste it in my code. And then I would need to update it each time I bump micrometer version.

custom MetricsSender
Considered that as well - this class receives OtlpMetricsSender.Request as an input, where payload data is already encoded in byte[]. So if I want to hook in, I would need to convert bytes back to classes (otel protobuf classes in this case), do the filtering, and convert it back to bytes. It's super-dirty, super-inefficient (from CPU and memory point of view), and hard to maintain in a log run, since it could be broken on library version update.

So I would really appreciate a consideration of a conventional filter, or at least make OtlpMeterRegistry less difficult to extend.

Thank you.

[github-actions]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.