Open telemetry .net -> open telemetry collector -> Prometheus and counter metric woes

[headnoodle]

Hi, I am quite new to open telemetry, Prometheus,etc but have added code to my application that sends metrics to the open telemetry collector via these .net api packages. Everything seems to work fine but I have an issue with counters and how they are implemented in Prometheus.

I have a counter metric defined in .net that can contain one of a few tags, these are sometimes incremented before the first push to the collector starts so the first value that is added to Prometheus may not be 0. This is causing me problems as if I do an increase in promql it registers as 0 as it has nothing to compare it against, and when it does have enough data it only compares it against the initial value and not 0.

Other websites I have looked at suggest initialising all counters with 0 in the client app before sending real metrics, this would also mean that I would have to initialise any of the counters with labels as 0 too but luckily I know all the label combinations so that won’t be a huge issue.

my question is, is this possible? I tried to do an Add on the metric of 0 but that didn’t seem to work.

Another option is that I somehow hijack the otlp exporter so that it sends a zero for the metric the first time it is encounter for a specific label,and on the next push it would send the correct value, but I haven’t got a clue if I can inject that functionality into the existing libraries

Any ideas would be appreciated.

Dean.

[matiduck]

I faced the same issue, the only difference is that I use PrometheusExporter instead of OtlpEporter. Could anyone suggest an option to achieve that?

[utpilla]

@headnoodle Could you try your scenario by directly exporting to Prometheus instead of going through Collector and check if the issue persists even when directly sending metrics to Prometheus? You could refer to this sample app on how to send metrics directly to Prometheus: https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/metrics/getting-started-prometheus-grafana

[matiduck]

@utpilla, I made a sample application based on a documentation you provided (link). As I added ObservableCounter and Counter in this example app, first one is available in Prometheus's metric endpoint with 0 value, when latter isn't. I checked the code and found that Counter isn't picked up by exporter because it is not returned by GetMetricsBatch as it does not have any metric points assigned. As I correctly understood, both of the instruments are created differently because the differ in their temporality preference (Observable instrument is cumulative, regular one is delta). Therefore, during Observable instrument creation, InitializeZeroTagPointIfNotInitialized method is called and it creates initial point. which does not happen for regular Counter.
Is this intentional behavior that regular Counter does not have any initial metric point assigned?

Also, I tried to work around the problem by `initializing` a counter with 0 value (by calling counter.Add(0). Sadly, this approach does not work as to be able to increase a counter instrument needs to have some subscription first, which it does not have until MeterProviderSdk is created (by Prometheus's middleware). When there is not subscription, this update call is ignored.
I wasn't able to find any plausible solution so far that allows me to set initial value. I will try to find one, but I'm open to any suggestions.

Thank you in advance for your help and your time!

[matiduck]

@utpilla, thanks for quick response!
My expectation is that as soon as application starts, even if I don't call any method, my counter should be available on Prometheus endpoint with a default 0 value. This counter availability is important after a service restart, as currently Prometheus server is able to scrap this counter only after it increments for the first time (become visible), which could not happen immediately, resulting in no records in time series database during that time.
I'm also concerned whether current implementation could impact a counter resets. As our counter will only be available after one increment, meaning that other Prometheus functions like rate wouldn't be able to correctly identify the counter reset and wouldn't be able to show data accurately.

I hope that this answers your questions and clarify my goal.

[cijothomas]

I don't call any method, my counter should be available on Prometheus endpoint with a default 0 value

If nothing has been reported, what labels/dimensions should the 0 be reported with?

[matiduck]

@cijothomas, to be honest I haven't thought that far. My gut feeling (although I don't know how complicated it would be to achieve that) would be to initialize instrument with default value for every label used there as it ensures continuity of data points.

[cijothomas]

There is no way we would know all key/value pairs ahead of time, so that cannot be done.

Maybe the expectation is incorrect. Are you seeing such behavior in offiicial prometheus clients, but not in otel libraries?

[cijothomas]

https://github.com/prometheus-net/prometheus-net#when-are-metrics-published Could be a prometheus specific feature..

@matiduck I think if you want OTel to behave like Prometheus, then this needs to raised as a OTel spec issue! Currently, OTel spec does not require one to publish metrics without any values ever being reported!