Record metrics with random sampling for hot codepaths

[Serophots]

I wondered whether it would be within the scope of this crate to provide an API where metric recordings were randomly sampled, i.e. say only 10% of histogram records actually go through as a mechanism for saving performance on hot codepaths whilst retaining representative data. I am imagining histogram!(...).record_p10(...) or perhaps with the particular percentile passed as an argument.

This could very easily be implemented on the users side and perhaps should be left at that.

[tobz]

Definitely a helper utility that would live outside of the core types/macros. Something we could easily add to metrics-util.

My main question would be: are you thinking about this from a call overhead standpoint, or an amount-of-data-recorded-and-sent standpoint? (or something else?)

[Serophots]

I was thinking about it from the perspective of reducing call overhead although admittedly I don't have a very good idea about how large this overhead is and whether it is worth mitigating.

I just ran a very crude test with a prometheus exporter which indicates it would save on call overhead enough to make me interested for my production use case

[tobz]

Gotcha, that makes sense.

Yeah, I think there's probably a world where a helper could improve on that in terms of a cheap PRNG that provides the sampling behavior. If you could hold the handle to the metric, that will be hard to beat in terms of overhead.

[Serophots]

Ah, I had not thought of holding onto the metrics instances. That seems to improve things significantly for me where I won't need this sampling technique. Thanks for your help.

[tobz]

Ah, nice! Yeah, looking up a metric and then performing the operation might end up being like 5x - 10x slower than just holding the handle and doing it that.

Glad that helped you get over the bottleneck you were hitting.