Metrics OOM issues

[jdmarshall]

I was experimenting with converting some telemetry code to OpenTelemetry with the newly merged metrics code and was running into some CPU problems. One of the first things I did was to construct a benchmark to try to narrow down problems, and fairly quickly ran into OOM issues with that (particularly in CI).

With log rotation implementations, it's a common feature to allow log rotation on an interval but also with a threshold, so that at high rates the logs are rotated sooner. As near as I can tell, there is no such facility in sdk-metrics, which might explain a lot of the behavior of my errant process. I'm not entirely clear how I would go about fixing that. The code that handles the interval seems to be entirely divorced from the code doing the collection.

[legendecas]

Would you mind providing a minimum reproducible of your problem? I'm not sure I get a clear view of the issue.

[jdmarshall]

Hey, forgot to follow up.

Problem turned out to me an impedance mismatch with the way you call statsd, which we are moving away from.

With OT, creating a counter with the same name ends up accumulating multiple instances, and the overhead of dealing with them goes up linearly. From my tracing through the code I had mistaken some reuse logic being at the metric level but it's actually one level down from there.

I had to use a lookup table to avoid creating the same counter/histogram twice.