Flushing the same metric multiple times in the same second overwrites previous values

[scmorse]

If you increment a counter, flush the message, and then before the UTC timestamp has rolled into the next second, increment the same counter with the same tags (and flush again), only one of your two increments will show up in datadog. I think Datadog has some de-duplication / unique indexing that only tracks a unique metric per second. I think this library would need to keep in-memory up to 1 second of messages, to be able to track and send the right stats.

[eeubank]

@scmorse we were seeing that same issue with this library. Since it seems to have been abandoned we've switched to datadog-lambda-js and have had better results.

[Mr0grog]

I just finished a PR that adds distribution metrics, which won’t have this problem (datadog-lambda-js only sends distributions, which is how it works around this issue). If the core issue here is still causing you problems and doesn’t get addressed before the next release, you might try using a distribution instead.

[Mr0grog]

For what it’s worth, I’d be happy to accept a PR for this. At first blush, I imagine a decent approach would be to keep everything inside lib/aggregators.js.

• In flush(), just keep a copy of all the flushed metric objects (not their serialized output!) whose timestamp matches the current second.
• In addPoint(), if the second hasn’t rolled over since those were recorded, look up the metric object from that set of copies before looking it up from this.buffer.
• Next time flush() is called, add any of those old copies that have been dirtied to list of flushed metrics.

I’m going to treat this as a new feature rather than a bug, since this is more-or-less expected behavior on Datadog’s end. I believe you’d see the same thing with their agent, for example, if you forced it to flush twice in one second.

This feature is most relevant when you just happen to have auto-flushed less than a second before your program exits (at which point you are hopefully calling a manual flush!).

However, if you are in a situation where you are manually flushing super frequently or you have a lot of parallel processes running with the same tags and host setting (or starting and stopping a process multiple times a second), you should really be using distributions instead of standard gauge/counter/histogram metrics, as noted in the earlier comment. That just means calling metrics.distribution(name, value) in this library instead of metrics.gauge(name, value).