Resetting label combination values back to 0

[sander-deryckere]

We're trying to track the count of objects with a number of custom labels with moderate cardinality, but combined it can cause a high cardinality of labels, and thus certain combinations of labels to disappear.

Generic example: customers with their state, city, their year of birth, the year/month they last bought something, year/month they last logged in, category of products that's most popular for them, ...

We have an async job that calculates those counts every few minutes, and exports them via the prometheus-exporter as gauges.

Sometimes, it happens that a label combination doesn't exist anymore. And we noticed that in those cases, the prometheus exporter keeps reporting that set of labels to prometheus with the latest value it received.

Some cases are pretty hard to detect from the server's pov and have a very big impact. I.e. if a product category got renamed, all those counts will continue appearing under the old category and under the new one.

Is there a way to force the metrics of a certain gauge to be cleared for every new set of observations?

I assume this requires some concurrency controls, since the metrics can be polled at every moment. But there could be an option to overwrite all observations at all for a gauge:

client.register(:gauge, "my_app_customers").replace_observations({{state: "WA", ...} => 1, {state: "VI", ...} => 5, ...})

[sander-deryckere]

We have now added a workaround where we keep track of all our historical label combinations, and explicitly nillify them if they don't appear in the new results anymore.

But it's pretty fragile IMO (easy to go out of sync), and ideally this should be implemented on the exporter side. It's also not completely thread-safe. If a poll operation from Prometheus happens between setting the counts and clearing the old tags, it will still results in double counts for a short moment.