Fix broken rules with Prometheus v3

[leonnicolas]

With Prometheus v3 histograms are normalized and this breaks rules that
select on the le label.
This PR updates the label selectors to use the new normalized value.

Before:

{le="1"}

Now with Prometheus v3:

{le="1.0"}

Fixes issue: #1005

Signed-off-by: leonnicolas [email protected]

[povilasv]

Does it / will it work Prometheus 2?

[leonnicolas]

Does it / will it work Prometheus 2?

No, If we want to support both, I guess we could do something like

metrics{ls~="1|1.0"}

Would be fine with me. I have no idea how much this could impact load on Prometheus

[povilasv]

I guess we need to do both then, would be good to test load. Or maybe some feature flag?

[leonnicolas]

How would a feature flag work/look like? This gave me the idea to fix this downstream in kube-prometheus with this PR.

[povilasv]

🤔

  _config+:: {
    prometheusV3: true|false',
  },

But probably best to support both options and at some point deprecate v2, once it's been like a year or so of stable v3

@skl WDYT?

[skl]

@povilasv @leonnicolas I was going to reply here but thought it would be easier to create a PR instead:

• fix(rules): handle apiserver normalized buckets #1008

TL;DR

• No performance issue observed using regex matcher on these rules
• Suggest we go the regex route to support both v2/v3 prometheus as well as the issues reported in Fix: Handle float apiserver buckets #965

Let me know what you think.