Prom Error: multiple matches for labels: grouping labels must ensure unique matches

[cgrinds]

Query

topk(
  5,
    avg_over_time(
      volume_read_latency{cluster=~"(cluster1)",datacenter=~"rtp"}[3h]
    )
  * on (cluster, svm, volume) group_left (aggr)
    volume_labels{aggr=~".*",cluster=~"cluster1",datacenter=~"rtp"}
)

Prometheus sometimes returns the following error for the above query.
multiple matches for labels: grouping labels must ensure unique matches

This happens because of how group_left works. vector_a group_left(label-list) vector_b.

group_left means the left side is the many-side and the right side is the one-side. The one side must return a unique set of values for the labels in label-list.

In the example above, the one-side is volume_labels which means that vector expression must return unique values for aggr. If they don't, Prometheus will return the multiple matches for labels: grouping labels must ensure unique matches error.

---

Examples

Using the following metrics/labels as an example, let's take a look at some queries that succeed and fail.

avg_over_time(volume_read_latency{volume="dummy"}[2d])

• {aggr="test1", cluster="umeng-aff300-01-02", node="umeng-aff300-02", style="flexvol", svm="hasvm2", volume="dummy"}
  0
• {aggr="umeng_aff300_aggr2", cluster="umeng-aff300-01-02", node="umeng-aff300-03", style="flexvol", svm="hasvm2", volume="dummy"}
  0.012811504590593971

---

avg_over_time(volume_read_latency{volume="dummy"}[10d])

• {aggr="test1", cluster="umeng-aff300-01-02", node="umeng-aff300-02", style="flexvol", svm="hasvm2", volume="dummy"}
  0
• {aggr="umeng_aff300_aggr2", cluster="umeng-aff300-01-02", node="umeng-aff300-02", style="flexvol", svm="hasvm2", volume="dummy"}
  0
• {aggr="umeng_aff300_aggr2", cluster="umeng-aff300-01-02", node="umeng-aff300-03", style="flexvol", svm="hasvm2", volume="dummy"}
  0.005758743393763292

---

volume_labels{volume="dummy"}

• volume_labels{aggr="test1", antiRansomwareState="disabled", cluster="umeng-aff300-01-02", datacenter="nane", instance="localhost:13002", isEncrypted="false", isHardwareEncrypted="false", is_sis_volume="true", job="prometheus", junction_path="/dummy", node="umeng-aff300-02", root_volume="No", snaplock_type="non_snaplock", snapshot_autodelete="false", snapshot_policy="default", state="online", style="flexvol", svm="hasvm2", svm_root="false", type="rw", volume="dummy"}
  1

---

This query succeeds

avg_over_time(volume_read_latency{volume="dummy"}[2d]) * on(cluster, svm, volume) group_left(aggr) volume_labels{cluster="umeng-aff300-01-02"}

• {aggr="test1", cluster="umeng-aff300-01-02", node="umeng-aff300-02", style="flexvol", svm="hasvm2", volume="dummy"}
  0
• {aggr="test1", cluster="umeng-aff300-01-02", node="umeng-aff300-03", style="flexvol", svm="hasvm2", volume="dummy"}
  0.012827728530556734

---

This query fails

avg_over_time(volume_read_latency{volume="dummy"}[10d]) * on(cluster, svm, volume) group_left(aggr) volume_labels{cluster="umeng-aff300-01-02"}

• Error executing query: multiple matches for labels: grouping labels must ensure unique matches

This query fails because the resulting vector is ambiguous. It's ambiguous because group_left(aggr) copies the aggr value from the right side into the resulting vector, which creates a label collision in the result. In other words, when the left and right vectors are combined with group_left(aggr), the resulting vector would no longer be unique since the first two metrics are now identical:

• {aggr="test1", cluster="umeng-aff300-01-02", node="umeng-aff300-02", svm="hasvm2", volume="dummy"}
• {aggr="test1", cluster="umeng-aff300-01-02", node="umeng-aff300-02", svm="hasvm2", volume="dummy"}
• {aggr="test1", cluster="umeng-aff300-01-02", node="umeng-aff300-03", svm="hasvm2", volume="dummy"}

---

This query succeeds

avg_over_time(volume_read_latency{volume="dummy"}[10d]) * on(cluster, svm, volume) group_left volume_labels{cluster="umeng-aff300-01-02"}

• {aggr="test1", cluster="umeng-aff300-01-02", node="umeng-aff300-02", style="flexvol", svm="hasvm2", volume="dummy"}
  0
• {aggr="umeng_aff300_aggr2", cluster="umeng-aff300-01-02", node="umeng-aff300-02", style="flexvol", svm="hasvm2", volume="dummy"}
  0
• {aggr="umeng_aff300_aggr2", cluster="umeng-aff300-01-02", node="umeng-aff300-03", style="flexvol", svm="hasvm2", volume="dummy"}
  0.005758743393763292

Reference

• https://iximiuz.com/en/posts/prometheus-vector-matching/
• https://www.section.io/blog/prometheus-querying/
• https://prometheus.io/docs/prometheus/latest/querying/operators/#many-to-one-and-one-to-many-vector-matches
• https://github.com/prometheus/prometheus/blob/6ef9ed0bc33299499df9addee5b007070c5d32b8/promql/engine.go#L2306