kube_pod_info cluster label

[bergur88]

Hi
I installed from the helm repo and my setup is missing the cluster label from kube_pod_info.
Can anybody tell me where that is supposed to come from, or what the source of it is ?
It is not a part of kube-state-metrics according to the docs

kube_pod_info{cluster="$cluster"}

[brancz]

Cluster is typically something you configure via external_labels in Prometheus. This is for when you have multiple Promettheus servers (for example one in each cluster) and you query them centrally, so you can distinguish them.

[pslobo]

Cluster is typically something you configure via external_labels in Prometheus. This is for when you have multiple Promettheus servers (for example one in each cluster) and you query them centrally, so you can distinguish them.

I've recently stumbled on this issue, and while the external label is there level=info ts=2021-08-26T14:24:50.034779213Z caller=sidecar.go:188 msg="successfully loaded prometheus external labels" external_labels="{cluster=\"XXX-dev\", prometheus=\"observability/k8s\", prometheus_replica=\"prometheus-k8s-1\"}" , I still see no mention of cluster in kube_pod_info. Any idea if this takes a while to propogate or if I may be missing something else @brancz?

TIA

[paulfantom]

If one configures cluster label via external_labels on prometheus instance A, then that label:

• is NOT available in any metric when querying prometheus instance A directly
• is available in all metrics exported via remote_write or federation in instance B (to which metrics are shipped).