Etcd Mixin Dashboard - Multi-Cluster

[jameshearttech]

We are using Helm chart kube-prometheus-stack to deploy to Kubernetes. This dashboard works fine for a single cluster, but I suspect it does not work for multi-cluster. Can we make this work the same as the Kubernetes mixin with regard to multi-cluster?

[jmhbnz]

Hey @jameshearttech - Thanks for your question, for context we recently made a tweak to the cluster variable for the dashboard in #16778.

To clarify, are you talking about using the dashboard for multiple etcd clusters running on top of one kubernetes cluster, or multiple kubernetes clusters?

If you can include a bit more detail I can try to set up an environment and take a look.

[jameshearttech]

Our use case is multiple Kubernetes clusters with etcd stacked on the control plane. Initially, we will have 3 clusters: mgmt, test, and prod. We actually haven't start shipping metrics from test and prod to Thanos, but we're working up to it, which is why I'm trying to figure out how the dashboards will work.

All the dashboards we are using from Grafana community as well as the Kubernetes mixin work with or have been modified to work with the external label cluster. The etcd mixin dashboard cluster variable query returns the name of the job (e.g., kube-etcd) rather than the external label (e.g., mgmt). Also, the datasource needs to be default (i.e., Thanos) rather than Prometheus for multi-cluster.

I modified the current dashboard, which is deployed in our mgmt cluster to work with the default datasource and external label cluster. This needs more work, but it is heading in the right direction.

etcd-mixin-multicluster.json

[jmhbnz]

Thanks for the context. I can potentially do some testing in the weekend coming if no-one beats me to it. Will report back once I do :)