How to import dashboards by grafana sidecar

[xixiangzouyibian]

Hi,

I referred the document: https://www.rabbitmq.com/kubernetes/operator/operator-monitoring.html and tried to import example dashboards into grafana: https://github.com/rabbitmq/cluster-operator/tree/main/observability/grafana/dashboards.

But the document was so brief both in rabbitmq and grafana side, which took me a long time to figure out how I needed to specify helm chart values during the grafana installation.

My environment was (following the rabbitmq document to "kubectl apply all of examples"):

And I tried to configure following values for grafana helm chart:

service:
  type: LoadBalancer
sidecar:
  skipTlsVerify: true
  dashboards:
    enabled: true
  datasources:
    enabled: true

But it didn't wok well that the grafana UI said no datasources found, and the dashboard was not fully imported:

Please advise how to configure the grafana helm chart values to import the example dashboards.

[Zerpet]

I've reproduced this issue, although I'm unsure if the root cause is the same.

The problem is that the sidecar tries to do basic authentication using Grafana credentials in the HTTP request to https://grafana.com. This is not correct, and it results in the following response from Grafana dot com API:

{
  "code": "InvalidCredentials",
  "message": "Invalid password",
  "requestId": "18f1a4bd-fc89-4716-ad16-67682896831a"
}

If you tail the logs from Grafana server (Grafana running in your k8s), do you see errors similar to this?

logger=cleanup t=2023-04-25T10:48:51.29230316Z level=info msg="Completed cleanup jobs" duration=23.356ms
logger=provisioning.dashboard type=file name=sidecarProvider t=2023-04-25T10:49:12.801080962Z level=error msg="failed to load dashboard from " file=/tmp/dashboards/rabbitmq-overview-dashboard.json error="Dashboard title cannot be empty"
logger=provisioning.dashboard type=file name=sidecarProvider t=2023-04-25T10:49:12.801169587Z level=error msg="failed to load dashboard from " file=/tmp/dashboards/rabbitmq-quorum-queues-raft-dashboard.json error="Dashboard title cannot be empty"
logger=provisioning.dashboard type=file name=sidecarProvider t=2023-04-25T10:49:42.800807685Z level=error msg="failed to load dashboard from " file=/tmp/dashboards/rabbitmq-quorum-queues-raft-dashboard.json error="Dashboard title cannot be empty"
logger=provisioning.dashboard type=file name=sidecarProvider t=2023-04-25T10:49:42.800884102Z level=error msg="failed to load dashboard from " file=/tmp/dashboards/rabbitmq-overview-dashboard.json error="Dashboard title cannot be empty"

[Zerpet]

Regarding the data sources, I can import data sources without issues. I created the following Secret:

---
apiVersion: v1
kind: Secret
metadata:
  name: prometheus-ds
  namespace: grafana
  labels:
    grafana_datasource: "true"
stringData:
  prometheus.yaml: |-
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://prometheus-server.prometheus.svc
      access: server
      isDefault: true

Note that my Prometheus is deployed in prometheus namespace using Prometheus Helm Chart. The only relevant changes I made to the data source helm values are:

sidecar:
  datasources:
    enabled: true
    label: grafana_datasource
    labelValue: "true"
    skipReload: false

[xixiangzouyibian]

I've reproduced this issue, although I'm unsure if the root cause is the same.

The problem is that the sidecar tries to do basic authentication using Grafana credentials in the HTTP request to https://grafana.com. This is not correct, and it results in the following response from Grafana dot com API:

{
  "code": "InvalidCredentials",
  "message": "Invalid password",
  "requestId": "18f1a4bd-fc89-4716-ad16-67682896831a"
}

If you tail the logs from Grafana server (Grafana running in your k8s), do you see errors similar to this?

logger=cleanup t=2023-04-25T10:48:51.29230316Z level=info msg="Completed cleanup jobs" duration=23.356ms
logger=provisioning.dashboard type=file name=sidecarProvider t=2023-04-25T10:49:12.801080962Z level=error msg="failed to load dashboard from " file=/tmp/dashboards/rabbitmq-overview-dashboard.json error="Dashboard title cannot be empty"
logger=provisioning.dashboard type=file name=sidecarProvider t=2023-04-25T10:49:12.801169587Z level=error msg="failed to load dashboard from " file=/tmp/dashboards/rabbitmq-quorum-queues-raft-dashboard.json error="Dashboard title cannot be empty"
logger=provisioning.dashboard type=file name=sidecarProvider t=2023-04-25T10:49:42.800807685Z level=error msg="failed to load dashboard from " file=/tmp/dashboards/rabbitmq-quorum-queues-raft-dashboard.json error="Dashboard title cannot be empty"
logger=provisioning.dashboard type=file name=sidecarProvider t=2023-04-25T10:49:42.800884102Z level=error msg="failed to load dashboard from " file=/tmp/dashboards/rabbitmq-overview-dashboard.json error="Dashboard title cannot be empty"

Hi Zerpet, Thanks for your response. Actually I checked my grafana-server, and it didn't report any error, just some warnings to suggest enable TLS.

[xixiangzouyibian]

Regarding the data sources, I can import data sources without issues. I created the following Secret:

---
apiVersion: v1
kind: Secret
metadata:
  name: prometheus-ds
  namespace: grafana
  labels:
    grafana_datasource: "true"
stringData:
  prometheus.yaml: |-
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://prometheus-server.prometheus.svc
      access: server
      isDefault: true

Note that my Prometheus is deployed in prometheus namespace using Prometheus Helm Chart. The only relevant changes I made to the data source helm values are:

sidecar:
  datasources:
    enabled: true
    label: grafana_datasource
    labelValue: "true"
    skipReload: false

My way is following the rabbitmq document: https://www.rabbitmq.com/kubernetes/operator/operator-monitoring.html#prom-operator, install prometheus operator first, then install prometheus by "prometheus" CRD, which referred: https://prometheus-operator.dev/docs/user-guides/getting-started/#installing-the-operator.

Although rabbitmq document provides the podmonitor and servicemonitor spec to apply, it doesn't work well due to we must configure the label, so that we can specify the label name in prometheus spec for installation. (This also cost me a lot of time to figure out why prometheus cannot find the podmonitor and servicemonitor resources). After I verified prometheus found the resources successfully, the process was pending on configuring grafana sidecar due to lack of neccessary configuration example reference.
Actually I'm very appreciate that rabbitmq can provide the wonderful dashboard for user to monitor, but from my practice, some of the itermediate configuration is essential, but there is no documentation of the steps, which can only a pity.

I have now deployed prometheus and grafana in one click using kube-prometheus-stack to replace the previous installation. but from the provided dashboard, not quite as cool as rabbitmq-overview indeed. (manually import rabbitmq-overview still does not work well, no data or N/A shown in the dashboard)

I'll look into it later, or try the secret you provided to see if it works.

[Zerpet]

Although rabbitmq document provides the podmonitor and servicemonitor spec to apply, it doesn't work well due to we must configure the label, so that we can specify the label name in prometheus spec for installation.

That is correct. We have the following comment in the YAML:

https://github.com/rabbitmq/cluster-operator/blob/ad59bc756637792e461a0e85524fe4f670f103ef/observability/prometheus/monitors/rabbitmq-servicemonitor.yml#L1C1-L7

I guess we could add the same warning to the docs.

After I verified prometheus found the resources successfully, the process was pending on configuring grafana sidecar due to lack of neccessary configuration example reference.

I agree the sidecar configuration is not well documented on the Grafana side. It also took me a while to figure out how to use it properly. However, I don't think we should overlap our docs with Grafana specific configuration. I noticed an issue with our dashboards that I'm planning to fix in #1334.

(manually import rabbitmq-overview still does not work well, no data or N/A shown in the dashboard)

Do you have errors reported in the dashboards?
For example, if metric series are duplicated (due to monitoring RabbitMQ twice), the dashboard will break and show no-data and N/A, and it will also show a red message on each panel.

[xixiangzouyibian]

Yes, overlap docs are not neccessary from RabbitMQ side, but I suggest enrich the examples of necessary configurations(like: secret, data-source configuration and so on) on this path: https://github.com/rabbitmq/cluster-operator/tree/main/observability/grafana/dashboards, not only dashboards floder. Because only dashboards can not make grafana working well.

[xixiangzouyibian]

Do you have errors reported in the dashboards?
For example, if metric series are duplicated (due to monitoring RabbitMQ twice), the dashboard will break and show no-data and N/A, and it will also show a red message on each panel.

The error is no data sources found, which looks grafana can not find prometheus data sources successfully.
Also empty found from data source drop-down menu in grafana ui.

[Zerpet]

Yes, overlap docs are not neccessary from RabbitMQ side, but I suggest enrich the examples of necessary configurations(like: secret, data-source configuration and so on) [...]

I don't think we should document data-sources in our repo, because it depends entirely on how a user deploys Prometheus. For example, Prometheus can be deployed in any namespace, it can be queried directly, via Thanos, via Grafana Mimir, and other methods. Each deployment type would require a specific data source example, and probably clarifications and concrete instructions. At that point, we would be documenting how to configure Grafana data-sources, and the best place for those instructions is Grafana docs, not RabbitMQ docs 🙂

The error is no data sources found, which looks grafana can not find prometheus data sources successfully.
Also empty found from data source drop-down menu in grafana ui.

It looks like Prometheus data source is not configured correctly. If you deployed Prometheus using Prometheus Operator, it probably deployed a Service to access Prometheus. In that case, your data source URL should be: http://<service-name>.<namespace>.svc.

Pay also attention to the labels in your Secret, in our example: grafana_datasource: "true". This label must match the Helm Chart value sidecar.datasources.label and sidecar.datasources.labelValue.

P.s: I'm going to convert this issue to a discussion, since this is a troubleshooting effort.