Add RabbitMQ Alerts 🚨 (#667)

* Add RabbitMQ alerts

Add Prometheus rules for RabbitMQ and cluster operator.
Add Alertmanager config for Slack notifications.
Add RabbitMQ-Alerts Grafana dashboard.

Co-authored-by: Gerhard Lazu <[email protected]>
Co-authored-by: Michal Kuratczyk <[email protected]>

* Fix typo

* Clarify that some labels are provided by the PromQL expression

* Don't use match label selectors in Prometheus spec

* Add output how to open UIs by port-forwarding

Co-authored-by: Gerhard Lazu <[email protected]>
Co-authored-by: Michal Kuratczyk <[email protected]>

Author: 3 people

docs/examples/prometheus/.ci-skip

@@ -0,0 +1,27 @@
+# Observability
+
+This directory contains scrape targets, RabbitMQ alerting rules, Alertmanager configuration, and RabbitMQ dashboards published in a format that is compatible with [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator).
+
+If you don't have Prometheus and Grafana installed, the quickest way to try out RabbitMQ observability is as follows:
+
+Make sure
+1. `kubectl` (version 1.18+) pointing to any running Kubernetes cluster
+1. `helm` (version 3+) being installed
+
+Optionally, if you want to receive Slack notifications for RabbitMQ alerts, you will need a Slack Webhook URL (see [here](https://api.slack.com/messaging/webhooks) how to create one).
+
+```bash
+# Optionally, to receive Slack notifications on RabbitMQ alerts, set the name of the Slack channel and the Slack Webhook URL:
+export SLACK_CHANNEL='#my-channel'
+export SLACK_API_URL='https://hooks.slack.com/services/paste/your/token'
+
+./quickstart.sh
+```
+
+This will install Prometheus Operator, Prometheus, kube-state-metrics, Alertmanager, Grafana and will set up RabbitMQ scrape targets, RabbitMQ alerting rules, Slack notifications, and RabbitMQ Grafana dashboards.
+Note that the [quickstart.sh](./quickstart.sh) script is not a production-ready setup. Refer to the official Prometheus and Grafana documentation on how to deploy a production-ready monitoring stack.
+
+Learn more on RabbitMQ monitoring in:
+* [RabbitMQ Prometheus documentation](https://www.rabbitmq.com/prometheus.html)
+* [Operator monitoring documentation](https://www.rabbitmq.com/kubernetes/operator/operator-monitoring.html)
+* [TGIR S01E07: How to monitor RabbitMQ?](https://youtu.be/NWISW6AwpOE)

docs/examples/prometheus/README.md

@@ -0,0 +1,10 @@
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: erlang-distribution-dashboard
+  labels:
+    grafana_dashboard: "true"
+data:
+  # https://grafana.com/grafana/dashboards/11352
+  erlang-distribution-dashboard.json.url: https://grafana.com/api/dashboards/11352/revisions/7/download

observability/README.md

@@ -0,0 +1,10 @@
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: erlang-distributions-compare-dashboard
+  labels:
+    grafana_dashboard: "true"
+data:
+  # https://grafana.com/grafana/dashboards/10988
+  erlang-distributions-compare-dashboard.json.url: https://grafana.com/api/dashboards/10988/revisions/11/download

observability/grafana/dashboards/erlang-distribution.yml

@@ -0,0 +1,10 @@
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: erlang-memory-allocators-dashboard
+  labels:
+    grafana_dashboard: "true"
+data:
+  # https://grafana.com/grafana/dashboards/11350
+  erlang-memory-allocators-dashboard.json.url: https://grafana.com/api/dashboards/11350/revisions/5/download