add monitoring mixin for the rendering service (#255)

Signed-off-by: bergquist <[email protected]>

Author: bergquist

mixin/.gitignore

@@ -0,0 +1,3 @@
+/alerts.yaml
+/rules.yaml
+dashboards_out

mixin/Makefile

@@ -0,0 +1,13 @@
+all: fmt lint build clean
+
+fmt:
+	./scripts/format.sh
+
+lint:
+	./scripts/lint.sh
+
+build:
+	./scripts/build.sh
+
+clean:
+	rm -rf dashboards_out alerts.yaml rules.yaml

mixin/README.md

@@ -0,0 +1,20 @@
+# Grafana Render service Mixin
+
+To build, you need to have `mixtool` and `jsonnetfmt` installed. If you
+have a working Go development environment, it's easiest to run the following:
+
+```bash
+$ go get github.com/monitoring-mixins/mixtool/cmd/mixtool
+$ go get github.com/google/go-jsonnet/cmd/jsonnetfmt
+```
+
+You can then build the Prometheus rules files `alerts.yaml` and
+`rules.yaml` and a directory `dashboard_out` with the JSON dashboard files
+for Grafana render service:
+
+```bash
+$ make build
+```
+
+For more advanced uses of mixins, see
+https://github.com/monitoring-mixins/docs.

mixin/alerts/alerts.yaml

@@ -0,0 +1,13 @@
+groups:
+  - name: GrafanaImageRendererAlerts
+    rules:
+      - alert: GrafanaImageRendererAlertsRequestsFailing
+        for: 5m
+        expr: |
+          (100 * namespace_job_statuscode:grafana_image_renderer_service_http_request_duration_seconds_count:rate5m{status_code=~"5.."}
+          / 
+          namespace_job_statuscode:grafana_image_renderer_service_http_request_duration_seconds_count:rate5m) > 5
+        labels:
+          severity: 'warning'
+        annotations:
+          message: "'{{ $labels.namespace }}' / '{{ $labels.job }}' is experiencing {{ $value | humanize }}% errors"