Remote rendering: Collect and expose Prometheus metrics (#71)

Support enable Prometheus metrics endpoint, exposing Node.js 
and render request latency (histogram) metrics.
Adds HA-setup and load test to ease load testing current solution, 
but also prepares for comparing implementation of #66.

Closes #69

Author: marefr

.dockerignore

@@ -1,11 +1,15 @@
 .circleci
+.github
 artifacts
 build
+devenv
 dist
-docker
 docs
 node_modules
 scripts
+.prettierignore
+.prettierrc.json
 Dockerfile
 Makefile
 plugin_start*
+tslint.json

devenv/docker/ha/README.md

@@ -0,0 +1,84 @@
+# Grafana Image Rendering High Availability (HA) test setup
+
+A set of docker compose services which together creates a Grafana Image Rendering HA test setup with capability of easily
+scaling up/down number of Grafana image renderer instances.
+
+Included services
+
+* Grafana
+* Grafana image renderer service
+* Mysql - Grafana configuration database
+* Prometheus - Monitoring of Grafana and used as data source of provisioned alert rules
+* cAdvisor - Docker monitoring
+* Nginx - Reverse proxy for Grafana, Grafana image renderer and Prometheus. Enables browsing Grafana/renderer/Prometheus UI using a hostname
+
+## Prerequisites
+
+### Build grafana image renderer docker container
+
+Build a Grafana image renderer docker container and tag it as grafana/grafana-image-renderer:dev.
+
+```bash
+$ cd <grafana-image-renderer repo>
+$ docker build -t grafana/grafana-image-renderer:dev .
+```
+
+### Virtual host names
+
+#### Alternative 1 - Use dnsmasq
+
+```bash
+$ sudo apt-get install dnsmasq
+$ echo 'address=/loc/127.0.0.1' | sudo tee /etc/dnsmasq.d/dnsmasq-loc.conf > /dev/null
+$ sudo /etc/init.d/dnsmasq restart
+$ ping whatever.loc
+PING whatever.loc (127.0.0.1) 56(84) bytes of data.
+64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.076 ms
+--- whatever.loc ping statistics ---
+1 packet transmitted, 1 received, 0% packet loss, time 1998ms
+```
+
+#### Alternative 2 - Manually update /etc/hosts
+
+Update your `/etc/hosts` to be able to access Grafana and/or Prometheus UI using a hostname.
+
+```bash
+$ cat /etc/hosts
+127.0.0.1       grafana.loc
+127.0.0.1       renderer.loc
+127.0.0.1       prometheus.loc
+```
+
+## Start services
+
+```bash
+$ cd <grafana-image-renderer repo>/devenv/docker/ha
+$ docker-compose up -d
+```
+
+Browse
+* http://grafana.loc/
+* http://renderer.loc/
+* http://prometheus.loc/
+
+Check for any errors
+
+```bash
+$ docker-compose logs | grep error
+```
+
+You can also provide environment variables for Grafana and Grafana image renderer docker image versions
+
+```bash
+$ GRAFANA_VERSION=6.5.0 RENDERER_VERSION=1.0.7 docker-compose up -d
+```
+
+### Scale Grafana instances up/down
+
+Scale number of Grafana image renderer instances to `<instances>`
+
+```bash
+$ docker-compose up --scale renderer=<instances> -d
+# for example 3 instances
+$ docker-compose up --scale renderer=3 -d
+```

devenv/docker/ha/docker-compose.yaml

@@ -0,0 +1,95 @@
+version: "2.1"
+
+services:
+  nginx-proxy:
+    image: jwilder/nginx-proxy
+    ports:
+      - "80:80"
+    volumes:
+      - /var/run/docker.sock:/tmp/docker.sock:ro
+    depends_on:
+      db:
+        condition: service_healthy
+
+  db:
+    image: mysql:5.6
+    environment:
+      MYSQL_ROOT_PASSWORD: rootpass
+      MYSQL_DATABASE: grafana
+      MYSQL_USER: grafana
+      MYSQL_PASSWORD: password
+    command: [mysqld, --character-set-server=utf8mb4, --collation-server=utf8mb4_unicode_ci, --innodb_monitor_enable=all, --max-connections=1001]
+    ports:
+      - 3306
+    healthcheck:
+      test: ["CMD", "mysqladmin" ,"ping", "-h", "localhost"]
+      timeout: 10s
+      retries: 10
+
+  mysqld-exporter:
+    image: prom/mysqld-exporter
+    environment:
+      - DATA_SOURCE_NAME=root:rootpass@(db:3306)/
+    ports:
+      - 9104
+    depends_on:
+      db:
+        condition: service_healthy
+
+  grafana:
+    image: grafana/grafana:${GRAFANA_VERSION}-ubuntu
+    volumes:
+      - ./grafana/provisioning/:/etc/grafana/provisioning/
+    environment:
+      - VIRTUAL_HOST=grafana.loc
+      - GF_SERVER_ROOT_URL=http://grafana.loc
+      - GF_DATABASE_NAME=grafana
+      - GF_DATABASE_USER=grafana
+      - GF_DATABASE_PASSWORD=password
+      - GF_DATABASE_TYPE=mysql
+      - GF_DATABASE_HOST=db:3306
+      - GF_DATABASE_MAX_OPEN_CONN=300
+      - GF_SERVER_ROUTER_LOGGING=true
+      - GF_LOG_CONSOLE_FORMAT=json
+      - GF_LOG_FILTERS=rendering:debug
+      - GF_RENDERING_SERVER_URL=http://renderer:8081/render
+      - GF_RENDERING_CALLBACK_URL=http://grafana:3000
+    ports:
+      - 3000
+    links:
+      - nginx-proxy
+    depends_on:
+      db:
+        condition: service_healthy
+
+  renderer:
+    image: grafana/grafana-image-renderer:${RENDERER_VERSION}
+    environment:
+      - VIRTUAL_HOST=renderer.loc
+      - ENABLE_METRICS=true
+    ports:
+      - 8081
+    mem_limit: 500mb
+    memswap_limit: 1gb
+
+  prometheus:
+    image: prom/prometheus:v2.14.0
+    volumes:
+      - ./prometheus/:/etc/prometheus/
+    environment:
+      - VIRTUAL_HOST=prometheus.loc
+    ports:
+      - 9090
+
+  cadvisor:
+    image: google/cadvisor:latest
+    environment:
+      - VIRTUAL_HOST=cadvisor.loc
+    ports:
+    - 8080
+    volumes:
+    - /:/rootfs:ro
+    - /var/run:/var/run:ro
+    - /sys:/sys:ro
+    - /var/lib/docker/:/var/lib/docker:ro
+    - /dev/disk:/dev/disk/:ro

devenv/docker/ha/grafana/provisioning/dashboards/dashboards.yaml

@@ -0,0 +1,7 @@
+apiVersion: 1
+
+providers:
+ - name: 'General'
+   type: file
+   options:
+     path: /etc/grafana/provisioning/dashboards/general