20211005 Prometheus revamp - master branch - PR 1 of 3

A [Discord thread](https://discord.com/channels/638610460567928832/638610461109256194/891417109920362518)
revealed several problems with Prometheus including:

* `build.sh` not being run
* `config.yml` not being in the right place
* `config.yml` scraping not being adjusted if CAdvisor and Node
Exporter were selected in the menu.

This Pull Request proposes a revamp of Prometheus:

1. A single service definition which includes Prometheus, CAdvisor and
Node Exporter. Note that this changes the `prometheus` fragment from
the previous pairing of:

	```
	volumes:
	  - ./volumes/prometheus/data:/data
	command:
	  - '--storage.tsdb.path=/data'
	```

	to:

	```
	volumes:
	  - ./volumes/prometheus/data:/prometheus
	```

	which is the default for the `--storage` argument.

2. A matching `config.yml` set to scrape information from all three
cooperating containers. The file is referenced as:

	```
	command:
	  - '--config.file=/prometheus/config/config.yml'
	```

	so it automatically becomes part of the persistent storage area.

3. Self-healing functionality similar to the Mosquitto example
implemented via a Dockerfile.

Removes `build.sh`, `service_cadvisor-arm.yml` and `service_node-exporter.yml`
from the template as no longer necessary.

Adds container documentation for Prometheus (previously non-existent).

Author: Paraphraser

.templates/prometheus/Dockerfile

@@ -0,0 +1,25 @@
+# Download base image
+FROM prom/prometheus:latest
+
+USER root
+
+# where IOTstack template files are stored
+ENV IOTSTACK_DEFAULTS_DIR="iotstack_defaults"
+ENV IOTSTACK_CONFIG_DIR="/prometheus/config/"
+
+# copy template files to image
+COPY --chown=nobody:nobody ${IOTSTACK_DEFAULTS_DIR} /${IOTSTACK_DEFAULTS_DIR}
+
+# add default config from image to template
+RUN cp /etc/prometheus/prometheus.yml /${IOTSTACK_DEFAULTS_DIR}
+
+# replace the docker entry-point script
+ENV IOTSTACK_ENTRY_POINT="docker-entrypoint.sh"
+COPY ${IOTSTACK_ENTRY_POINT} /${IOTSTACK_ENTRY_POINT}
+RUN chmod 755 /${IOTSTACK_ENTRY_POINT}
+ENTRYPOINT ["/docker-entrypoint.sh"]
+ENV IOTSTACK_ENTRY_POINT=
+
+USER nobody
+
+# EOF

.templates/prometheus/build.sh

@@ -0,0 +1,33 @@
+#!/bin/ash
+set -e
+
+# set defaults for config structure ownership
+UID="${IOTSTACK_UID:-nobody}"
+GID="${IOTSTACK_GID:-nobody}"
+
+# were we launched as root?
+if [ "$(id -u)" = "0" -a -d /"$IOTSTACK_DEFAULTS_DIR" ]; then
+
+   # yes! ensure that the IOTSTACK_CONFIG_DIR exists
+   mkdir -p "$IOTSTACK_CONFIG_DIR"
+   
+   # populate runtime directory from the defaults
+   for P in /"$IOTSTACK_DEFAULTS_DIR"/* ; do
+
+      C=$(basename "$P")
+
+      if [ ! -e "$IOTSTACK_CONFIG_DIR/$C" ] ; then
+
+         cp -a "$P" "$IOTSTACK_CONFIG_DIR/$C"
+
+      fi
+
+   done
+
+   # enforce correct ownership
+   chown -R "$UID":"$GID" "$IOTSTACK_CONFIG_DIR"
+   
+fi
+
+# launch prometheus with supplied arguments
+exec /bin/prometheus "$@"

.templates/prometheus/config.yml

@@ -0,0 +1,12 @@
+global:
+  scrape_interval: 10s
+  evaluation_interval: 10s
+
+scrape_configs:
+  - job_name: "iotstack"
+    static_configs:
+      - targets:
+        - localhost:9090
+        - cadvisor:8080
+        - nodeexporter:9100
+

.templates/prometheus/docker-entrypoint.sh

@@ -1,15 +1,47 @@
 prometheus:
   container_name: prometheus
-  image: prom/prometheus:latest
+  build: ./.templates/prometheus/.
   restart: unless-stopped
   user: "0"
   ports:
     - "9090:9090"
+  environment:
+    - IOTSTACK_UID=1000
+    - IOTSTACK_GID=1000
   volumes:
-    - ./services/prometheus/config.yml:/etc/prometheus/config.yml
-    - ./volumes/prometheus/data:/data
+    - ./volumes/prometheus/data:/prometheus
   command:
-    - '--config.file=/etc/prometheus/config.yml'
-    - '--storage.tsdb.path=/data'
+    - '--config.file=/prometheus/config/config.yml'
+    #    defaults are:
+    #    - --config.file=/etc/prometheus/prometheus.yml
+    #    - --storage.tsdb.path=/prometheus
+    #    - --web.console.libraries=/usr/share/prometheus/console_libraries
+    #    - --web.console.templates=/usr/share/prometheus/consoles
+  depends_on:
+    - cadvisor
+    - nodeexporter
+  networks:
+    - iotstack_nw
+
+cadvisor:
+  container_name: cadvisor
+  image: zcube/cadvisor:latest
+  restart: unless-stopped
+  ports:
+    - "8082:8080"
+  volumes:
+    - /:/rootfs:ro
+    - /var/run:/var/run:rw
+    - /sys:/sys:ro
+    - /var/lib/docker/:/var/lib/docker:ro
+  networks:
+    - iotstack_nw
+
+nodeexporter:
+  container_name: nodeexporter
+  image: prom/node-exporter:latest
+  restart: unless-stopped
+  expose:
+    - "9100"
   networks:
     - iotstack_nw