20211005 Prometheus revamp - old-menu branch - PR 2 of 3

A [Discord thread](https://discord.com/channels/638610460567928832/638610461109256194/891417109920362518)
revealed several problems with Prometheus including:

* `build.sh` not being run
* `config.yml` not being in the right place
* `config.yml` scraping not being adjusted if CAdvisor and Node
Exporter were selected in the menu.

This Pull Request proposes a revamp of Prometheus:

1. A single service definition which includes Prometheus, CAdvisor and
Node Exporter. Note that this changes the `prometheus` fragment from
the previous pairing of:

	```
	volumes:
	  - ./volumes/prometheus/data:/data
	command:
	  - '--storage.tsdb.path=/data'
	```

	to:

	```
	volumes:
	  - ./volumes/prometheus/data:/prometheus
	```

	which is the default for the `--storage` argument.

2. A matching `config.yml` set to scrape information from all three
cooperating containers. The file is referenced as:

	```
	command:
	  - '--config.file=/prometheus/config/config.yml'
	```

	so it automatically becomes part of the persistent storage area.

3. Self-healing functionality similar to the Mosquitto example
implemented via a Dockerfile.

Removes `build.sh`, `service_cadvisor-arm.yml`, `service_node-exporter.yml`
and the empty `prometheus.env` from the template as no longer necessary.

Author: Paraphraser

.templates/prometheus/Dockerfile

@@ -0,0 +1,25 @@
+# Download base image
+FROM prom/prometheus:latest
+
+USER root
+
+# where IOTstack template files are stored
+ENV IOTSTACK_DEFAULTS_DIR="iotstack_defaults"
+ENV IOTSTACK_CONFIG_DIR="/prometheus/config/"
+
+# copy template files to image
+COPY --chown=nobody:nobody ${IOTSTACK_DEFAULTS_DIR} /${IOTSTACK_DEFAULTS_DIR}
+
+# add default config from image to template
+RUN cp /etc/prometheus/prometheus.yml /${IOTSTACK_DEFAULTS_DIR}
+
+# replace the docker entry-point script
+ENV IOTSTACK_ENTRY_POINT="docker-entrypoint.sh"
+COPY ${IOTSTACK_ENTRY_POINT} /${IOTSTACK_ENTRY_POINT}
+RUN chmod 755 /${IOTSTACK_ENTRY_POINT}
+ENTRYPOINT ["/docker-entrypoint.sh"]
+ENV IOTSTACK_ENTRY_POINT=
+
+USER nobody
+
+# EOF

.templates/prometheus/build.sh

@@ -0,0 +1,33 @@
+#!/bin/ash
+set -e
+
+# set defaults for config structure ownership
+UID="${IOTSTACK_UID:-nobody}"
+GID="${IOTSTACK_GID:-nobody}"
+
+# were we launched as root?
+if [ "$(id -u)" = "0" -a -d /"$IOTSTACK_DEFAULTS_DIR" ]; then
+
+   # yes! ensure that the IOTSTACK_CONFIG_DIR exists
+   mkdir -p "$IOTSTACK_CONFIG_DIR"
+   
+   # populate runtime directory from the defaults
+   for P in /"$IOTSTACK_DEFAULTS_DIR"/* ; do
+
+      C=$(basename "$P")
+
+      if [ ! -e "$IOTSTACK_CONFIG_DIR/$C" ] ; then
+
+         cp -a "$P" "$IOTSTACK_CONFIG_DIR/$C"
+
+      fi
+
+   done
+
+   # enforce correct ownership
+   chown -R "$UID":"$GID" "$IOTSTACK_CONFIG_DIR"
+   
+fi
+
+# launch prometheus with supplied arguments
+exec /bin/prometheus "$@"

.templates/prometheus/config.yml

@@ -0,0 +1,12 @@
+global:
+  scrape_interval: 10s
+  evaluation_interval: 10s
+
+scrape_configs:
+  - job_name: "iotstack"
+    static_configs:
+      - targets:
+        - localhost:9090
+        - cadvisor:8080
+        - nodeexporter:9100
+

.templates/prometheus/docker-entrypoint.sh

@@ -1,17 +1,41 @@
   prometheus:
     container_name: prometheus
-    image: prom/prometheus:latest
+    build: ./.templates/prometheus/.
     restart: unless-stopped
     user: "0"
     ports:
-      - 9090:9090
-    env_file:
-      - ./services/prometheus/prometheus.env
-    depends_on:
+      - "9090:9090"
+    environment:
+      - IOTSTACK_UID=1000
+      - IOTSTACK_GID=1000
     volumes:
-      - ./services/prometheus/config.yml:/etc/prometheus/config.yml
-      - ./volumes/prometheus/data:/data
+      - ./volumes/prometheus/data:/prometheus
     command:
-      - '--config.file=/etc/prometheus/config.yml'
-      - '--storage.tsdb.path=/data'
+      - '--config.file=/prometheus/config/config.yml'
+      #    defaults are:
+      #    - --config.file=/etc/prometheus/prometheus.yml
+      #    - --storage.tsdb.path=/prometheus
+      #    - --web.console.libraries=/usr/share/prometheus/console_libraries
+      #    - --web.console.templates=/usr/share/prometheus/consoles
+    depends_on:
+      - cadvisor
+      - nodeexporter
 
+  cadvisor:
+    container_name: cadvisor
+    image: zcube/cadvisor:latest
+    restart: unless-stopped
+    ports:
+      - "8082:8080"
+    volumes:
+      - /:/rootfs:ro
+      - /var/run:/var/run:rw
+      - /sys:/sys:ro
+      - /var/lib/docker/:/var/lib/docker:ro
+
+  nodeexporter:
+    container_name: nodeexporter
+    image: prom/node-exporter:latest
+    restart: unless-stopped
+    expose:
+      - "9100"