Merge pull request #465 from ukkopahis/telegraf-auto-includes

telegraf: add automatic docker and temp monitoring

Author: Slyke

.templates/telegraf/Dockerfile

@@ -19,6 +19,8 @@ COPY ${IOTSTACK_DEFAULTS_DIR} /${IOTSTACK_DEFAULTS_DIR}
 ENV BASELINE_CONFIG=/${IOTSTACK_DEFAULTS_DIR}/telegraf-reference.conf
 ENV IOTSTACK_CONFIG=/${IOTSTACK_DEFAULTS_DIR}/telegraf.conf
 RUN cp /etc/telegraf/telegraf.conf ${BASELINE_CONFIG} && \
+    cat /${IOTSTACK_DEFAULTS_DIR}/auto_include/*.conf >> ${BASELINE_CONFIG} && \
+    rm -r /${IOTSTACK_DEFAULTS_DIR}/auto_include && \
     chmod 444 ${BASELINE_CONFIG} && \
     grep -v -e "^[ ]*#" -e "^[ ]*$" ${BASELINE_CONFIG} >${IOTSTACK_CONFIG} && \
     sed -i '/^\[\[outputs.influxdb\]\]/a\ \ urls = ["http://influxdb:8086"]' ${IOTSTACK_CONFIG}

.templates/telegraf/iotstack_defaults/auto_include/inputs.cpu_temp.conf

@@ -0,0 +1,5 @@
+[[inputs.file]]
+  files = ["/sys/class/thermal/thermal_zone0/temp"]
+  name_override = "cpu_temperature"
+  data_format = "value"
+  data_type = "integer"

.templates/telegraf/iotstack_defaults/additions/inputs.docker.conf renamed to .templates/telegraf/iotstack_defaults/auto_include/inputs.docker.conf

@@ -22,12 +22,13 @@ The purpose of the Dockerfile is to:
 │       ├── Dockerfile ❶
 │       ├── entrypoint.sh ❷
 │       ├── iotstack_defaults
-│       │   └── additions ❸
-│       └── service.yml ❹
+│       │   ├── additions ❸
+│       │   └── auto_include ❹
+│       └── service.yml ❺
 ├── services
 │   └── telegraf
-│       └── service.yml ❺
-├── docker-compose.yml ❻
+│       └── service.yml ❻
+├── docker-compose.yml
 └── volumes
     └── telegraf ❼
         ├── additions ❽
@@ -38,9 +39,10 @@ The purpose of the Dockerfile is to:
 1. The *Dockerfile* used to customise Telegraf for IOTstack.
 2. A replacement for the `telegraf` container script of the same name, extended to handle container self-repair.
 3. The *additions folder*. See [Applying optional additions](#optionalAdditions).
-4. The *template service definition*.
-5. The *working service definition* (only relevant to old-menu, copied from ❹).
-6. The *Compose* file (includes ❹).
+4. The *auto_include folder*. Additions automatically applied to
+   `telegraf.conf`. See [Automatic includes to telegraf.conf](#autoInclude).
+5. The *template service definition*.
+6. The *working service definition* (only relevant to old-menu, copied from ❹).
 7. The *persistent storage area* for the `telegraf` container.
 8. A working copy of the *additions folder* (copied from ❸). See [Applying optional additions](#optionalAdditions).
 9. The *reference configuration file*. See [Changing Telegraf's configuration](#editConfiguration).
@@ -214,21 +216,31 @@ When you make a change to `telegraf.conf`, you activate it by restarting the con
 $ cd ~/IOTstack
 $ docker-compose restart telegraf
 ```
+
+### <a name="autoInclude"> Automatic includes to telegraf.conf </a>
+
+* `inputs.docker.conf` instructs Telegraf to collect metrics from Docker. Requires kernel control
+  groups to be enabled to collect memory usage data. If not done during initial installation,
+  enable by running (reboot required):
+  ```
+  echo $(cat /boot/cmdline.txt) cgroup_memory=1 cgroup_enable=memory | sudo tee /boot/cmdline.txt
+  ```
+* `inputs.cpu_temp.conf' collects cpu temperature.
 
 ### <a name="optionalAdditions"> Applying optional additions </a>
 
 The *additions folder* (see [Significant directories and files](#significantFiles)) is a mechanism for additional *IOTstack-ready* configuration options to be provided for Telegraf.
 
-At the time of writing (October 2021), two additions are provided:
+Currently there is one addition:
 
-1. `inputs.docker.conf` provided by @tablatronix, which instructs Telegraf to collect metrics from Docker.
-2. `inputs.mqtt_consumer.conf` which formed part of the [gcgarner/IOTstack telegraf configuration](https://github.com/gcgarner/IOTstack/blob/master/.templates/telegraf/telegraf.conf) and instructs Telegraf to subscribe to a metric feed from the Mosquitto broker. This assumes, of course, that something is publishing those metrics.
+1. `inputs.mqtt_consumer.conf` which formed part of the [gcgarner/IOTstack telegraf configuration](https://github.com/gcgarner/IOTstack/blob/master/.templates/telegraf/telegraf.conf) and instructs Telegraf to subscribe to a metric feed from the Mosquitto broker. This assumes, of course, that something is publishing those metrics.
 
-Using `inputs.docker.conf` as the example, applying that addition to your Telegraf configuration file involves:
+Using `inputs.mqtt_consumer.conf` as the example, applying that addition to
+your Telegraf configuration file involves:
 
 ```
 $ cd ~/IOTstack/volumes/telegraf
-$ grep -v "^#" additions/inputs.docker.conf | sudo tee -a telegraf.conf >/dev/null
+$ grep -v "^#" additions/inputs.mqtt_consumer.conf | sudo tee -a telegraf.conf >/dev/null
 $ cd ~/IOTstack
 $ docker-compose restart telegraf
 ```

docs/Containers/Telegraf.md

@@ -178,6 +178,17 @@ $ sudo apt update
 $ sudo apt install libseccomp2 -t buster-backports
 ```
 
+### patch 3 - kernel control groups
+
+Kernel control groups need to be enabled in order to monitor container specific
+usage. This makes commands like `docker stats` fully work. Also needed for full
+monitoring of docker resource usage by the telegraf container.
+
+Enable by running (takes effect after reboot):
+```
+echo $(cat /boot/cmdline.txt) cgroup_memory=1 cgroup_enable=memory | sudo tee /boot/cmdline.txt
+```
+
 ## <a name="aboutSudo"> a word about the `sudo` command </a>
 
 Many first-time users of IOTstack get into difficulty by misusing the `sudo` command. The problem is best understood by example. In the following, you would expect `~` (tilde) to expand to `/home/pi`. It does:

docs/Getting-Started.md

@@ -275,6 +275,14 @@ function do_env_checks() {
 	fi
 }
 
+function do_kernel_checks() {
+    if ! grep -q "cgroup_memory=1 cgroup_enable=memory" /boot/cmdline.txt; then
+        echo "Kernel cgroups not enabled. Adding kernel parameters." >&2
+        echo "You will need to restart your system before the changes take effect."
+        echo $(cat /boot/cmdline.txt) cgroup_memory=1 cgroup_enable=memory | sudo tee /boot/cmdline.txt
+    fi
+}
+
 touch .new_install
 echo "Enter in the sudo password when prompted, to install dependencies"
 
@@ -306,3 +314,4 @@ if [ ! "$(user_in_group docker)" == "true" ]; then
 	sudo usermod -G "docker" -a $USER
 fi
 do_env_checks
+do_kernel_checks