change NHC to use templating instead of autoconfiguration

Author: sjpb

ansible/roles/nhc/README.md

@@ -21,83 +21,46 @@ To enable node health checks, ensure the `nhc` group contains the `compute` grou
 compute
 ```
 
-This will automatically:
+When the `anisble/site.yml` playbook is run this will automatically:
 1. Add NHC-related configuration to the `slurm.conf` Slurm configuration file.
-The default configuration is defined in `openhpc_config_nhc`
-(see [environments/common/inventory/group_vars/all/openhpc.yml](../../../environments/common/inventory/group_vars/all/openhpc.yml)).
-It will run healthchecks on all `IDLE` nodes which are not `DRAINED` or `NOT_RESPONDING`
-every 300 seconds. See [slurm.conf parameters](https://slurm.schedmd.com/slurm.conf.html)
-`HealthCheckInterval`, `HealthCheckNodeState`, `HealthCheckProgram`. These may
-be overriden if required by redefining `openhpc_config_nhc` in e.g.
-`environments/site/inventory/group_vars/nhc/yml`.
-
-2. Define a default configuration for health checks for each compute node
-individually using [nhc-genconf](https://github.com/mej/nhc?tab=readme-ov-file#config-file-auto-generation)
-The generated checks include:
+   The default configuration is defined in `openhpc_config_nhc`
+   (see [environments/common/inventory/group_vars/all/openhpc.yml](../../../environments/common/inventory/group_vars/all/openhpc.yml)).
+   It will run healthchecks on all `IDLE` nodes which are not `DRAINED` or
+   `NOT_RESPONDING` every 300 seconds. See [slurm.conf parameters](https://slurm.schedmd.com/slurm.conf.html)
+   `HealthCheckInterval`, `HealthCheckNodeState`, `HealthCheckProgram`. These
+   may be overriden if required by redefining `openhpc_config_nhc` in e.g.
+   `environments/site/inventory/group_vars/nhc/yml`.
+
+2. Template out node health check rules using Ansible facts for each compute
+   node. Currently these check:
     - Filesystem mounts
-    - Filesystem space
-    - CPU info
-    - Memory and swap
-    - Network interfaces
-    - Various processes
+    - Ethernet interfaces
 
     See `/etc/nhc/nhc.conf` on a compute node for the full configuration.
 
-The automatically generated checks may be modified or disabled using the
-`nhc_replacements` role variable described below.
-
 If a node healthcheck run fails, Slurm will mark the node `DOWN`. With the
 default [alerting configuration](../../../docs/alerting.md) this will trigger
 an alert.
 
-## Updating Health Checks
-
-The above approach assumes that when the `site.yml` playbook is run all nodes
-are functioning correctly. Therefore if changes are made to aspects covered by
-the healthchecks (see above) without re-running this playbook, use the following
-to update the autogenerated health checks:
-
-  ```shell
-  ansible-playbook ansible/extras.yml --tags nhc
-  ```
-
 ## Role Variables
 
-- `nhc_config_changes`: Optional, default empty list. A list of mappings
-  defining replacements in the autogenerated health checks. Mappings may have
-  any keys from [ansible.builtin.lineinfile](https://docs.ansible.com/ansible/latest/collections/ansible/builtin/lineinfile_module.html).
-  Note that the NHC [configuration line format](https://github.com/mej/nhc?tab=readme-ov-file#configuration-file-syntax) is:
-
-      TARGET || CHECK
-
-  where for autogenerated checks `TARGET` is the hostname. So a regex like:
-
-      '^(\s+\S+\s+)\|\|(\s+.*)$'
-
-  captures the `TARGET` and `||` separator as `\1` and the actual check as `\2`.
-  For example the following configuration will comment-out checks for a
-  specific interface for all nodes - in reality using `state: absent` would be
-  better if the check is not required, but this shows the regex syntax:
-
-  ```yaml
-  nhc_config_changes:
-    - regexp:  '^(\s+\S+\s+\|\|\s+)(check_hw_eth eth0)$'
-      line: '#\1\2'
-      backrefs: yes
-  ```
+- `nhc_config_template`: Template to use. Default is the in-role template
+  providing rules described above.
+- `nhc_config_extra`: Possibly multiline string defining [additional rules](https://github.com/mej/nhc/blob/master/README.md) to
+  add. Jinja templating may be used. Default is empty string.
 
 ## Structure
 
 This role contains 3x task files, which run at different times:
-- `main.yml`: Runs from `site.yml` -> `slurm.yml`. Generates health check
-  configuration.
-- `export.yml`: Runs from `site.yml` -> `extras.yml` via role `compute_init`
-  tasks `export.yml`. Copies the generated health check configuration to the
-  control node NFS share for compute-init. 
-- `import.yml`: Runs on boot via `compute_init/files/compute-init.yml` and
-  copies the node's generated health check configuration from the control node
-  NFS share to local disk.
+- `main.yml`: Runs from `site.yml` -> `slurm.yml`. Templates health check
+  configuration to nodes.
+- `export.yml`: Runs from `site.yml` -> `final.yml` via role `compute_init`
+  tasks `export.yml`. Templates health check configuration to the cluster NFS
+  share for compute-init. 
+- `boot.yml`: Runs on boot via `compute_init/files/compute-init.yml`. Copies
+  the node's generated health check configuration from the cluster share to
+  local disk.
 
 Note that the `stackhpc.openhpc` role:
 - Installs the required package
-- Configures slurm
+- Configures slurm.conf parameterss

ansible/roles/nhc/defaults/main.yml

@@ -1 +1,2 @@
-nhc_config_changes: []
+nhc_config_template: nhc.conf.j2
+nhc_config_extra: ''

ansible/roles/nhc/tasks/import.yml renamed to ansible/roles/nhc/tasks/boot.yml

@@ -1,10 +1,6 @@
-- name: Slurp generated NHC configuration
-  ansible.builtin.slurp:
-    src: /etc/nhc/nhc.conf
-  register: _nhc_generated_conf
-
-- name: Write generated NHC configuration to control node exports
-  ansible.builtin.copy:
+# Used for compute-init
+- name: Template out host specific NHC config
+  ansible.builtin.template:
+    src: "{{ nhc_config_template }}"
     dest: "/exports/cluster/hostconfig/{{ inventory_hostname }}/nhc.conf"
-    content: "{{ _nhc_generated_conf.content | b64decode }}"
   delegate_to: "{{ groups['control'] | first }}"

ansible/roles/nhc/tasks/export.yml

@@ -1,32 +1,7 @@
-- name: Create temporary file for autogenerated NHC configuration
-  ansible.builtin.tempfile:
-  register: nhc_tempfile
 
-- name: Run NHC autoconfiguration
-  ansible.builtin.command:
-    cmd: "nhc-genconf -c {{ nhc_tempfile.path }}"
-  register: _nhc_genconf
-  failed_when: _nhc_genconf.rc > 1 # https://github.com/mej/nhc/issues/158
-
-- name: Apply modifications to autogenerated NHC configuration
-  ansible.builtin.lineinfile: "{{ item | combine(nhc_lineinfile_defaults) }}"
-  loop: "{{ nhc_config_changes }}"
-  vars:
-    nhc_lineinfile_defaults:
-      path: "{{ nhc_tempfile.path }}"
-
-- name: Remove timestamp from autogenerated NHC configuration
-  # for idempotency
-  ansible.builtin.replace:
-    path: "{{ nhc_tempfile.path }}"
-    # note this matches a multiline string:
-    regexp: '# This file was automatically generated by nhc-genconf\n#.*'
-    replace: '# This file was automatically generated by nhc-genconf\n# (timestamp removed for idempotency)'
-
-- name: Copy modified NHC configuration to active location
-  ansible.builtin.copy:
-    remote_src: true
-    src: "{{ nhc_tempfile.path }}"
+- name: Template out NHC configuration
+  ansible.builtin.template:
+    src: "{{ nhc_config_template }}"
     dest: /etc/nhc/nhc.conf
     owner: root
     group: root

ansible/roles/nhc/tasks/main.yml

@@ -0,0 +1,15 @@
+# {{ ansible_managed }}
+
+## Filesystem checks
+{% for mount in ansible_mounts %}
+{% set mount_mode = 'rw' if 'rw' in mount.options.split(',') else 'ro' %}
+{{ ansible_fqdn }} || check_fs_mount_{{ mount_mode }} -t "{{ mount.fstype }}" -s "{{ mount.device }}" -f "{{ mount.mount }}"
+{% endfor %}
+
+## Ethernet interface checks
+{% for iface in ansible_interfaces | select('match', 'eth') %}
+{{ ansible_fqdn }} || check_hw_eth {{ iface }}
+{% endfor %}
+
+## Site-specific checks
+{{ nhc_config_extra }}