Add support for Node Health Checks

[sjpb]

See ansible/roles/nhc/README.md. Checks are templated out on each compute node during site.yml and then restored after a slurm-controlled rebuild, so that checks after rebuild ensure the node is in the same state as it was before.

Note this also moves the configuration of nfs-exported data for compute-init to the end of the site.yml playbook, to make ordering of roles requiring compute-init support easier.

[sjpb]

Image build: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/14621365864

[sjpb]

image build: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/14621525855

[sjpb]

Fat image build: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/15727955573

[sjpb]

Manual checks that this works:

• Deploy cluster at this branch
• Remove NFS mount and check node goes DOWN. Remount.
• Remove ansible-init sentinel, run ansible-init again and check that works

[sjpb]

Failed manual checks when deploying:

TASK [nhc : Fetch generated NHC config to control node storage] ********************************************************
Wednesday 18 June 2025  09:37:50 +0000 (0:00:00.134)       0:12:55.714 ******** 
fatal: [RL9-compute-1]: FAILED! => {}

MSG:

Unable to create local directories(/exports/cluster/hostconfigRL9-compute-1): [Errno 13] Permission denied: b'/exports'
fatal: [RL9-compute-0]: FAILED! => {}

MSG:

Unable to create local directories(/exports/cluster/hostconfigRL9-compute-0): [Errno 13] Permission denied: b'/exports'

[sjpb]

Image build: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/15731700893

[sjpb]

Ok so the problem is:

• The image we're using on this branch now has the npc-ohpc package in it.
• In CI, there is a point where the control node and login node are on that image (b/c TF has upgraded them to it) but the compute nodes aren't (b/c they have slurm-controlled rebuild). We run site.yml at that point.
• The slurm.yml playbook skips the openhpc installs for speed. But this means the compute nodes don't get that package. Then the playbook tries to configure nhc, which fails.

[sjpb]

Commit above changed approach b/c there is a problem; on first slurm-controlled upgrade to this branch, site.yml is run while the compute nodes haven't been upgraded and hence don't have the nhc binaries to perform autoconfiguration.

[sjpb]

Fat image build: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/15737604861

[sjpb]

Fatimage build: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/15762514081

[sjpb]

Fat image build: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/15763342110

[sjpb]

Image build: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/15776990066

[sjpb]

Above is failing on the /etc/nhc dir not existing. This is b/c that is created on install of the nhc-ohpc package, which is only in the new image, which the nodes don't have when running site.yml after upgrading login/control nodes.

[sjpb]

Image build: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/15779325754

[wtripp180901]

Think most of these refactors have been reviewed previously and NHC installs all look sensible so I'm happy to approve