[SLURM] Support health check scripts for worker nodes

[SifeiLi]

Problem Description

At a high level, we will provide scripts that run DCGM l4 test as a slurm job to check slurm worker node, along with customizable test template and README. This only applies to GPU instances

Details

The script/instruction solution consists of the following components:

1. DCGM L4 DeepHealthCheck Script (run as slurm job)
   Experience: Automatic GPU DCGM health check runs as slurm job, bad nodes get flagged/replaced without manual intervention

2. Customizable Template for Test Script (run as slurm job)
   Users can modify test template to add their own tests, this script provides flexibility to define "healthy" based on their workload requirements

3. Documentation: Instructions and Best Practices
   User can follow guides to run health checks or configure it as prolog - automatic health check runs before job starts