Auto-restart slurmctld on failure after 1 second.

Signed-off-by: Giacomo Marciani <[email protected]>

Author: gmarciani

CHANGELOG.md

@@ -13,6 +13,7 @@ This file is used to list changes made in each version of the AWS ParallelCluste
   - xdcv: `2024.0.631-1`
   - gl: `2024.0.1078-1`
   - web_viewer: `2024.0-18131-1`
+- Auto-restart slurmctld on failure.
 
 **BUG FIXES**
 - Fix an issue in the way we get region when manage volumes so that it can correctly handle local zone.

cookbooks/aws-parallelcluster-slurm/spec/unit/recipes/config_slurmctld_systemd_service_spec.rb

@@ -32,6 +32,8 @@
       it 'creates the service definition for slurmctld with the correct settings' do
         is_expected.to render_file('/etc/systemd/system/slurmctld.service')
           .with_content("After=network-online.target munge.service remote-fs.target")
+          .with_content("Restart=on-failure")
+          .with_content("RestartSec=1s")
       end
     end
   end

cookbooks/aws-parallelcluster-slurm/templates/default/slurm/head_node/slurmctld.service.erb

@@ -12,6 +12,8 @@ ExecReload=/bin/kill -HUP $MAINPID
 LimitNOFILE=562930
 LimitMEMLOCK=infinity
 LimitSTACK=infinity
+Restart=on-failure
+RestartSec=1s
 
 [Install]
 WantedBy=multi-user.target