Fix issues in slurm and torque so that multiple nodes can be added with a cold start

a. Add ASG Max number of dummy-compute nodes as part of potentially available resources to both slurm and torque
b. Make stack_name available to the nodewatcher so that the daemon that controls scaledown has access to the stack's status. Scaledown process will only start once stack status is complete

Signed-off-by: Balaji Sridharan <[email protected]>

Author: fnubalaj

templates/default/nodewatcher.cfg.erb

@@ -2,4 +2,5 @@
 region = <%= node['cfncluster']['cfn_region'] %>
 scheduler = <%= node['cfncluster']['cfn_scheduler'] %>
 proxy = <%= node['cfncluster']['cfn_proxy'] %>
-scaledown_idletime = <%= node['cfncluster']['cfn_scaledown_idletime'] %>
+scaledown_idletime = <%= node['cfncluster']['cfn_scaledown_idletime'] %>
+stack_name = <%= node['cfncluster']['stack_name'] %>

templates/default/slurm.conf.erb

@@ -102,6 +102,15 @@ JobCompType=jobcomp/none
 # when manually editing! Default node is there to allow
 # submission of jobs to an empty cluster.
 #PARTITION:compute
-NodeName=dummy-compute Procs=2048 State=UNKNOWN
+<%
+def append_dummy
+    if node['cfncluster']['cfn_max_queue_size'].to_i > 1
+        "dummy-compute[1-" + node['cfncluster']['cfn_max_queue_size'] + "]"
+    else
+        "dummy-compute"
+    end
+end
+%>
+NodeName=<%= append_dummy %> Procs=2048 State=UNKNOWN
 #NodeName= Procs=1 State=UNKNOWN
-PartitionName=compute Nodes=dummy-compute Default=YES MaxTime=INFINITE State=UP
+PartitionName=compute Nodes=<%= append_dummy %> Default=YES MaxTime=INFINITE State=UP

templates/default/torque.setup.erb

@@ -80,6 +80,7 @@ echo set server managers += $USER | qmgr
 qmgr -c 'set server scheduling = true'
 qmgr -c 'set server keep_completed = 300'
 qmgr -c 'set server mom_job_sync = true'
+qmgr -c 'set server resources_available.nodect = <%= node['cfncluster']['cfn_max_queue_size'] %>'
 
 # create default queue
 
@@ -89,6 +90,7 @@ qmgr -c 'set queue batch started = true'
 qmgr -c 'set queue batch enabled = true'
 qmgr -c 'set queue batch resources_default.walltime = 1:00:00'
 qmgr -c 'set queue batch resources_default.nodes = 1'
+qmgr -c 'set queue batch resources_available.nodect = <%= node['cfncluster']['cfn_max_queue_size'] %>'
 
 qmgr -c 'set server default_queue = batch'