Merge branch 'v2.7.3' of https://github.com/oci-hpc/oci-hpc-clusternetwork-dev into v2.7.3

Author: arnaudfroidmont

README.md

@@ -11,8 +11,7 @@ allow service compute_management to read app-catalog-listing in tenancy
 allow group user to manage all-resources in compartment compartmentName
 ```
 
-## What is cluster resizing (resize.py) ?
-TODO
+
 
 ## What is cluster autoscaling ?
 TODO
@@ -38,8 +37,144 @@ or:
 `Allow dynamic-group instance_principal to manage all-resources in compartment compartmentName`
 
 
-# Resizing (via resize.py or OCI console)
-TODO
+# Cluster Network Resizing (via resize.py)
+
+Cluster resizing refers to ability to add or remove nodes from an existing cluster network.  It only applies to nodes with RDMA RoCEv2 (aka: cluster network) NICs, so HPC clusters created using BM.HPC2.36, BM.Optimized3.36 and BM.GPU4.8.  Apart from add/remove, the resize.py script can also be used to reconfigure the nodes. 
+
+Resizing of HPC cluster with Cluster Network consist of 2 major sub-steps:
+- Add/Remove node (IaaS provisioning) to cluster – uses OCI Python SDK 
+- Configure the nodes (uses Ansible)
+  -  Configures newly added nodes to be ready to run the jobs
+  -  Reconfigure services like Slurm to recognize new nodes on all nodes
+  -  Update rest of the nodes, when any node/s are removed (eg: Slurm config, /etc/hosts, etc.)
+ 
+## resize.py usage 
+
+The resize.py is deployed on the bastion node as part of the HPC cluster Stack deployment.  
+
+```
+playbooks/resize.py
+
+python3 playbooks/resize.py -h
+usage: resize.py [-h] [--compartment_ocid COMPARTMENT_OCID]
+                 [--cluster_name CLUSTER_NAME] [--nodes NODES [NODES ...]]
+                 [--slurm_only_update [{true,false}]]
+                 [{add,remove,list,reconfigure}] [number]
+
+Script to resize the CN
+
+positional arguments:
+  {add,remove,list,reconfigure}
+                        Mode type. add/remove node options, implicitly
+                        configures newly added nodes. Also implicitly
+                        reconfigure/restart services like Slurm to recognize
+                        new nodes. Similarly for remove option, terminates
+                        nodes and implicitly reconfigure/restart services like
+                        Slurm on rest of the cluster nodes to remove reference
+                        to deleted nodes.
+  number                Number of nodes to add or delete if a list of
+                        hostnames is not defined
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --compartment_ocid COMPARTMENT_OCID
+                        OCID of the compartment, defaults to the Compartment
+                        OCID of the localhost
+  --cluster_name CLUSTER_NAME
+                        Name of the cluster to resize. Defaults to the name
+                        included in the bastion
+  --nodes NODES [NODES ...]
+                        Number of nodes to add or delete if a list of
+                        hostnames is not defined
+  --slurm_only_update [{true,false}]
+                        To update /etc/hosts, slurm config and restart slurm
+                        services.
+[opc@assuring-woodcock-bastion ~]$
+
+
+```
+
+**Add nodes** 
+
+Consist of the following sub-steps:
+- Add node (IaaS provisioning) to cluster – uses OCI Python SDK 
+- Configure the nodes (uses Ansible)
+  -  Configures newly added nodes to be ready to run the jobs
+  -  Reconfigure services like Slurm to recognize new nodes on all nodes
+
+Add one node 
+```
+python3 playbooks/resize.py add 1
+
+```
+
+Add three node 
+```
+python3 playbooks/resize.py add 3
+
+```
+
+
+**Remove nodes** 
+
+Consist of the following sub-steps:
+- Remove node/s (IaaS termination) from cluster – uses OCI Python SDK 
+- Reconfigure rest of the nodes in the cluster  (uses Ansible)
+  -  Remove reference to removed node/s on rest of the nodes (eg: update /etc/hosts, slurm configs, etc.)
+ 
+
+Remove specific node:  
+```
+python3 playbooks/resize.py remove --nodes inst-dpi8e-assuring-woodcock
+```
+or 
+
+Remove a list of nodes (space seperated):  
+```
+python3 playbooks/resize.py remove --nodes inst-dpi8e-assuring-woodcock inst-ed5yh-assuring-woodcock
+```
+or 
+Remove one node randomly:  
+```
+python3 playbooks/resize.py remove 1
+```
+or 
+Remove 3 nodes randomly:  
+```
+python3 playbooks/resize.py remove 3
+
+```
+
+**Reconfigure nodes** 
+
+This allows users to reconfigure nodes (Ansible tasks) of the cluster.  
+
+If you would like to do a **slurm config update ONLY** on all nodes of the cluster.   
+
+```
+python3  playbooks/resize.py reconfigure --slurm_only_update true
+```
+
+Full reconfiguration of all nodes of the cluster.   This will run the same steps, which are ran when a new cluster is created.   If you manually updated configs which are created/updated as part of cluster configuration, then this command will overwrite your manual changes.   
+
+```
+python3 playbooks/resize.py reconfigure
+```
+
+If you would like to fully reconfigure ONLY a specific node/nodes (space seperated).
+
+```
+ python3  playbooks/resize.py  reconfigure [--nodes NODES [NODES ...]]
+ Example:  python3  resize.py reconfigure --nodes inst-gsezk-topical-goblin inst-jvpps-topical-goblin 
+```
+
+
+
+## Resizing (via OCI console)
+**Things to consider:**  
+- If you resize from OCI console to reduce cluster network/instance pool size(scale down),  the OCI platform decides which node to terminate (oldest node first)
+- OCI console only resizes the Cluster Network/Instance Pool, but it doesn't execute the ansible tasks (HPC Cluster Stack) required to configure the newly added nodes or to update the existing nodes when a node is removed (eg: updating /etc/hosts, slurm config, etc).   
+
 
 
 # Autoscaling

configure.sh

@@ -8,6 +8,18 @@
 #
 execution=1
 
+if [ -n "$1" ]; then
+  playbook=$1
+else
+  playbook="/opt/oci-hpc/playbooks/site.yml"
+fi
+
+if [ -n "$2" ]; then
+  inventory=$2
+else
+  inventory="/etc/ansible/hosts"
+fi
+
 ssh_options="-i ~/.ssh/cluster.key -o StrictHostKeyChecking=no"
 
 if [ -f /opt/oci-hpc/playbooks/inventory ] ; then 
@@ -58,7 +70,7 @@ done
 
 if [[ $execution -eq 1 ]] ; then
   ANSIBLE_HOST_KEY_CHECKING=False ansible --private-key ~/.ssh/cluster.key all -m setup --tree /tmp/ansible > /dev/null 2>&1
-  ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook --private-key ~/.ssh/cluster.key /opt/oci-hpc/playbooks/site.yml
+  ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook --private-key ~/.ssh/cluster.key $playbook -i $inventory
 else
 
         cat <<- EOF > /tmp/motd