bug: vmset is not reliable with hostnames (#3)

* bug: vmset is not reliable with hostnames

We cannot be guaranteed that the lead broker is flux00000 so we need
to provide automation to update/fix the issue. I am also adding a
script for OSU to install the benchmarks

Signed-off-by: vsoch <[email protected]>

Author: vsoch

tutorial/azure/README.md

@@ -71,15 +71,81 @@ follower_address=$(az vmss list-instance-public-ips -g terraform-testing -n flux
 ssh -i ./id_azure azureuser@${follower_address}
 ```
 
+Note that if the lead broker doesn't come up as flux_0 (flux with all zeros, Azure is not predicable like that) we will need to update.
+
+```bash
+lead_broker=$(az vmss list-instances -g terraform-testing -n flux | jq -r .[0].osProfile.computerName)
+echo "The lead broker is ${lead_broker}"
+```
+
+Here is how you can fix all your brokers:
+
+```bash
+for address in $(az vmss list-instance-public-ips -g terraform-testing -n flux | jq -r .[].ipAddress)
+ do
+   echo "Updating $address"
+   scp -i ./id_azure update_brokers.sh azureuser@${address}:/tmp/update_brokers.sh
+   ssh -i ./id_azure azureuser@$address "/bin/bash /tmp/update_brokers.sh flux $lead_broker"
+done
+```
+
+Note that I've also provided a script to install the OSU benchmarks with the same strategy above:
+
+```bash
+for address in $(az vmss list-instance-public-ips -g terraform-testing -n flux | jq -r .[].ipAddress)
+ do
+   echo "Updating $address"
+   scp -i ./id_azure install_osu.sh azureuser@${address}:/tmp/install_osu.sh
+   ssh -i ./id_azure azureuser@$address "/bin/bash /tmp/install_osu.sh"
+done
+```
+
+This installs to /usr/local/libexec/osu-benchmarks/mpi.
+
 ### 3. Checks
 
 Check the cluster status, the overlay status, and try running a job:
 
 ```bash
-$ flux resource list
+flux resource list
 ```
 ```bash
-$ flux run -N 2 hostname
+flux run -N 2 hostname
+```
+
+Try running a benchmark!
+
+```bash
+flux run -N2 /usr/local/libexec/osu-micro-benchmarks/mpi/collective/osu_allreduce 
+flux run -N2 -n2 /usr/local/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_latency
+```
+```console
+# OSU MPI Latency Test v5.8
+# Size          Latency (us)
+0                       1.57
+1                       1.56
+2                       1.56
+4                       1.56
+8                       1.57
+16                      1.57
+32                      1.70
+64                      1.76
+128                     1.80
+256                     2.31
+512                     2.36
+1024                    2.52
+2048                    2.70
+4096                    3.46
+8192                    3.96
+16384                   5.24
+32768                   6.85
+65536                   9.18
+131072                 14.20
+262144                 17.30
+524288                 27.94
+1048576                50.00
+2097152                92.04
+4194304               177.34
 ```
 
 ### 4. Cleanup

tutorial/azure/install_osu.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+
+cd /tmp
+OSU_VERSION=5.8
+wget http://mvapich.cse.ohio-state.edu/download/mvapich/osu-micro-benchmarks-$OSU_VERSION.tgz
+tar zxvf ./osu-micro-benchmarks-5.8.tgz
+cd osu-micro-benchmarks-5.8/
+./configure CC=mpicc CXX=mpicxx
+make -j 4 && sudo make install
+
+# installs to /usr/local/libexec/osu-benchmarks/mpi

tutorial/azure/main.tf

@@ -71,7 +71,6 @@ locals {
   tags = {
     flux_core = "0-68-0"
   }
-  application_port = 8081
 }
 
 resource "random_pet" "id" {}
@@ -167,6 +166,10 @@ resource "azurerm_linux_virtual_machine_scale_set" "vmss" {
     caching              = "ReadWrite"
   }
 
+  identity {
+    type = "SystemAssigned"
+  }
+
   admin_ssh_key {
     username   = local.admin_user
     public_key = azapi_resource_action.ssh_public_key_gen.output.publicKey

tutorial/azure/start-script.sh

@@ -1,8 +1,13 @@
 #!/bin/bash
 
+# In case the user wants to play with this.
+sudo pip install azure-cli
+
 # Assume a huge number. This will error with Azure because they 
 # eventually dive into alpha numeric, but this works for a small demo
 NODELIST=${template_name}000[000-999]
+
+# The lead broker can be anything, azure is not predictable
 lead_broker=${template_name}000000
 
 flux R encode --hosts=$NODELIST --local > R

tutorial/azure/update_brokers.sh

@@ -0,0 +1,64 @@
+#!/bin/bash
+
+template_name=${1}
+lead_broker=${2}
+template_username=${3:-azureuser}
+template_ethernet_device=${4:-eth0}
+start_number=$(echo "${lead_broker/flux/""}")
+start_number=$(echo "${start_number/000/""}")
+
+# Assume a huge number. This will error with Azure because they 
+# eventually dive into alpha numeric, but this works for a small demo
+NODELIST=${template_name}000[$start_number-999]
+
+# Write updated resource file
+flux R encode --hosts=$NODELIST --local > R
+sudo mv R /etc/flux/system/R
+sudo chown ${template_username} /etc/flux/system/R
+
+# Write updated broker.toml
+cat <<EOF | tee /tmp/broker.toml
+# Flux needs to know the path to the IMP executable
+[exec]
+imp = "/usr/libexec/flux/flux-imp"
+
+# Allow users other than the instance owner (guests) to connect to Flux
+# Optionally, root may be given "owner privileges" for convenience
+[access]
+allow-guest-user = true
+allow-root-owner = true
+
+# Point to resource definition generated with flux-R(1).
+# Uncomment to exclude nodes (e.g. mgmt, login), from eligibility to run jobs.
+[resource]
+path = "/etc/flux/system/R"
+
+# Point to shared network certificate generated flux-keygen(1).
+# Define the network endpoints for Flux's tree based overlay network
+# and inform Flux of the hostnames that will start flux-broker(1).
+[bootstrap]
+curve_cert = "/etc/flux/system/curve.cert"
+
+default_port = 8050
+default_bind = "tcp://${template_ethernet_device}:%p"
+default_connect = "tcp://%h:%p"
+
+# Rank 0 is the TBON parent of all brokers unless explicitly set with
+# parent directives.
+# The actual ip addresses (for both) need to be added to /etc/hosts
+# of each VM for now.
+hosts = [
+   { host = "$NODELIST" },
+]
+# Speed up detection of crashed network peers (system default is around 20m)
+[tbon]
+tcp_user_timeout = "2m"
+EOF
+
+sudo mv /tmp/broker.toml /etc/flux/system/conf.d/broker.toml
+
+# See the README.md for commands how to set this manually without systemd
+sudo systemctl daemon-reload
+sudo systemctl restart flux.service
+sleep 2
+sudo systemctl status flux.service