osu re-runs - not a success

I brought up the cluster, verified that infiniband installed
successfully, and then tested osu. There were consistent segfaults
across the board, and only one set of nodes that worked.

Signed-off-by: vsoch <[email protected]>

Author: vsoch

experiments/azure/aks/cpu/size256/README.md

@@ -29,6 +29,25 @@ az aks create \
 az aks get-credentials --resource-group aks-gpu-testing-west --name aks-gpu-testing-cluster
 ```
 
+And for the redo run in March 2025 for OSU:
+
+```bash
+az aks create \
+    --resource-group flux-usernetes \
+    --name performance-study-256 \
+    --ppg /subscriptions/3e173a37-8f81-492f-a234-ca727b72e6f8/resourceGroups/flux-usernetes/providers/Microsoft.Compute/proximityPlacementGroups/aks-placement-test \
+    --network-plugin azure \
+    --node-count 99 \
+    --node-vm-size standard_hb120-96rs_v3 \
+    --location southcentralus \
+    --enable-node-public-ip \
+    --vm-set-type VirtualMachineScaleSets \
+    --load-balancer-sku standard \
+    --generate-ssh-keys
+```
+
+You'll need to manually scale up to 256 for the VMSet.
+
 ### 1. Setup
 
 Note that I needed to create this entirely in the UI, and you can't do it automatically. We are required to have at least one node in the agent pool. For testing I used one static, and for production I allowed autoscaling 1-3, not knowing what might be needed. Once your deployment is ready and you can use the Connect -> cloud shell to connect, register the feature for AKSInfinibandSupport:
@@ -54,6 +73,12 @@ Run this final step:
 az provider register --namespace Microsoft.ContainerService
 ```
 
+This was an extra command needed the second time to get the credentials:
+
+```bash
+az aks get-credentials --name performance-study-256 --resource-group flux-usernetes
+```
+
 Note that if you shell in now and install `ibverbs-utils` and do `ibv_devices` it will be empty.
 If you are doing this in the cloud shell, you'll next want to copy the entirety of the `~/.kube/config` to your local machine to access the cluster. Let's try to install infiniband next, and we will use a container that is also built with ubuntu 22.04 drivers.
 
@@ -103,7 +128,7 @@ Now we are ready for different MiniCluster setups. For each of the below, to she
 ```bash
 kubectl exec -it flux-sample-0-xxx bash
 ```
-Next, choose a cluster size in one of the experiment folders.
+
 Monitoring:
 
 ```bash
@@ -114,12 +139,6 @@ kubectl create namespace monitoring
 kubectl apply -f deploy
 ```
 
-Install the Flux Operator:
-
-```bash
-kubectl apply -f ./flux-operator.yaml
-```
-
 Now we are ready for different MiniCluster setups. For each of the below, to shell in to the lead broker (index 0) you do:
 
 ```bash
@@ -184,6 +203,8 @@ kubectl delete -f crd/single-node.yaml
 
 #### OSU
 
+Note that this second time, the container pull took over 10 minutes.
+
 ```bash
 kubectl logs -n monitoring event-exporter-6bf9c87d4d-v4rtr -f  |& tee ./events-osu-$(date +%s).json
 kubectl apply -f ./crd/osu.yaml
@@ -236,13 +257,27 @@ export app=osu
 output=./results/$app
 mkdir -p $output
 
+chmod +x flux-run-combinations.sh
 ./flux-run-combinations.sh 256 $app
 
 for i in $(seq 1 5); do     
   echo "Running iteration $i"
     time flux run --setattr=user.study_id=$app-256-iter-$i -N256 -n 24576 -o cpu-affinity=per-task /opt/osu-benchmark/build.openmpi/mpi/collective/osu_allreduce
 done
 
+# Just successful ones
+for jobid in $(flux jobs --filter=completed --json | jq -r .jobs[].id)
+  do
+    # Get the job study id
+    study_id=$(flux job info $jobid jobspec | jq -r ".attributes.user.study_id")    
+    echo "Parsing jobid ${jobid} and study id ${study_id}"
+    flux job attach $jobid &> $output/${study_id}-${jobid}.out 
+    echo "START OF JOBSPEC" >> $output/${study_id}-${jobid}.out 
+    flux job info $jobid jobspec >> $output/${study_id}-${jobid}.out 
+    echo "START OF EVENTLOG" >> $output/${study_id}-${jobid}.out 
+    flux job info $jobid guest.exec.eventlog >> $output/${study_id}-${jobid}.out
+done
+
 # When they are done:
 for jobid in $(flux jobs -a --json | jq -r .jobs[].id)
   do
@@ -257,6 +292,7 @@ for jobid in $(flux jobs -a --json | jq -r .jobs[].id)
 done
 
 oras push ghcr.io/converged-computing/metrics-operator-experiments/performance:aks-infiniband-cpu-256-$app $output
+oras push ghcr.io/converged-computing/metrics-operator-experiments/performance:aks-infiniband-cpu-256-$app-rerun $output
 ```
 ```bash
 kubectl delete -f ./crd/osu.yaml
@@ -578,6 +614,9 @@ oras push ghcr.io/converged-computing/metrics-operator-experiments/performance:a
 ```bash
 kubectl delete -f ./crd/quicksilver.yaml
 ```
+
+The second attempt was not successful. The creation was different (I had to manually ask for credentials) and all of the osu tests segfaulted except for one. I ran the script 4x to see if more successful results occurred and then gave up, as it's an expensive cluster.
+
 ### Clean Up
 
 When you are done, delete the cluster from the web interface.

experiments/azure/aks/cpu/size256/results/osu-rerun/osu-2-iter-1-684644630528.out

@@ -0,0 +1,37 @@
+# OSU MPI Latency Test v5.8
+# Size          Latency (us)
+0                       1.59
+1                       1.58
+2                       1.58
+4                       1.58
+8                       1.59
+16                      1.59
+32                      1.74
+64                      1.79
+128                     1.85
+256                     2.35
+512                     2.47
+1024                    2.58
+2048                    2.75
+4096                    3.48
+8192                    4.09
+16384                   5.10
+32768                   6.59
+65536                   9.01
+131072                 13.22
+262144                 17.26
+524288                 27.81
+1048576                49.79
+2097152                92.36
+4194304               177.80
+START OF JOBSPEC
+{"resources": [{"type": "node", "count": 2, "with": [{"type": "slot", "count": 1, "with": [{"type": "core", "count": 1}], "label": "task"}]}], "tasks": [{"command": ["/opt/osu-benchmark/build.openmpi/mpi/pt2pt/osu_latency"], "slot": "task", "count": {"per_slot": 1}}], "attributes": {"system": {"duration": 0, "cwd": "/opt/azhpc-images/ubuntu/ubuntu-22.x/ubuntu-22.04-hpc", "shell": {"options": {"rlimit": {"cpu": -1, "fsize": -1, "data": -1, "stack": 8388608, "core": -1, "nofile": 1048576, "as": -1, "rss": -1, "nproc": -1}, "cpu-affinity": "per-task"}}, "constraints": {"hostlist": ["flux-sample-0,flux-sample-29"]}}, "user": {"study_id": "osu-2-iter-1"}}, "version": 1}
+START OF EVENTLOG
+{"timestamp":1743294664.2062294,"name":"init"}
+{"timestamp":1743294664.2068343,"name":"starting"}
+{"timestamp":1743294664.2358925,"name":"shell.init","context":{"service":"0-shell-fJz6hgzj","leader-rank":0,"size":2}}
+{"timestamp":1743294664.2392285,"name":"shell.start","context":{"taskmap":{"version":1,"map":[[0,2,1,1]]}}}
+{"timestamp":1743294666.3936222,"name":"shell.task-exit","context":{"localid":0,"rank":0,"state":"Exited","pid":366,"wait_status":0,"signaled":0,"exitcode":0}}
+{"timestamp":1743294666.4000165,"name":"complete","context":{"status":0}}
+{"timestamp":1743294666.4000454,"name":"done"}
+

experiments/azure/aks/cpu/size256/results/osu-rerun/osu-2-iter-1-723500662784.out

@@ -0,0 +1,36 @@
+# OSU MPI Bandwidth Test v5.8
+# Size      Bandwidth (MB/s)
+1                       3.85
+2                       7.87
+4                      15.51
+8                      31.56
+16                     62.61
+32                    121.06
+64                    243.40
+128                   488.32
+256                   925.33
+512                  1724.95
+1024                 3260.50
+2048                 5980.09
+4096                 9182.91
+8192                13711.41
+16384               18938.37
+32768               21164.41
+65536               22852.02
+131072              23628.27
+262144              24008.90
+524288              24114.70
+1048576             24228.32
+2097152             24397.00
+4194304             24486.00
+START OF JOBSPEC
+{"resources": [{"type": "node", "count": 2, "with": [{"type": "slot", "count": 1, "with": [{"type": "core", "count": 1}], "label": "task"}]}], "tasks": [{"command": ["/opt/osu-benchmark/build.openmpi/mpi/pt2pt/osu_bw"], "slot": "task", "count": {"per_slot": 1}}], "attributes": {"system": {"duration": 0, "cwd": "/opt/azhpc-images/ubuntu/ubuntu-22.x/ubuntu-22.04-hpc", "shell": {"options": {"rlimit": {"cpu": -1, "fsize": -1, "data": -1, "stack": 8388608, "core": -1, "nofile": 1048576, "as": -1, "rss": -1, "nproc": -1}, "cpu-affinity": "per-task"}}, "constraints": {"hostlist": ["flux-sample-0,flux-sample-29"]}}, "user": {"study_id": "osu-2-iter-1"}}, "version": 1}
+START OF EVENTLOG
+{"timestamp":1743294666.520262,"name":"init"}
+{"timestamp":1743294666.5208848,"name":"starting"}
+{"timestamp":1743294666.5504811,"name":"shell.init","context":{"service":"0-shell-fL1JH1fM","leader-rank":0,"size":2}}
+{"timestamp":1743294666.5542791,"name":"shell.start","context":{"taskmap":{"version":1,"map":[[0,2,1,1]]}}}
+{"timestamp":1743294667.665215,"name":"shell.task-exit","context":{"localid":0,"rank":0,"state":"Exited","pid":372,"wait_status":0,"signaled":0,"exitcode":0}}
+{"timestamp":1743294667.6716905,"name":"complete","context":{"status":0}}
+{"timestamp":1743294667.6717153,"name":"done"}
+

paper/osu-benchmarks/1-run-analysis.py

@@ -237,6 +237,17 @@ def parse_data(indir, outdir, files):
         if exp.size == 2:
             continue
 
+        # We reran google cloud GPU size 16 with fixed results for latency and bw.
+        if (
+            exp.cloud == "google"
+            and exp.env_type == "gpu"
+            and exp.size == 16
+            and "osu-allreduce" not in filename
+            and "osu-fixed" not in filename
+        ):
+            print(f"Skipping old result {filename}")
+            continue
+
         # Cyclecloud GPU had multiple types, osu-dd and osu-hh
         # For Google Cloud GPU on GKE the point to point benchmarks
         # were run without GPU - never worked