Merge pull request #2386 from pareenaverma/main

ONNX on Azure Tech Review

Author: pareenaverma

content/learning-paths/servers-and-cloud-computing/onnx-on-azure/_index.md

@@ -7,22 +7,22 @@ cascade:
 
 minutes_to_complete: 60   
 
-who_is_this_for: This Learning Path introduces ONNX deployment on Microsoft Azure Cobalt 100 (Arm-based) virtual machines. It is designed for developers migrating ONNX-based applications from x86_64 to Arm with minimal or no changes.
+who_is_this_for: This Learning Path introduces ONNX deployment on Microsoft Azure Cobalt 100 (Arm-based) virtual machines. It is designed for developers deploying ONNX-based applications on Arm-based machines.
 
 learning_objectives:
     - Provision an Azure Arm64 virtual machine using Azure console, with Ubuntu Pro 24.04 LTS as the base image.
     - Deploy ONNX on the Ubuntu Pro virtual machine.
-    - Perform ONNX baseline testing and benchmarking on both x86_64 and Arm64 virtual machines.
+    - Perform ONNX baseline testing and benchmarking on Arm64 virtual machines.
 
 prerequisites:
     - A [Microsoft Azure](https://azure.microsoft.com/) account with access to Cobalt 100 based instances (Dpsv6).
     - Basic understanding of Python and machine learning concepts.
     - Familiarity with [ONNX Runtime](https://onnxruntime.ai/docs/) and Azure cloud services.
 
-author: Jason Andrews
+author: Pareena Verma    
 
 ### Tags
-skilllevels: Advanced
+skilllevels: Introductory
 subjects: ML
 cloud_service_providers: Microsoft Azure
 

content/learning-paths/servers-and-cloud-computing/onnx-on-azure/background.md

@@ -6,7 +6,7 @@ weight: 2
 layout: "learningpathall"
 ---
 
-## Cobalt 100 Arm-based processor
+## Azure Cobalt 100 Arm-based processor
 
 Azure’s Cobalt 100 is built on Microsoft's first-generation, in-house Arm-based processor: the Cobalt 100. Designed entirely by Microsoft and based on Arm’s Neoverse N2 architecture, this 64-bit CPU delivers improved performance and energy efficiency across a broad spectrum of cloud-native, scale-out Linux workloads. These include web and application servers, data analytics, open-source databases, caching systems, and more. Running at 3.4 GHz, the Cobalt 100 processor allocates a dedicated physical core for each vCPU, ensuring consistent and predictable performance. 
 
@@ -16,6 +16,6 @@ To learn more about Cobalt 100, refer to the blog [Announcing the preview of new
 ONNX (Open Neural Network Exchange) is an open-source format designed for representing machine learning models.  
 It provides interoperability between different deep learning frameworks, enabling models trained in one framework (such as PyTorch or TensorFlow) to be deployed and run in another.  
 
-ONNX models are serialized into a standardized format that can be executed by the **ONNX Runtime**, a high-performance inference engine optimized for CPU, GPU, and specialized hardware accelerators. This separation of model training and inference allows developers to build flexible, portable, and production-ready AI workflows.  
+ONNX models are serialized into a standardized format that can be executed by the ONNX Runtime, a high-performance inference engine optimized for CPU, GPU, and specialized hardware accelerators. This separation of model training and inference allows developers to build flexible, portable, and production-ready AI workflows.  
 
 ONNX is widely used in cloud, edge, and mobile environments to deliver efficient and scalable inference for deep learning models. Learn more from the [ONNX official website](https://onnx.ai/) and the [ONNX Runtime documentation](https://onnxruntime.ai/docs/).    

content/learning-paths/servers-and-cloud-computing/onnx-on-azure/baseline.md

@@ -7,12 +7,11 @@ layout: learningpathall
 ---
 
 
-## Baseline testing using ONNX Runtime: 
+## Baseline Testing using ONNX Runtime: 
 
-This test measures the inference latency of the ONNX Runtime by timing how long it takes to process a single input using the `squeezenet-int8.onnx model`. It helps evaluate how efficiently the model runs on the target hardware.
-
-Create a **baseline.py** file with the below code for baseline test of ONNX:
+The purpose of this test is to measure the inference latency of ONNX Runtime on your Azure Cobalt 100 VM. By timing how long it takes to process a single input through the SqueezeNet INT8 model, you can validate that ONNX Runtime is functioning correctly and get a baseline performance measurement for your target hardware.
 
+Create a file named `baseline.py` with the following code:
 ```python
 import onnxruntime as ort
 import numpy as np
@@ -29,12 +28,12 @@ end = time.time()
 print("Inference time:", end - start)
 ```
 
-Run the baseline test:
+Run the baseline script to measure inference time:
 
 ```console
 python3 baseline.py
 ```
-You should see an output similar to:
+You should see output similar to:
 ```output
 Inference time: 0.0026061534881591797
 ```
@@ -45,8 +44,11 @@ input tensor of shape (1, 3, 224, 224):
 - 224 x 224: image resolution (common for models like SqueezeNet)
 {{% /notice %}}
 
+This indicates the model successfully executed a single forward pass through the SqueezeNet INT8 ONNX model and returned results.
+
 #### Output summary:
 
-- Single inference latency: ~2.60 milliseconds (0.00260 sec) 
-- This shows the initial (cold-start) inference performance of ONNX Runtime on CPU using an optimized int8 quantized model. 
-- This demonstrates that the setup is fully working, and ONNX Runtime efficiently executes quantized models on Arm64. 
+Single inference latency(0.00260 sec): This is the time required for the model to process one input image and produce an output. The first run includes graph loading, memory allocation, and model initialization overhead.
+Subsequent inferences are usually faster due to caching and optimized execution.
+
+This demonstrates that the setup is fully working, and ONNX Runtime efficiently executes quantized models on Arm64. 

content/learning-paths/servers-and-cloud-computing/onnx-on-azure/benchmarking.md

@@ -6,59 +6,63 @@ weight: 6
 layout: learningpathall
 ---
 
-Now that you’ve set up and run the ONNX model (e.g., SqueezeNet), you can use it to benchmark inference performance using Python-based timing or tools like **onnxruntime_perf_test**. This helps evaluate the ONNX Runtime efficiency on Azure Arm64-based Cobalt 100 instances.
-
-You can also compare the inference time between Cobalt 100 (Arm64) and similar D-series x86_64-based virtual machine on Azure.
+Now that you have validated ONNX Runtime with Python-based timing (e.g., SqueezeNet baseline test), you can move to using a dedicated benchmarking utility called `onnxruntime_perf_test`. This tool is designed for systematic performance evaluation of ONNX models, allowing you to capture more detailed statistics than simple Python timing.
+This helps evaluate the ONNX Runtime efficiency on Azure Arm64-based Cobalt 100 instances and other x86_64 instances. architectures.
 
 ## Run the performance tests using onnxruntime_perf_test
-The **onnxruntime_perf_test** is a performance benchmarking tool included in the ONNX Runtime source code. It is used to measure the inference performance of ONNX models under various runtime conditions (like CPU, GPU, or other execution providers).
+The `onnxruntime_perf_test` is a performance benchmarking tool included in the ONNX Runtime source code. It is used to measure the inference performance of ONNX models and supports multiple execution providers (like CPU, GPU, or other execution providers). on Arm64 VMs, CPU execution is the focus.
 
 ### Install Required Build Tools
+Before building or running `onnxruntime_perf_test`, you will need to install a set of development tools and libraries. These packages are required for compiling ONNX Runtime and handling model serialization via Protocol Buffers.
 
 ```console
 sudo apt update
 sudo apt install -y build-essential cmake git unzip pkg-config
 sudo apt install -y protobuf-compiler libprotobuf-dev libprotoc-dev git
 ```
-Then verify:
+Then verify protobuf installation:
 ```console
 protoc --version
 ```
-You should see an output similar to:
+You should see output similar to:
 
 ```output
 libprotoc 3.21.12
 ```
 ### Build ONNX Runtime from Source:
 
-The benchmarking tool, **onnxruntime_perf_test**, isn’t available as a pre-built binary artifact for any platform. So, you have to build it from the source, which is expected to take around 40-50 minutes. 
+The benchmarking tool `onnxruntime_perf_test`, isn’t available as a pre-built binary for any platform. So, you will have to build it from the source, which is expected to take around 40 minutes. 
 
-Clone onnxruntime:
+Clone onnxruntime repo:
 ```console
 git clone --recursive https://github.com/microsoft/onnxruntime 
 cd onnxruntime
 ```
-Now, build the benchmark as below:
+Now, build the benchmark tool:
 
 ```console
 ./build.sh --config Release --build_dir build/Linux --build_shared_lib --parallel --build --update --skip_tests 
 ```
-This will build the benchmark tool inside ./build/Linux/Release/onnxruntime_perf_test. 
+You should see the executable at:
+```output
+./build/Linux/Release/onnxruntime_perf_test
+```
 
 ### Run the benchmark
-Now that the benchmarking tool has been built, you can benchmark the **squeezenet-int8.onnx** model, as below:
+Now that you have built the benchmarking tool, you can run inference benchmarks on the SqueezeNet INT8 model:
 
 ```console
-./build/Linux/Release/onnxruntime_perf_test -e cpu -r 100 -m times -s -Z -I <path-to-squeezenet-int8.onnx>
+./build/Linux/Release/onnxruntime_perf_test -e cpu -r 100 -m times -s -Z -I ../squeezenet-int8.onnx
 ```
-- **e cpu**: Use the CPU execution provider (not GPU or any other backend). 
-- **r 100**: Run 100 inferences. 
-- **m times**: Use "repeat N times" mode. 
-- **s**: Show detailed statistics. 
-- **Z**: Disable intra-op thread spinning (reduces CPU usage when idle between runs). 
-- **I**: Input the ONNX model path without using input/output test data.
+Breakdown of the flags:
+  -e cpu → Use the CPU execution provider.
+  -r 100 → Run 100 inference passes for statistical reliability.
+  -m times → Run in “repeat N times” mode. Useful for latency-focused measurement.
+  -s → Show detailed per-run statistics (latency distribution).
+  -Z → Disable intra-op thread spinning. Reduces CPU waste when idle between runs, especially on high-core systems like Cobalt 100.
+  -I → Input the ONNX model path directly, skipping pre-generated test data.
 
-You should see an output similar to:
+You should see output similar to:
 
 ```output
 Disabling intra-op thread spinning between runs
@@ -84,12 +88,12 @@ P999 Latency: 0.00190312 s
 ```
 ### Benchmark Metrics Explained  
 
-- **Average Inference Time**: The mean time taken to process a single inference request across all runs. Lower values indicate faster model execution.  
-- **Throughput**: The number of inference requests processed per second. Higher throughput reflects the model’s ability to handle larger workloads efficiently.  
-- **CPU Utilization**: The percentage of CPU resources used during inference. A value close to 100% indicates full CPU usage, which is expected during performance benchmarking.  
-- **Peak Memory Usage**: The maximum amount of system memory (RAM) consumed during inference. Lower memory usage is beneficial for resource-constrained environments. 
-- **P50 Latency (Median Latency)**: The time below which 50% of inference requests complete. Represents typical latency under normal load.   
-- **Latency Consistency**: Describes the stability of latency values across all runs. "Consistent" indicates predictable inference performance with minimal jitter.  
+  * Average Inference Time: The mean time taken to process a single inference request across all runs. Lower values indicate faster model execution.  
+  * Throughput: The number of inference requests processed per second. Higher throughput reflects the model’s ability to handle larger workloads efficiently.  
+  * CPU Utilization: The percentage of CPU resources used during inference. A value close to 100% indicates full CPU usage, which is expected during performance benchmarking.  
+  * Peak Memory Usage: The maximum amount of system memory (RAM) consumed during inference. Lower memory usage is beneficial for resource-constrained environments. 
+  * P50 Latency (Median Latency): The time below which 50% of inference requests complete. Represents typical latency under normal load.   
+  * Latency Consistency: Describes the stability of latency values across all runs. "Consistent" indicates predictable inference performance with minimal jitter.  
 
 ### Benchmark summary on Arm64:
 Here is a summary of benchmark results collected on an Arm64 **D4ps_v6 Ubuntu Pro 24.04 LTS virtual machine**.
@@ -109,30 +113,12 @@ Here is a summary of benchmark results collected on an Arm64 **D4ps_v6 Ubuntu Pr
 | **Latency Consistency**    | Consistent                   |
 
 
-### Benchmark summary on x86
-Here is a summary of benchmark results collected on x86 **D4s_v6 Ubuntu Pro 24.04 LTS virtual machine**.
-
-| **Metric**                | **Value on Virtual Machine** |
-|----------------------------|-------------------------------|
-| **Average Inference Time** | 1.413 ms                     |
-| **Throughput**             | 707.48 inferences/sec        |
-| **CPU Utilization**        | 100%                         |
-| **Peak Memory Usage**      | 38.80 MB                     |
-| **P50 Latency**            | 1.396 ms                     |
-| **P90 Latency**            | 1.501 ms                     |
-| **P95 Latency**            | 1.520 ms                     |
-| **P99 Latency**            | 1.794 ms                     |
-| **P999 Latency**           | 1.794 ms                     |
-| **Max Latency**            | 1.794 ms                     |
-| **Latency Consistency**    | Consistent                   |
-
-
-### Highlights from Ubuntu Pro 24.04 Arm64 Benchmarking
+### Highlights from Benchmarking on Azure Cobalt 100 Arm64 VMs
 
-When comparing the results on Arm64 vs x86_64 virtual machines:
-- **Low-Latency Inference:** Achieved consistent average inference times of ~1.86 ms on Arm64.  
-- **Strong and Stable Throughput:** Sustained throughput of over 538 inferences/sec using the `squeezenet-int8.onnx` model on D4ps_v6 instances.  
-- **Lightweight Resource Footprint:** Peak memory usage stayed below 37 MB, with CPU utilization around 96%, ideal for efficient edge or cloud inference.  
-- **Consistent Performance:** P50, P95, and Max latency remained tightly bound, showcasing reliable performance on Azure Cobalt 100 Arm-based infrastructure.
+The results on Arm64 virtual machines demonstrate:
+- Low-Latency Inference: Achieved consistent average inference times of ~1.86 ms on Arm64.  
+- Strong and Stable Throughput: Sustained throughput of over 538 inferences/sec using the `squeezenet-int8.onnx` model on D4ps_v6 instances.  
+- Lightweight Resource Footprint: Peak memory usage stayed below 37 MB, with CPU utilization around 96%, ideal for efficient edge or cloud inference.  
+- Consistent Performance: P50, P95, and Max latency remained tightly bound, showcasing reliable performance on Azure Cobalt 100 Arm-based infrastructure.
 
-You have now benchmarked ONNX on an Azure Cobalt 100 Arm64 virtual machine and compared results with x86_64.
+You have now successfully benchmarked inference time of ONNX models on an Azure Cobalt 100 Arm64 virtual machine.

content/learning-paths/servers-and-cloud-computing/onnx-on-azure/create-instance.md

@@ -1,18 +1,24 @@
 ---
-title: Create an Arm based cloud virtual machine using Microsoft Cobalt 100 CPU 
+title: Create an Arm-based Azure VM with Cobalt 100
 weight: 3
 
 ### FIXED, DO NOT MODIFY
 layout: learningpathall
 ---
 
-## Introduction
+## Set up your development environment
 
-There are several ways to create an Arm-based Cobalt 100 virtual machine : the Microsoft Azure console, the Azure CLI tool, or using your choice of IaC (Infrastructure as Code). This guide will use the Azure console to create a virtual machine with Arm-based Cobalt 100 Processor. 
+There is more than one way to create an Arm-based Cobalt 100 virtual machine: 
 
-This learning path focuses on the general-purpose virtual machine of the D series. Please read the guide on [Dpsv6 size series](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/general-purpose/dpsv6-series) offered by Microsoft Azure.  
+- The Microsoft Azure portal
+- The Azure CLI
+- Your preferred infrastructure as code (IaC) tool
 
-If you have never used the Microsoft Cloud Platform before, please review the microsoft [guide to Create a Linux virtual machine in the Azure portal](https://learn.microsoft.com/en-us/azure/virtual-machines/linux/quick-create-portal?tabs=ubuntu). 
+In this Learning Path, you will use the Azure portal to create a virtual machine with the Arm-based Azure Cobalt 100 processor. 
+
+You will focus on the general-purpose virtual machines in the D-series. For further information, see the Microsoft Azure guide for the [Dpsv6 size series](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/general-purpose/dpsv6-series).
+
+While the steps to create this instance are included here for convenience, for further information on setting up Cobalt on Azure, see [Deploy a Cobalt 100 virtual machine on Azure Learning Path](/learning-paths/servers-and-cloud-computing/cobalt/).
 
 #### Create an Arm-based Azure Virtual Machine