Merge pull request #2544 from nikhil-arm/vllm_accuracy_comparison

[feat]: add vllm accuracy benchmarking flow

Author: pareenaverma

content/learning-paths/servers-and-cloud-computing/vllm-acceleration/4-accuracy-benchmarking.md

@@ -0,0 +1,113 @@
+---
+title: Evaluate Accuracy with LM Evaluation Harness
+weight: 5
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Why accuracy benchmarking
+
+The lm-evaluation-harness is the standard way to measure model accuracy across common academic benchmarks (for example, MMLU, HellaSwag, GSM8K) and runtimes (Hugging Face, vLLM, llama.cpp, etc.). In this module, you will run accuracy tests for both BF16 and INT4 deployments of your model served by vLLM on Arm-based servers.
+
+You will:
+  * Install lm-eval-harness with vLLM support
+  * Run benchmarks on a BF16 model and an INT4 (weight-quantized) model
+  * Interpret key metrics and compare quality across precisions
+
+{{% notice Note %}}
+Results depend on CPU, dataset versions, and model choice. Use the same tasks and few-shot settings when comparing BF16 and INT4 to ensure a fair comparison.
+{{% /notice %}}
+
+## Prerequisites
+
+Before you start:
+  * Complete the optimized build in “Overview and Optimized Build” and validate your vLLM install.
+  * Optionally quantize a model using the “Quantize an LLM to INT4 for Arm Platform” module. We’ll reference the output directory name from that step.
+
+## Install lm-eval-harness
+
+Install the harness with vLLM extras in your active Python environment:
+
+```bash
+pip install "lm_eval[vllm]"
+pip install ray
+```
+
+{{% notice Tip %}}
+If your benchmarks include gated models or datasets, run `huggingface-cli login` first so the harness can download what it needs.
+{{% /notice %}}
+
+## Recommended runtime settings for Arm CPU
+
+Export the same performance-oriented environment variables used for serving. These enable Arm-optimized kernels through oneDNN+ACL and consistent thread pinning:
+
+```bash
+export VLLM_TARGET_DEVICE=cpu
+export VLLM_CPU_KVCACHE_SPACE=32
+export VLLM_CPU_OMP_THREADS_BIND="0-$(($(nproc)-1))"
+export VLLM_MLA_DISABLE=1
+export ONEDNN_DEFAULT_FPMATH_MODE=BF16
+export OMP_NUM_THREADS="$(nproc)"
+export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libtcmalloc_minimal.so.4
+```
+
+{{% notice Note %}}
+`LD_PRELOAD` uses tcmalloc to reduce allocator contention. Install it via `sudo apt-get install -y libtcmalloc-minimal4` if you haven’t already.
+{{% /notice %}}
+
+## Accuracy Benchmarking Meta‑Llama‑3.1‑8B‑Instruct BF16 model
+
+Run with a non-quantized model. Replace the model ID as needed.
+
+```bash
+lm_eval \
+  --model vllm \
+  --model_args \
+    pretrained=meta-llama/Meta-Llama-3.1-8B-Instruct,dtype=bfloat16,max_model_len=4096,enforce_eager=True \
+  --tasks mmlu,hellaswag \
+  --batch_size auto \
+  --output_path results
+```
+
+## Accuracy Benchmarking INT4 quantized model
+
+Use the INT4 quantization recipe & script from previous steps to quantize `meta-llama/Meta-Llama-3.1-8B-Instruct` model
+
+Channelwise INT4 (MSE):
+
+```bash
+lm_eval \
+  --model vllm \
+  --model_args \
+    pretrained=Meta-Llama-3.1-8B-Instruct-w4a8dyn-mse-channelwise,dtype=float32,max_model_len=4096,enforce_eager=True \
+  --tasks mmlu,hellaswag \
+  --batch_size auto \
+  --output_path results
+```
+
+## Interpreting results
+
+The harness prints per-task and aggregate scores (for example, `acc`, `acc_norm`, `exact_match`). Higher is generally better. Compare BF16 vs INT4 on the same tasks to assess quality impact.
+
+Practical tips:
+  * Use the same tasks and few-shot settings across runs.
+  * For quick iteration, you can add `--limit 200` to run on a subset.
+
+## Example results for Meta‑Llama‑3.1‑8B‑Instruct model
+
+These illustrative results are representative; actual scores may vary across hardware, dataset versions, and harness releases. Higher values indicate better accuracy.
+
+| Variant                         | MMLU (acc±err)    | HellaSwag (acc±err) |
+|---------------------------------|-------------------|---------------------|
+| BF16                            | 0.5897 ± 0.0049   | 0.7916 ± 0.0041     |
+| INT4 Groupwise minmax (G=32)    | 0.5831 ± 0.0049   | 0.7819 ± 0.0041     |
+| INT4 Channelwise MSE            | 0.5712 ± 0.0049   | 0.7633 ± 0.0042     |
+
+Use these as ballpark expectations to check whether your runs are in a reasonable range, not as official targets.
+
+## Next steps
+
+  * Try additional tasks to match your usecase: `gsm8k`, `winogrande`, `arc_easy`, `arc_challenge`.
+  * Sweep quantization recipes (minmax vs mse; channelwise vs groupwise, group size) to find a better accuracy/performance balance.
+  * Record both throughput and accuracy to choose the best configuration for your workload.

content/learning-paths/servers-and-cloud-computing/vllm-acceleration/_index.md

@@ -7,14 +7,15 @@ cascade:
 
 minutes_to_complete: 60
 
-who_is_this_for: This learning path is designed for software developers and AI engineers who want to build and optimize vLLM for Arm-based servers, quantize large language models (LLMs) to INT4, and serve them efficiently through an OpenAI-compatible API.
+who_is_this_for: This learning path is designed for software developers and AI engineers who want to build and optimize vLLM for Arm-based servers, quantize large language models (LLMs) to INT4, serve them efficiently through an OpenAI-compatible API, and benchmark model accuracy using the LM Evaluation Harness.
 
 learning_objectives:
     - Build an optimized vLLM for aarch64 with oneDNN and the Arm Compute Library(ACL).
     - Set up all runtime dependencies including PyTorch, llmcompressor, and Arm-optimized libraries.
     - Quantize an LLM (DeepSeek‑V2‑Lite) to 4-bit integer (INT4) precision.
     - Run and serve both quantized and BF16 (non-quantized) variants using vLLM.
     - Use OpenAI‑compatible endpoints and understand sequence and batch limits.
+    - Evaluate accuracy using the LM Evaluation Harness on BF16 and INT4 models with vLLM.
 
 prerequisites:
     - An Arm-based Linux server (Ubuntu 22.04+ recommended) with a minimum of 32 vCPUs, 64 GB RAM, and 64 GB free disk space. 
@@ -32,6 +33,7 @@ operatingsystems:
     - Linux
 tools_software_languages:
     - vLLM
+    - LM Evaluation Harness
     - LLM
     - Generative AI
     - Python
@@ -54,6 +56,10 @@ further_reading:
         title: Build and Run vLLM on Arm Servers
         link: /learning-paths/servers-and-cloud-computing/vllm/
         type: website
+    - resource:
+        title: LM Evaluation Harness (GitHub)
+        link: https://github.com/EleutherAI/lm-evaluation-harness
+        type: github