Skip to content

Commit 940af1f

Browse files
authored
Add the instruction to run e2e validation manually before release (#21023)
Signed-off-by: Huy Do <[email protected]>
1 parent 92615d7 commit 940af1f

File tree

1 file changed

+33
-0
lines changed

1 file changed

+33
-0
lines changed

RELEASE.md

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,3 +52,36 @@ After branch cut, we approach finalizing the release branch with clear criteria
5252
* Release branch specific changes (e.g. change version identifiers or CI fixes)
5353

5454
Please note: **No feature work allowed for cherry picks**. All PRs that are considered for cherry-picks need to be merged on trunk, the only exception are Release branch specific changes.
55+
56+
## Manual validations
57+
58+
### E2E Performance Validation
59+
60+
Before each release, we perform end-to-end performance validation to ensure no regressions are introduced. This validation uses the [vllm-benchmark workflow](https://github.com/pytorch/pytorch-integration-testing/actions/workflows/vllm-benchmark.yml) on PyTorch CI.
61+
62+
**Current Coverage:**
63+
* Models: Llama3, Llama4, and Mixtral
64+
* Hardware: NVIDIA H100 and AMD MI300x
65+
* *Note: Coverage may change based on new model releases and hardware availability*
66+
67+
**Performance Validation Process:**
68+
69+
**Step 1: Get Access**
70+
Request write access to the [pytorch/pytorch-integration-testing](https://github.com/pytorch/pytorch-integration-testing) repository to run the benchmark workflow.
71+
72+
**Step 2: Review Benchmark Setup**
73+
Familiarize yourself with the benchmark configurations:
74+
* [CUDA setup](https://github.com/pytorch/pytorch-integration-testing/tree/main/vllm-benchmarks/benchmarks/cuda)
75+
* [ROCm setup](https://github.com/pytorch/pytorch-integration-testing/tree/main/vllm-benchmarks/benchmarks/rocm)
76+
77+
**Step 3: Run the Benchmark**
78+
Navigate to the [vllm-benchmark workflow](https://github.com/pytorch/pytorch-integration-testing/actions/workflows/vllm-benchmark.yml) and configure:
79+
* **vLLM branch**: Set to the release branch (e.g., `releases/v0.9.2`)
80+
* **vLLM commit**: Set to the RC commit hash
81+
82+
**Step 4: Review Results**
83+
Once the workflow completes, benchmark results will be available on the [vLLM benchmark dashboard](https://hud.pytorch.org/benchmark/llms?repoName=vllm-project%2Fvllm) under the corresponding branch and commit.
84+
85+
**Step 5: Performance Comparison**
86+
Compare the current results against the previous release to verify no performance regressions have occurred. Here is an
87+
example of [v0.9.1 vs v0.9.2](https://hud.pytorch.org/benchmark/llms?startTime=Thu%2C%2017%20Apr%202025%2021%3A43%3A50%20GMT&stopTime=Wed%2C%2016%20Jul%202025%2021%3A43%3A50%20GMT&granularity=week&lBranch=releases/v0.9.1&lCommit=b6553be1bc75f046b00046a4ad7576364d03c835&rBranch=releases/v0.9.2&rCommit=a5dd03c1ebc5e4f56f3c9d3dc0436e9c582c978f&repoName=vllm-project%2Fvllm&benchmarkName=&modelName=All%20Models&backendName=All%20Backends&modeName=All%20Modes&dtypeName=All%20DType&deviceName=All%20Devices&archName=All%20Platforms).

0 commit comments

Comments
 (0)