You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: RELEASE.md
+33Lines changed: 33 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -52,3 +52,36 @@ After branch cut, we approach finalizing the release branch with clear criteria
52
52
* Release branch specific changes (e.g. change version identifiers or CI fixes)
53
53
54
54
Please note: **No feature work allowed for cherry picks**. All PRs that are considered for cherry-picks need to be merged on trunk, the only exception are Release branch specific changes.
55
+
56
+
## Manual validations
57
+
58
+
### E2E Performance Validation
59
+
60
+
Before each release, we perform end-to-end performance validation to ensure no regressions are introduced. This validation uses the [vllm-benchmark workflow](https://github.com/pytorch/pytorch-integration-testing/actions/workflows/vllm-benchmark.yml) on PyTorch CI.
61
+
62
+
**Current Coverage:**
63
+
* Models: Llama3, Llama4, and Mixtral
64
+
* Hardware: NVIDIA H100 and AMD MI300x
65
+
**Note: Coverage may change based on new model releases and hardware availability*
66
+
67
+
**Performance Validation Process:**
68
+
69
+
**Step 1: Get Access**
70
+
Request write access to the [pytorch/pytorch-integration-testing](https://github.com/pytorch/pytorch-integration-testing) repository to run the benchmark workflow.
71
+
72
+
**Step 2: Review Benchmark Setup**
73
+
Familiarize yourself with the benchmark configurations:
Navigate to the [vllm-benchmark workflow](https://github.com/pytorch/pytorch-integration-testing/actions/workflows/vllm-benchmark.yml) and configure:
79
+
***vLLM branch**: Set to the release branch (e.g., `releases/v0.9.2`)
80
+
***vLLM commit**: Set to the RC commit hash
81
+
82
+
**Step 4: Review Results**
83
+
Once the workflow completes, benchmark results will be available on the [vLLM benchmark dashboard](https://hud.pytorch.org/benchmark/llms?repoName=vllm-project%2Fvllm) under the corresponding branch and commit.
84
+
85
+
**Step 5: Performance Comparison**
86
+
Compare the current results against the previous release to verify no performance regressions have occurred. Here is an
87
+
example of [v0.9.1 vs v0.9.2](https://hud.pytorch.org/benchmark/llms?startTime=Thu%2C%2017%20Apr%202025%2021%3A43%3A50%20GMT&stopTime=Wed%2C%2016%20Jul%202025%2021%3A43%3A50%20GMT&granularity=week&lBranch=releases/v0.9.1&lCommit=b6553be1bc75f046b00046a4ad7576364d03c835&rBranch=releases/v0.9.2&rCommit=a5dd03c1ebc5e4f56f3c9d3dc0436e9c582c978f&repoName=vllm-project%2Fvllm&benchmarkName=&modelName=All%20Models&backendName=All%20Backends&modeName=All%20Modes&dtypeName=All%20DType&deviceName=All%20Devices&archName=All%20Platforms).
0 commit comments