Skip to content

[CI] Reduce Blackwell Fusion test runtime by filtering tests and only run all tests in nightly #28074

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mgoin merged 6 commits into from
Nov 7, 2025
Merged

[CI] Reduce Blackwell Fusion test runtime by filtering tests and only run all tests in nightly #28074

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
9cb815d
Add optional Blackwell Fusion E2E Tests CI group with filtered test c…
Copilot Nov 4, 2025
d789a5c
Address review comments: add both custom ops, use smaller FP4 model, …
Copilot Nov 5, 2025
b1c0671
Fix YAML parsing error by using single quotes in -k filter
Copilot Nov 5, 2025
c937253
Use YAML folded scalar syntax for long pytest command to fix parsing
Copilot Nov 6, 2025
3bd55f7
wrap in quotes
ProExpertProg Nov 6, 2025
24f8274
comments, escape yaml
ProExpertProg Nov 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion .buildkite/test-pipeline.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -471,7 +471,7 @@ steps:
- tests/compile
commands:
- pytest -v -s compile/test_full_graph.py
- pytest -v -s compile/test_fusions_e2e.py
- pytest -v -s compile/test_fusions_e2e.py -k "-quant_fp8"

- label: Cudagraph test
timeout_in_minutes: 20
Expand Down Expand Up @@ -925,6 +925,27 @@ steps:
- pytest -v -s tests/compile/test_silu_mul_quant_fusion.py
# this runner has 2 GPUs available even though num_gpus=2 is not set
- pytest -v -s tests/compile/test_fusion_all_reduce.py
- pytest -v -s tests/compile/test_fusions_e2e.py::test_tp2_attn_quant_allreduce_rmsnorm -k "True and Llama-3.1 and -quant_fp8 and -rms_norm"
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot Nov 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Ensure Blackwell fusion smoke test actually runs target case

The new command filters test_tp2_attn_quant_allreduce_rmsnorm with -k "True and Llama-3.1 and -quant_fp8 and -rms_norm". In a pytest -k expression the leading - is parsed as logical NOT, so this expression matches parameterizations that do not contain quant_fp8 or rms_norm. Because every parameterization of this test includes rms_norm (and the FP8 ones include quant_fp8), the filter deselects every case and the step exits with 0 selected tests. The non-optional pipeline thus never runs any Blackwell fusion end-to-end test, which defeats the purpose of adding the focused coverage.

Useful? React with 👍 / 👎.


- label: Blackwell Fusion E2E Tests # 30 min
timeout_in_minutes: 40
working_dir: "/vllm-workspace/"
gpu: b200
optional: true
num_gpus: 2
source_file_dependencies:
- csrc/quantization/fp4/
- vllm/model_executor/layers/quantization/utils/flashinfer_utils.py
- vllm/v1/attention/backends/flashinfer.py
- vllm/compilation/
# can affect pattern matching
- vllm/model_executor/layers/layernorm.py
- vllm/model_executor/layers/activation.py
- vllm/model_executor/layers/quantization/input_quant_fp8.py
- tests/compile/test_fusions_e2e.py
commands:
- nvidia-smi
# Run all e2e fusion tests
- pytest -v -s tests/compile/test_fusions_e2e.py

- label: Blackwell GPT-OSS Eval
Expand Down
12 changes: 5 additions & 7 deletions tests/compile/test_fusions_e2e.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,11 +54,11 @@ class ModelBackendTestCase(NamedTuple):

MODELS_FP4 = [
ModelBackendTestCase(
model_name="nvidia/Llama-4-Scout-17B-16E-Instruct-FP4",
model_name="nvidia/Llama-3.1-8B-Instruct-FP4",
model_kwargs=dict(max_model_len=1024, kv_cache_dtype="fp8"),
backend=_Backend.FLASHINFER,
attention_fusions=48,
allreduce_fusions=96,
attention_fusions=32,
allreduce_fusions=65,
),
]

Expand Down Expand Up @@ -95,8 +95,7 @@ class ModelBackendTestCase(NamedTuple):
),
]

# TODO(luka) test both in nightly
CUSTOM_OPS_FP8 = ["-quant_fp8"] # , "+quant_fp8"]
CUSTOM_OPS_FP8 = ["-quant_fp8", "+quant_fp8"]


@pytest.mark.parametrize(
Expand Down Expand Up @@ -171,8 +170,7 @@ def test_attn_quant(
assert int(matches[0]) == attention_fusions


# TODO(luka) test both in nightly
CUSTOM_OPS_RMS_NORM = ["-rms_norm"] # , "+rms_norm"]
CUSTOM_OPS_RMS_NORM = ["-rms_norm", "+rms_norm"]


def custom_ops_product(*custom_ops_lists: list[str]) -> Iterable[str]:
Expand Down