[Test] L2 & L3 Test Case Stratification Design for Omni Model by yenuo26 · Pull Request #1272 · vllm-project/vllm-omni

yenuo26 · 2026-02-09T02:57:51Z

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

L2 & L3 Test Case Stratification Design for Omni Model: refer to #1218
Related documentation can be found：#1167

The main changes are as follows:
1.Added test-merge.yaml to manage merge-level test suites in the future.
2.Standardized the existing Omni3 online use cases and integrated both L2 and L3 level execution logic into a single script, differentiated during execution via the --run-level parameter.
3.Removed the default configuration test scenarios from the original online use cases, retaining only the async_chunk scenario. Default configurations will be covered by offline use cases.
4.Removed the test_build_and_log_summary test case (a new UT case covering this logic will be submitted in #891) and migrated test_async_omni.py to the tests/engine directory.

Test Plan

1.run offline case
/workspace/.venv/bin/python -m pytest -sv tests/e2e/offline_inference/test_qwen3_omni.py --html=report.html --self-contained-html
2.run online case
L2
/workspace/.venv/bin/python -m pytest -sv tests/e2e/online_serving/test_qwen3_omni.py -m core_model --run-level="core_model" --html=report.html --self-contained-html
/workspace/.venv/bin/python -m pytest -sv tests/e2e/online_serving/test_qwen3_omni.py --html=report.html --self-contained-html
L3
/workspace/.venv/bin/python -m pytest -sv tests/e2e/online_serving/test_qwen3_omni.py -m advanced_model --run-level="advanced_model" --html=report.html --self-contained-html
3.run abort case
/workspace/vllm-omni# /workspace/.venv/bin/python -m pytest -sv tests/engine/test_abort.py --html=report.html --self-contained-html

Test Result

1.offline case

2.online case
L2

L3

3.abort case

CI Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

Signed-off-by: wangyu31577 <wangyu31577@hundsun.com>

…ci-qwen3

Signed-off-by: wangyu31577 <wangyu31577@hundsun.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 66a1f1ef67

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

tests/conftest.py

hsliuustc0106 · 2026-02-09T03:15:08Z

.buildkite/test-merge.yml

+                  path: /mnt/hf-cache
+                  type: DirectoryOrCreate
+
+  # - label: "Bagel Text2Img Model Test with H100"


will this be included in PR-merge?

Yes, I will contact the use case author to see how this use case can be split.

hsliuustc0106 · 2026-02-09T03:15:41Z

tests/e2e/offline_inference/test_qwen2_5_omni.py

 from vllm.envs import VLLM_USE_MODELSCOPE
 from vllm.multimodal.image import convert_image_mode

+from tests.conftest import OmniRunner


why move it here?

Because the original conftest only had the vllmrunner class, it seemed unnecessary to keep it as a separate file. Moreover, after merging it into the unified conftest, the functions for validating online use cases have been reused.

hsliuustc0106 · 2026-02-09T03:18:08Z

tests/engine/test_async_omni_engine_abort.py

@@ -8,13 +8,13 @@
 from vllm import SamplingParams


change file name to test_async_omni_engine_abort

already rename

Signed-off-by: yenuo26 <410167048@qq.com>

tjtanaa · 2026-02-09T06:30:55Z

.buildkite/pipeline.yml

+       - export VLLM_TEST_CLEAN_GPU_MEMORY="1"
+       - pytest -s -v tests/e2e/offline_inference/test_qwen3_omni.py
+       - pytest -s -v tests/e2e/online_serving/test_qwen3_omni.py -m "core_model" --run-level "core_model"
+       - pytest -s -v tests/engine/test_abort.py


This line needs to change as well?

test_abort.py to test_async_omni_engine_abort.py

yes, you're right, It has been modified.

…ons to use the new async engine abort test. Signed-off-by: yenuo26 <410167048@qq.com>

yenuo26 · 2026-02-09T10:24:11Z

@hsliuustc0106 @david6666666 please help add ready label

Signed-off-by: yenuo26 <410167048@qq.com>

…ci-qwen3 Signed-off-by: yenuo26 <410167048@qq.com>

hsliuustc0106 · 2026-02-09T11:44:21Z

fix precommit & resolve conflicts

Signed-off-by: yenuo26 <410167048@qq.com>

…ci-qwen3

Signed-off-by: yenuo26 <410167048@qq.com>

yenuo26 · 2026-02-09T12:46:38Z

fix precommit & resolve conflicts

fixed

…fline inference tests. Updated pytest command to focus on specific tests and removed unnecessary import statements. Signed-off-by: yenuo26 <410167048@qq.com>

This commit moves the `kill_process_tree` function into the `OmniServer` class as a private method `_kill_process_tree`, enhancing encapsulation. The method is now used in the context manager exit to ensure proper cleanup of resources. The previous standalone function has been removed to streamline the code. Signed-off-by: yenuo26 <410167048@qq.com>

Signed-off-by: yenuo26 <410167048@qq.com>

This commit adds a new private method `_cleanup_process` to the `OmniRunner` class, which iterates through running processes to terminate any related to "vllm" or "core". This method is called during the context manager exit to ensure proper resource cleanup. Signed-off-by: yenuo26 <410167048@qq.com>

This commit modifies the `_cleanup_process` method in the `OmniRunner` class to remove the "vllm" keyword from the process termination logic, focusing solely on processes related to "core". This change streamlines the cleanup process during context manager exit. Signed-off-by: yenuo26 <410167048@qq.com>

This commit modifies the `_cleanup_process` method in the `OmniRunner` class to change the process keyword from "core" to "enginecore". This adjustment refines the process filtering during cleanup operations. Signed-off-by: yenuo26 <410167048@qq.com>

Signed-off-by: yenuo26 <410167048@qq.com>

Signed-off-by: wangyu <53896905+yenuo26@users.noreply.github.com>

Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>

hsliuustc0106 · 2026-02-11T12:09:58Z

@gcanlin @zhenwei-intel PTAL

gcanlin · 2026-02-11T12:20:59Z

tests/e2e/stage_configs/rocm/qwen2_5_omni_ci.yaml

@@ -13,8 +13,8 @@ stage_args:
      model_arch: Qwen2_5OmniForConditionalGeneration
      worker_type: ar
      scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
-      max_model_len: 896
-      max_num_batched_tokens: 896
+      max_model_len: 2400


Why not also move rocm directory into tests/e2e/stage_configs/? we'd like to create the npu directory here as well.

i will move it

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Signed-off-by: wangyu31577 <wangyu31577@hundsun.com>

…ci-qwen3

hsliuustc0106 · 2026-02-11T14:08:18Z

.buildkite/test-merge.yml

+    agents:
+      queue: "cpu_queue_premerge"
+
+  # - label: "Test on NPU"


rm these comments please

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Signed-off-by: wangyu31577 <wangyu31577@hundsun.com>

congw729

LGTM

hsliuustc0106

lgtm

…roject#1272) Signed-off-by: wangyu31577 <wangyu31577@hundsun.com> Signed-off-by: yenuo26 <410167048@qq.com> Signed-off-by: wangyu <53896905+yenuo26@users.noreply.github.com> Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com> Co-authored-by: wangyu31577 <wangyu31577@hundsun.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>

wangyu31577 and others added 7 commits February 5, 2026 19:50

split case

55b8128

Signed-off-by: wangyu31577 <wangyu31577@hundsun.com>

Merge branch 'vllm-project:main' into ci-qwen3

66851f7

split case

e310649

Signed-off-by: wangyu31577 <wangyu31577@hundsun.com>

Merge branch 'ci-qwen3' of https://github.com/yenuo26/vllm-omni into …

4c16595

…ci-qwen3

split case

f0e9aa2

Signed-off-by: wangyu31577 <wangyu31577@hundsun.com>

add L3 case

5d0f0a4

Signed-off-by: wangyu31577 <wangyu31577@hundsun.com>

Merge branch 'vllm-project:main' into ci-qwen3

66a1f1e

yenuo26 requested a review from hsliuustc0106 as a code owner February 9, 2026 02:57

chatgpt-codex-connector bot reviewed Feb 9, 2026

View reviewed changes

tests/conftest.py Show resolved Hide resolved

hsliuustc0106 reviewed Feb 9, 2026

View reviewed changes

This was referenced Feb 9, 2026

[RFC]: L2 & L3 Test Case Stratification Design for Omni Model JiusiServe/vllm-omni#99

Closed

[RFC]: vllm-omni CI/CD plan #400

Open

fix copilot

4563ebe

Signed-off-by: yenuo26 <410167048@qq.com>

yenuo26 force-pushed the ci-qwen3 branch from 79c6359 to 4563ebe Compare February 9, 2026 04:53

tjtanaa reviewed Feb 9, 2026

View reviewed changes

Update test commands in Buildkite pipeline and test merge configurati…

c47f8ab

…ons to use the new async engine abort test. Signed-off-by: yenuo26 <410167048@qq.com>

yenuo26 mentioned this pull request Feb 9, 2026

[CI] Reduce the time for Diffusion Sequence Parallelism Test #1283

Merged

5 tasks

yenuo26 and others added 3 commits February 9, 2026 19:07

fix conflicts

8551744

Signed-off-by: yenuo26 <410167048@qq.com>

Merge branch 'ci-qwen3' of https://github.com/yenuo26/vllm-omni into …

0464136

…ci-qwen3 Signed-off-by: yenuo26 <410167048@qq.com>

Merge branch 'main' into ci-qwen3

9a53e7f

yenuo26 added 4 commits February 9, 2026 20:31

Merge remote-tracking branch 'upstream/main' into ci-qwen3

433a2d1

Signed-off-by: yenuo26 <410167048@qq.com>

fix conflicts

d4df3cd

Signed-off-by: yenuo26 <410167048@qq.com>

Merge branch 'ci-qwen3' of https://github.com/yenuo26/vllm-omni into …

45462c3

…ci-qwen3

fix conflicts

e34748a

Signed-off-by: yenuo26 <410167048@qq.com>

yenuo26 force-pushed the ci-qwen3 branch from 9312fa3 to e34748a Compare February 9, 2026 12:45

hsliuustc0106 added the ready label to trigger buildkite CI label Feb 9, 2026

Refactor test commands in CI configuration and clean up imports in of…

a52af4f

…fline inference tests. Updated pytest command to focus on specific tests and removed unnecessary import statements. Signed-off-by: yenuo26 <410167048@qq.com>

yenuo26 and others added 10 commits February 10, 2026 20:46

add log

bbcad34

Signed-off-by: yenuo26 <410167048@qq.com>

add log

64f5d07

Signed-off-by: yenuo26 <410167048@qq.com>

add log

83e2a15

Signed-off-by: yenuo26 <410167048@qq.com>

retry CI

a08f2d4

Signed-off-by: yenuo26 <410167048@qq.com>

Merge branch 'main' into ci-qwen3

8c34689

Signed-off-by: wangyu <53896905+yenuo26@users.noreply.github.com>

Merge branch 'main' into ci-qwen3

c29c420

Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>

hsliuustc0106 requested a review from Copilot February 11, 2026 12:09

gcanlin reviewed Feb 11, 2026

View reviewed changes

Copilot AI reviewed Feb 11, 2026

View reviewed changes

wangyu31577 and others added 4 commits February 11, 2026 20:42

move rocm stage configs

b1720ef

Signed-off-by: wangyu31577 <wangyu31577@hundsun.com>

Merge branch 'main' into ci-qwen3

81103ae

retry ci

6aaa415

Signed-off-by: wangyu31577 <wangyu31577@hundsun.com>

Merge branch 'ci-qwen3' of https://github.com/yenuo26/vllm-omni into …

ca072e3

…ci-qwen3

hsliuustc0106 requested a review from Copilot February 11, 2026 14:06

hsliuustc0106 reviewed Feb 11, 2026

View reviewed changes

Copilot AI reviewed Feb 11, 2026

View reviewed changes

retry ci

fc92aad

Signed-off-by: wangyu31577 <wangyu31577@hundsun.com>

Copilot started reviewing on behalf of hsliuustc0106 February 11, 2026 17:46 View session

Merge branch 'main' into ci-qwen3

a3fcb6f

congw729 approved these changes Feb 12, 2026

View reviewed changes

hsliuustc0106 approved these changes Feb 12, 2026

View reviewed changes

hsliuustc0106 merged commit 20a8fe1 into vllm-project:main Feb 12, 2026
7 checks passed

yenuo26 deleted the ci-qwen3 branch February 12, 2026 06:34

amy-why-3459 mentioned this pull request Feb 13, 2026

[RFC]: Qwen3-Omni deployment #409

Open

33 tasks

Conversation

yenuo26 commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

CI Result

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yenuo26 commented Feb 9, 2026

Uh oh!

hsliuustc0106 commented Feb 9, 2026

Uh oh!

yenuo26 commented Feb 9, 2026

Uh oh!

hsliuustc0106 commented Feb 11, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

congw729 left a comment

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

yenuo26 commented Feb 9, 2026 •

edited

Loading