[CI] Modify some CI test cases to run on L4 environment to reduce H100 resource usage.#1543
[CI] Modify some CI test cases to run on L4 environment to reduce H100 resource usage.#1543hsliuustc0106 merged 10 commits intovllm-project:mainfrom
Conversation
- Updated the nightly test script to handle multiple pytest commands and capture exit statuses. - Changed model from "Qwen/Qwen3-Omni-30B-A3B-Instruct" to "Qwen/Qwen2.5-Omni-7B" in benchmark tests. - Updated stage configuration file for qwen2.5-omni. - Adjusted prompt in the online serving test to specify a word limit for the answer. Signed-off-by: yenuo26 <410167048@qq.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 291364629f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
- Consolidated the Benchmark & Engine Test steps in both test-merge.yml and test-ready.yml. - Changed the agent queue to "gpu_4_queue" and updated the Docker plugin configuration for better resource management. - Removed the deprecated stage configuration file for Qwen3 Omni Thinker. Signed-off-by: yenuo26 <410167048@qq.com>
Signed-off-by: wangyu <53896905+yenuo26@users.noreply.github.com>
Signed-off-by: yenuo26 <410167048@qq.com>
Signed-off-by: yenuo26 <410167048@qq.com>
Signed-off-by: yenuo26 <410167048@qq.com>
- Set mm_processor_cache_gb to 0 in qwen2_5_omni_ci.yaml, qwen2_5_omni_multiconnector.yaml, and qwen2_5_omni.yaml. - Removed skip marker from test_qwen2_5_omni.py to enable the test. Signed-off-by: yenuo26 <410167048@qq.com>
lishunyang12
left a comment
lishunyang12
left a comment
There was a problem hiding this comment.
Left a couple comments. The H100 -> L4 migration itself makes sense, but a few things need attention.
| engine_output_type: latent | ||
| enable_prefix_caching: false | ||
| max_num_batched_tokens: 32768 | ||
| mm_processor_cache_gb: 0 |
There was a problem hiding this comment.
Please see #1534 for the reason of the change.
There was a problem hiding this comment.
I saw #1534, makes sense for the CI config. But this same change is also added to the production stage configs (qwen2_5_omni.yaml and qwen2_5_omni_multiconnector.yaml) — disables the mm processor cache for all users, not just CI. Was that intentional? If it is only needed to work around an L4 memory constraint, keep it in the CI configs only.
There was a problem hiding this comment.
we make accuracy higher priority
| engine_output_type: latent | ||
| enable_prefix_caching: false | ||
| max_num_batched_tokens: 32768 | ||
| mm_processor_cache_gb: 0 |
There was a problem hiding this comment.
Please see #1534 for the reason of the change.
|
|
||
| - label: "Benchmark & Engine Test with H100" | ||
| timeout_in_minutes: 15 | ||
| - label: "Benchmark & Engine Test" |
There was a problem hiding this comment.
The old config had timeout_in_minutes: 15 at the Buildkite level. The inner timeout 15m only kills the bash process — if the Docker pull or container startup hangs, Buildkite will wait forever. Add timeout_in_minutes back.
|
|
||
|
|
||
| @pytest.mark.skip(reason="There is a known issue with stream error.") | ||
| @pytest.mark.advanced_model |
There was a problem hiding this comment.
Which fix resolved the stream error? Worth adding a comment or linking the PR in the commit message so this does not get re-skipped later.
|
|
||
| models = ["Qwen/Qwen3-Omni-30B-A3B-Instruct"] | ||
| stage_configs = [str(Path(__file__).parent.parent / "e2e" / "stage_configs" / "qwen3_omni_ci.yaml")] | ||
| models = ["Qwen/Qwen2.5-Omni-7B"] |
There was a problem hiding this comment.
Switching from Qwen3-30B to Qwen2.5-7B means benchmark numbers are no longer comparable across runs. If this test is meant to track perf regressions over time, consider keeping a Qwen3 benchmark on H100 (even if less frequent) alongside this L4 one.
| engine_output_type: latent | ||
| enable_prefix_caching: false | ||
| max_num_batched_tokens: 32768 | ||
| mm_processor_cache_gb: 0 |
There was a problem hiding this comment.
we make accuracy higher priority
4 participants
PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.
Purpose
Modify some CI test cases to run on L4 environment to reduce H100 resource usage.
Test Plan
/workspace/.venv/bin/python -m pytest -sv tests/benchmarks/test_serve_cli.py tests/engine/test_async_omni_engine_abort.py --html=report.html --self-contained-html2.test qwen2.5 example testcase
run in ci
Test Result
2.test qwen2.5 example testcase
Essential Elements of an Effective PR Description Checklist
--- - [ ] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)". - [ ] The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the [test style doc](https://docs.vllm.ai/projects/vllm-omni/en/latest/contributing/ci/tests_style/) - [ ] The test results. Please paste the results comparison before and after, or the e2e results. - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model. **Please run `mkdocs serve` to sync the documentation editions to `./docs`.** - [ ] (Optional) Release notes update. If your change is user-facing, please update the release notes draft.BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)