[RFC]: Support Prefill-Decode Disaggregation for vLLM-Omni Thinker Stage via vLLM KV Transfer

[natureofnature]

Motivation.

[RFC]: Support Prefill-Decode Disaggregation for vLLM-Omni Thinker Stage via vLLM KV Transfer

Proposed Change.

Please provide the detailed design document of the RFC using the template.

Feedback Period.

No response

CC List.

No response

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ahengljh]

Jinheng Li

[spencerr221]

Bingyu Liu

[ahengljh]

Feb 5

1. Still investigating 503 error when send requests to connector service.
2. Draft PR updated
3. Mooncake connector design document written

Feb 6

1. 503 Error mitigated,
2. and now trying to solve random decoding results

[spencerr221]

Feb 6
Fixed 503 Network Error testing vLLM KVTransfer on 82 in yellow zone.

[spencerr221]

The proposal has been reviewed and approved. Meeting minutes are as follows:

1. Avoid redundant configuration of engine_id: "omni-thinker-prefill" and is_prefill_only: true in the YAML config.
2. Relocate the configuration to engine_args.
3. Add stage id in generation of transfer_id to verify different stage PD pair

[ahengljh]

Feb 12,

1. Trying to align vllm-omni & vllm to investigate random decoding results
2. Validate omni behavior based on Bingyu's progress
   PR to omni: [Feature][WIP] Support Prefill-Decode disaggregation via vLLM KV transfer vllm-project/vllm-omni#1303

[spencerr221]

Fix the transfer_error on 82 server in yellow zone

[spencerr221]

Due to testing environment issues with the dependent library (vLLM), the completion of this PR will be deferred to the next iteration cycle.

[spencerr221]

Feb 13,
Resolved the garbled text output issue and successfully debugged the Qwen3-Omni text modality pipeline. The pipeline can now be executed on the 82 server in yellow zone via command instructions.

[spencerr221]

Feb 26,
Currently adapting the codebase for vLLM 0.16.0 compatibility, specifically addressing CUDA cuBLAS library call errors.

[ahengljh]

Feb 26,
Rebase current changes to the latest main, and fix some comments of the PR, still trying to debug latest errors with @spencerr221

[spencerr221]

Mar 2，
It is compatible with vllm version 0.16.0, and end-to-end test cases for modalities other than text are currently being developed.

[ahengljh]

Mar 2,

1. Keep rebasing
2. Solve more comments made during the weekend
3. Add more e2e tests