[New Model]: bitdance

[hsliuustc0106]

The model to consider.

https://github.com/shallowdream204/BitDance

The closest model vllm-omni already supports.

qwen-image

What's your difficulty of supporting the model you want?

No response

Use case and motivation

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[saileshd1402]

Hi @hsliuustc0106, if no one is working on this issue, can I give it a shot? TIA!

[saileshd1402]

Hi @hsliuustc0106,
I was reading up on the vllm-omni documentation and the implementation of this model and I had a doubt.

Should I implement BitDance entirely as a single-stage DiffusionPipeline following the Adding a Diffusion Model guide?

I initially thought it could be implemented as a multi-stage model since it has a AR and Diffusion phase, but it seems like this model has an interleaved AR-Diffusion loop (LLM generates a patch -> Diffusion head processes it -> latents feed back into the LLM. this repeats multiple times in one generate). From what I understand, since we support only DAG phase structures, we can't support this using multi-stage. Is my understanding correct?

Also one more doubt: Should we support both the original and the diffusers-ported versions of the models?

[hsliuustc0106]

Hi @hsliuustc0106, I was reading up on the vllm-omni documentation and the implementation of this model and I had a doubt.

Should I implement BitDance entirely as a single-stage DiffusionPipeline following the Adding a Diffusion Model guide?

I initially thought it could be implemented as a multi-stage model since it has a AR and Diffusion phase, but it seems like this model has an interleaved AR-Diffusion loop (LLM generates a patch -> Diffusion head processes it -> latents feed back into the LLM. this repeats multiple times in one generate). From what I understand, since we support only DAG phase structures, we can't support this using multi-stage. Is my understanding correct?

Also one more doubt: Should we support both the original and the diffusers-ported versions of the models?

@princepride @ZJY0516 WDYT

[princepride]

I can really feel the architecture was practically made for multi-stage deployment, especially given how exceptionally small the DiT component is.

However, if we can't establish a cyclic stage pipeline, we won't be able to utilize this design efficiently. At a bare minimum, we'd need to implement a feedback loop for the transmission: routing the hidden states from the AR stage to the DiT, and then feeding the resulting tokens from the DiT back to the AR.

Also one more doubt: Should we support both the original and the diffusers-ported versions of the models?

I checked their code, I think their implementations are essentially the same. Correct me if I was wrong.

[saileshd1402]

@princepride, I see makes sense. Just for clarity, since cyclic stage pipelines are not possible currently, I should implement it as a single stage pipeline but implement the feedback loop within it?

I checked their code, I think their implementations are essentially the same. Correct me if I was wrong.

yes, looks like the implementations are essentially the same, I'll make sure that both are supported in my PR

[princepride]

@princepride, I see makes sense. Just for clarity, since cyclic stage pipelines are not possible currently, I should implement it as a single stage pipeline but implement the feedback loop within it?

I checked their code, I think their implementations are essentially the same. Correct me if I was wrong.

yes, looks like the implementations are essentially the same, I'll make sure that both are supported in my PR

Yes, I believe you can first implement it as a single stage pipeline, @ZJY0516 what do you think?