Serving Video Generation Models (Wan2.2) with Distributed Inference

[jaideepsai-narayan]

Feature request

Description:

I would like to propose adding support for serving video generation models with distributed inference on Gaudi3 AI accelerators. Currently, there is support for serving video generation models using AMD GPUs, as detailed in the AMD blog post
However, support for video model serving on Gaudi3 is lacking. By adding support for Gaudi3, we can significantly improve the performance and scalability of video generation tasks.

Motivation

The demand for high-performance video generation models is growing, and while existing solutions rely heavily on GPUs for inference, Gaudi3 has the potential to dramatically improve both the scalability and efficiency of serving video generation models. Gaudi3's architecture is optimized for distributed workloads and parallel processing, but the current model serving mechanism could benefit from optimizations to ensure the efficient utilization of resources and improved throughput when handling large-scale video generation tasks.

Your contribution

I am happy to contribute to this PR 😊

Current Status: Distributed inference for video generation models on Gaudi3 is already in place, but there is room for improvement in the efficiency of serving and scaling the Wan models.

Any suggestions, insights, or shared articles related to optimizing model serving on Gaudi3, especially for video generation tasks, would be incredibly helpful for improving the serving pipeline and performance. Resources from the community or expert insights would be greatly appreciated 👍

[regisss]

Would the goal be to build a new serving framework focusing on video generation?

[jaideepsai-narayan]

Would the goal be to build a new serving framework focusing on video generation?

Hi @regisss, thank you for the quick response.

No need to build a new framework😀, but are there any existing scripts/frameworks available to serve video models efficiently on Gaudi?
A quick question: is batch processing supported when using the wan2.2-5B diffusers model on Gaudi3? I looked online but wasn’t able to find any information.

[regisss]

are there any existing scripts/frameworks available to serve video models efficiently on Gaudi?

Not that I'm aware of. We do have an example script for performing text-to-video and image-to-video geenration with Wan 2.2 and CogvideoX: https://github.com/huggingface/optimum-habana/tree/main/examples/stable-diffusion#text-to-video-with-wan-22

is batch processing supported when using the wan2.2-5B diffusers model on Gaudi3? I looked online but wasn’t able to find any information.

Looking at the source code here, a batch is created when the model is given a list of prompts as input. I hope that can help!

[jaideepsai-narayan]

@regisss Thank you so much, batch processing is working. I’m now looking for workarounds to serve video online using WAN 2.2,
any suggestions would really help me.

[regisss]

Do you mean "workarounds" related to the pipeline implementation? Or something else?

[jaideepsai-narayan]

Do you mean "workarounds" related to the pipeline implementation? Or something else?

Yes, workarounds related to the pipeline implementation.

[regisss]

Can you describe a bit more what you would like the pipeline to do please?

[jaideepsai-narayan]

Can you describe a bit more what you would like the pipeline to do please?

I don’t have deep knowledge of WAN internals, but from a serving perspective the pipeline should be compatible with async / non-blocking execution, so that it can be safely exposed as an online API and handle concurrent requests efficiently.