A Simple Design for Serving Video Generation Models with Distributed Inference — ROCm Blogs

[giscus[bot]]

A Simple Design for Serving Video Generation Models with Distributed Inference — ROCm Blogs

Minimalist FastAPI + Redis + Torchrun design for serving video generation models with distributed inference.

https://rocm.blogs.amd.com/artificial-intelligence/serving-videogen-v1/README.html

[Explorer-Dong]

Beautiful architecture, but I don't think it's very practical, as it takes a long time to instantiate Wan Pipeline. Perhaps we should separate the process of instantiating objects from the process of reasoning.

[sopiko99]

Hi @Explorer-Dong !

Thanks for reading the blog and for the thoughtful comment.

On the practicality point: in our setup, model initialization and inference are already separated. The Wan pipeline is instantiated once at Torchrun startup, and the workers then enter a loop that continuously pulls jobs from Redis. After initialization completes, all subsequent requests only trigger inference - the model is not reloaded per job.

In our tests, initialization time depends on model size (e.g., <2 minutes for Wan2.1-1.3B, longer for Wan2.2-A14B), but this cost is paid once and amortized across many requests. Steady-state inference latency is what users observe.

If you’re seeing different behavior or longer times, we’d be interested to hear more about your hardware and configuration - happy to compare notes.

Best Regards,
Sopiko and Albin

[Explorer-Dong]

Oh my God I missed this key line:

rocm-blogs/blogs/artificial-intelligence/serving-videogen-v1/src/mountpoint/wan22-generate-t2v.py

Line 481 in 2c5bef9

while True:

Thank you @sopiko99 very much for your timely reply.

I am currently implementing this engineering task that involves both FastAPI and torchrun. This blog has greatly inspired me :>

[sopiko99]

No worries @Explorer-Dong ! We are very happy that the blog has inspired you. Good luck with your work!

[WarrenKelley]

Wow, this blog post on serving video generation models is incredibly insightful! I'm curious about how the distributed inference setup scales with different workloads. Have you run any benchmarks that highlight its efficiency? Thanks for sharing such valuable information! Sprunki

[albitoft-amd]

Hi,

We are happy to hear that you found the blog insightful! 😄 Since the blog was primarily focused on design and less on benchmarks we did not run any extensive benchmarks to measure the scalability on larger workloads. However, based on the architecture it should be possible to scale the solution by adding more workers.