Diffusers + Torchao

[Tsukasuke]

I've looked at using TensorRT in the past, but there are a number of issues that has me hesitant atm. While looking for updates I found this interesting and thought I'd share.

Recently stumbled across Diffusers + Torchao
"We provide end-to-end inference and experimental training recipes to use torchao with diffusers in this repo. We demonstrate 53.88% speedup on Flux.1-Dev* and 21% speedup on CogVideoX-5b when comparing compiled quantized models against their standard bf16 counterparts**."

[lllyasviel]

I will take a look next week or so. You can repost again later if I forget this

[Demianvan]

Quantization is the future, and future is now! Finally, the official devs of the major models demosntrates low interest in low budget people/hardware, just think how many LORAs are made with lower than 12GB VRAM and chips below Ampere generation. All of that its only possible thanks this kind of initiatives.

[Tsukasuke]

Bump for repost.

[ivancollider]

This was posted on reddit today

https://www.reddit.com/r/StableDiffusion/comments/1ftex5m/pytorch_native_architecture_optimization_torchao/

I hope it gets implemented, it could increase speed in models like Flux and be faster