I'm eager for TensorRT

[aifartist]

About 2-3 days ago there was a reddit post about "Stable Diffusion Accelerated" API which uses TensorRT.
Today I actually got VoltaML working with TensorRT and for a 512x512 image at 25 steps I got:
[I] Running StableDiffusion pipeline
100%|█████████| 25/25 [00:00<00:00, 88.71it/s]
|------------|--------------|
| Module | Latency |
|------------|--------------|
| CLIP | 2.33 ms |
| UNet x 25 | 299.25 ms |
| VAE | 13.31 ms |
|------------|--------------|

[ghost]

What are the VRAM requirements? Does it stay the same?

I saw that they only offer txt2img via TensorRT. Its still a very nice boost but I really want to use it for animation which requires img2img. It should be possible though? I believe inpainting, masks etc. all need to be re-programmed for TensorRT because the model changed a lot

[aifartist]

The pace of work has picked up on the experimental branch.
I only got this working today for the first time and am examining the code.
I'm running the main branch. The experimental branch does have some code pieces with img2img in them so I assume work is happening for that.

I'll connect up with the main volta guy tomorrow when he is up and available.

[BetaDoggo]

You will require at least 8GB of vram to convert the models to the tensorRT format and you must use models compiled with the same compute version as the GPU you're using. Though I haven't been able to try it(I only have 6GB so I can't create the models), from what I've heard it roughly doubles vram requirements for equivelent generations.

[cooperdk]

You will require at least 8GB of vram to convert the models to the tensorRT format and you must use models compiled with the same compute version as the GPU you're using. Though I haven't been able to try it(I only have 6GB so I can't create the models), from what I've heard it roughly doubles vram requirements for equivelent generations.

On the other hand, it increases the generation speed with a factor so high that it would be stupid not to support it... I mean, under half a second for 25 steps (no report on what card it was with)

[aifartist]

On the other hand, it increases the generation speed with a factor so high that it would be stupid not to support it... I mean, under half a second for 25 steps (no report on what card it was with)

Sorry, I should have mentioned it. It is a 4090 on a system with a i9-13900K which also is important to keep up with the 4090.

[78Alpha]

Took ages to get running on windows, needing a precompiled version of TensorRT was a pain. The normal one doesn't work, and building yourself is not a task for everyone. It will want CUDA 11.8 but try to grab the lowest possible version leading to errors.

In the end, for standard 1.5 based models it worked at 24 it/s, which is double from Automatic1111 standard, albeit, at 512x512 using the whole card (RTX 3070).

It absolutely crapped its pants when I threw Anything v3.0 at it. It couldn't run it, or even compile it. The premade version also only runs on 40 series and above.

It's good for generating a bunch of classifications images. The lack of dynamic shapes and high cost in every other area makes it off putting.