In Image generation allow clip offload to particular GPU or CPU

[Thoron77]

Hi,

Thank you so much for all the work you do and may this New Year be much better than the last one :)

I was wondering if it would be possible to allow picking particular GPU for Clip in Image Generation instead of just offloading to cpu.
I have dedicated computer just for Image generation with two GPUs, if it would be possible I could use bigger quants for both Diffusion and Clip and gain a little bit of speed. Could be a dropdown like with text generation CPU, GPU1, GPU2, GPU3, GPU4 - no All of course ;) (Maybe one day ;) )

P.S.
Not related but I am a huge fan of auto fit in new Kobold, I have 4 GPUs for Text Generation and they have different sizes of vRAM and it was a pain to manually find optimal fit using just the Tensor split option and eventually regex CPU offload, especially with big models that barely fit. Now it is so easy and so fast. Thank you so much for adding that option. <3

[wbruna]

I was wondering if it would be possible to allow picking particular GPU for Clip in Image Generation instead of just offloading to cpu.

That's currently not supported by the stable-diffusion.cpp backend, unfortunately.

If you have enough system RAM, "Model CPU Offload" could be an alternative: it keeps the models loaded in RAM, moving to VRAM only the active models on-demand; so peak VRAM usage gets as low as the largest model.

[Thoron77]

Thank you for your reply, I've added an "Idea" on their GIT more as a feature request.

[Edit] And yes this is what I do, run Clip on CPU if I run out of vRAM.