fix: resolve precision issues in SDXL VAE under fp16

[leejet]

With sdxl_vae-fp16-fix.safetensors

.\bin\Release\sd.exe -m ..\..\stable-diffusion-webui\models\Stable-diffusion\sd_xl_base_1.0.safetensors -p "a lovely cat" -v --vae ..\..\stable-diffusion-webui\models\VAE\sdxl_vae-fp16-fix.safetensors -H 1024 -W 1024

Without sdxl_vae-fp16-fix.safetensors

.\bin\Release\sd.exe -m ..\..\stable-diffusion-webui\models\Stable-diffusion\sd_xl_base_1.0.safetensors -p "a lovely cat" -v -H 1024 -W 1024

[wbruna]

Works well with a few problematic checkpoints here. Thanks!

And after removing the check for external files, it's working even for a few alternative VAEs that always gave me black outputs. Any reason for not enabling the scaling factor for SDXL every time?

[leejet]

The VAE conv scaling may slightly affect image quality and generation speed in some cases, so I prefer to avoid using it whenever possible.
I’ve added a --force-sdxl-vae-conv-scale option to force-enable conv scaling for the SDXL VAE, which should meet your needs.

[wbruna]

And after removing the check for external files, it's working even for a few alternative VAEs that always gave me black outputs.

FWIW, vae-conv-direct also seems to be enough to avoid the issue for the problematic VAEs (although it's far slower on ROCm).

[Green-Sky]

This pr increases vram usage for VAE in all vae-conv-direct(?) cases.

SD1.5 768x1024

vae compute buffer size: 1920.19 MB -> vae compute buffer size: 2112.19 MB

$ result/bin/sd --vae-conv-direct -v --diffusion-fa -m models/CyberRealistic_V9_FP16.safetensors --sampling-method dpm++2m --scheduler karras -W 768 -H 1024 --cfg-scale 5 --steps 38 -n "lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry" -p "a lovely cat"

SD2 512x512

vae compute buffer size: 640.06 MB -> vae compute buffer size: 704.06 MB

$ result/bin/sd -m models/sd_turbo-f16-q8_0.gguf --cfg-scale 1 --steps 8 -p "a lovely cat" --vae-conv-direct -v