Qwen Image generation failure in v.1.101.1

[CorentinWicht]

Describe the Issue
Dear @LostRuins,

Thanks for realeasing V.101.1 and supporting Qwen Image generation models! I wanted to give it a try for it's higher prompt adherence but, though I don't have any error when loading the model + VAE/sdclips that you recommended the generated image is always black (see below).

Additional Information:
Operating System = RHEL 9.6
CPU = AMD Ryzen 9 7950X
GPUs = 2x RTX A600 (48Go VRAM)
KoboldCpp Version = 1.101

Here is the code line that I run (embedded within a systemd service):

/usr/bin/python /opt/koboldcpp-latest/koboldcpp.py /home/frillm/Models/Language/Llama-4-Scout-17B-16E-Instruct-Q6_K-00001-of-00002.gguf 8008 --multiuser 20 --highpriority --usecublas --websearch --tensor_split 41 57 --gpulayers 40 --nommap --preloadstory /opt/settings.json --sdmodel /home/frillm/Models/Images/Qwen/Qwen-Image-Edit-2509-Q4_K_S.gguf --sdvae /home/frillm/Models/Images/Qwen/qwen_image_vae.safetensors --sdclip1 /home/frillm/Models/Images/Qwen/Qwen2.5-VL-7B-Instruct.Q4_K_S.gguf --sdclip2 /home/frillm/Models/Images/Qwen/Qwen2.5-VL-7B-Instruct.mmproj-Q8_0.gguf --whispermodel /home/frillm/Models/Whisper/whisper-base.en-q5_1.bin

Here is what I get in the Kobold main UI:

Same if I use the /sdui endpoint:

Thanks for your help and best wishes,

C.

[LostRuins]

Hmm, what about this model? (it's not the edit model but the original qwen image) https://huggingface.co/QuantStack/Qwen-Image-GGUF/resolve/main/Qwen_Image-Q5_0.gguf

I remember we had precision issues that caused this before (see leejet/stable-diffusion.cpp#851 (comment)) but they should have been fixed.

Another thing you can try is the vulkan backend, if that works it most likely a precision issue.

[linuzel]

I have the same issue on Strix Halo.
It is also present with stable-diffusion.cpp directly.
By using this Q5 qwant, Vulkan backend works (but it's still much slower) while rocm produces black images.

[LostRuins]

@linuzel yes the implementation is taken from stable-diffusion.cpp
if you can repro it on the latest stable-diffusion.cpp build, i suggest you create a new issue there so leejet can take a look.
there have been a lot of recent fixes that I have not yet gotten around to merging.

[CorentinWicht]

Hmm, what about this model? (it's not the edit model but the original qwen image) https://huggingface.co/QuantStack/Qwen-Image-GGUF/resolve/main/Qwen_Image-Q5_0.gguf

I remember we had precision issues that caused this before (see leejet/stable-diffusion.cpp#851 (comment)) but they should have been fixed.

Another thing you can try is the vulkan backend, if that works it most likely a precision issue.

Dear @LostRuins,

thanks a lot for your support, I have tried and I am getting another error:

Do you have any documentation about the "Vulkan backend" ? I am not sure to understand what it is and if I can use it knowing that I have Nvidia GPUs (not AMDs). Would that also impact text generation ?

Best wishes,

C.

[LostRuins]

The error from the UI means the generation didn't succeed, the actual error would be in the terminal log.

Vulkan is simply an alternative to CUDA. It's using a different library to access the GPU. The Vulkan backend supports both Nvidia and AMD GPUs. You can use it simple by replacing --usecuda with --usevulkan (gpu id) (gpu id). However, the GPU device IDs can be different from CUDA, so setting it up on a multi-gpu setup may be a bit of a hassle.

If you really wanna try Vulkan, for a quick sanity test, don't load your Llama scout model first. We will do it single GPU.
Simply do a pure Qwen Image setup in vulkan, example command like:

python koboldcpp.py --usevulkan --sdclamped --sdmodel Qwen_Image-Q5_0.gguf --sdvae qwen_image_vae.safetensors --sdclip1 Qwen2.5-VL-7B-Instruct.Q4_K_S.gguf

this is the absolute minimum for running qwen image. If it fails to load, add --sdoffloadcpu or --sdvaecpu
Once loaded,
Open http://localhost:5001/sdui and enter a simple prompt "cat with a text sign that says fish" on default settings.
See if it works. If it doesn't, check the terminal for any error output If it works, we can slowly add on from there.

[CorentinWicht]

Dear @LostRuins,

I wanted to try the quick sanity test but I first had to compile the latest build (1.101) for Vulkan using make LLAMA_VULKAN=1. Unfortunately it failed with:

g++ -I. -Iggml/include -Iggml/src -Iggml/src/ggml-cpu -Iinclude -Isrc -I./common -I./vendor -I./vendor/stb -I./include -I./include/CL -I./otherarch -I./otherarch/tools -I./otherarch/sdcpp -I./otherarch/ttscpp/include -I./otherarch/ttscpp/src -I./otherarch/sdcpp/thirdparty -I./include/vulkan -O3 -fno-finite-math-only -std=c++17 -fPIC -DLOG_DISABLE_LOGS -D_GNU_SOURCE -DGGML_USE_CPU -DGGML_USE_CPU_REPACK -DNDEBUG -s -DGGML_USE_LLAMAFILE -pthread -Wno-multichar -Wno-write-strings -Wno-deprecated -Wno-deprecated-declarations -Wno-unused-variable -pthread  -DGGML_USE_VULKAN -DSD_USE_VULKAN -c ggml/src/ggml-vulkan/ggml-vulkan.cpp -o ggml-vulkan.o
g++ -I. -Iggml/include -Iggml/src -Iggml/src/ggml-cpu -Iinclude -Isrc -I./common -I./vendor -I./vendor/stb -I./include -I./include/CL -I./otherarch -I./otherarch/tools -I./otherarch/sdcpp -I./otherarch/ttscpp/include -I./otherarch/ttscpp/src -I./otherarch/sdcpp/thirdparty -I./include/vulkan -O3 -fno-finite-math-only -std=c++17 -fPIC -DLOG_DISABLE_LOGS -D_GNU_SOURCE -DGGML_USE_CPU -DGGML_USE_CPU_REPACK -DNDEBUG -s -DGGML_USE_LLAMAFILE -pthread -Wno-multichar -Wno-write-strings -Wno-deprecated -Wno-deprecated-declarations -Wno-unused-variable -pthread  -DGGML_USE_VULKAN -DSD_USE_VULKAN -c ggml/src/ggml-vulkan-shaders.cpp -o ggml-vulkan-shaders.o
g++ -I. -Iggml/include -Iggml/src -Iggml/src/ggml-cpu -Iinclude -Isrc -I./common -I./vendor -I./vendor/stb -I./include -I./include/CL -I./otherarch -I./otherarch/tools -I./otherarch/sdcpp -I./otherarch/ttscpp/include -I./otherarch/ttscpp/src -I./otherarch/sdcpp/thirdparty -I./include/vulkan -O3 -fno-finite-math-only -std=c++17 -fPIC -DLOG_DISABLE_LOGS -D_GNU_SOURCE -DGGML_USE_CPU -DGGML_USE_CPU_REPACK -DNDEBUG -s -DGGML_USE_LLAMAFILE -pthread -Wno-multichar -Wno-write-strings -Wno-deprecated -Wno-deprecated-declarations -Wno-unused-variable -pthread -DGGML_USE_VULKAN -DSD_USE_VULKAN -c otherarch/sdcpp/sdtype_adapter.cpp -o sdcpp_vulkan.o
g++ -I. -Iggml/include -Iggml/src -Iggml/src/ggml-cpu -Iinclude -Isrc -I./common -I./vendor -I./vendor/stb -I./include -I./include/CL -I./otherarch -I./otherarch/tools -I./otherarch/sdcpp -I./otherarch/ttscpp/include -I./otherarch/ttscpp/src -I./otherarch/sdcpp/thirdparty -I./include/vulkan -O3 -fno-finite-math-only -std=c++17 -fPIC -DLOG_DISABLE_LOGS -D_GNU_SOURCE -DGGML_USE_CPU -DGGML_USE_CPU_REPACK -DNDEBUG -s -DGGML_USE_LLAMAFILE -pthread -Wno-multichar -Wno-write-strings -Wno-deprecated -Wno-deprecated-declarations -Wno-unused-variable -pthread -DGGML_USE_VULKAN -DSD_USE_VULKAN -c tools/mtmd/clip.cpp -o llavaclip_vulkan.o
g++  -I. -Iggml/include -Iggml/src -Iggml/src/ggml-cpu -Iinclude -Isrc -I./common -I./vendor -I./vendor/stb -I./include -I./include/CL -I./otherarch -I./otherarch/tools -I./otherarch/sdcpp -I./otherarch/ttscpp/include -I./otherarch/ttscpp/src -I./otherarch/sdcpp/thirdparty -I./include/vulkan -O3 -fno-finite-math-only -std=c++17 -fPIC -DLOG_DISABLE_LOGS -D_GNU_SOURCE -DGGML_USE_CPU -DGGML_USE_CPU_REPACK -DNDEBUG -s -DGGML_USE_LLAMAFILE -pthread -Wno-multichar -Wno-write-strings -Wno-deprecated -Wno-deprecated-declarations -Wno-unused-variable -pthread -DGGML_USE_VULKAN -DSD_USE_VULKAN -c ggml/src/ggml-backend.cpp -o ggml-backend_vulkan.o
g++  -I. -Iggml/include -Iggml/src -Iggml/src/ggml-cpu -Iinclude -Isrc -I./common -I./vendor -I./vendor/stb -I./include -I./include/CL -I./otherarch -I./otherarch/tools -I./otherarch/sdcpp -I./otherarch/ttscpp/include -I./otherarch/ttscpp/src -I./otherarch/sdcpp/thirdparty -I./include/vulkan -O3 -fno-finite-math-only -std=c++17 -fPIC -DLOG_DISABLE_LOGS -D_GNU_SOURCE -DGGML_USE_CPU -DGGML_USE_CPU_REPACK -DNDEBUG -s -DGGML_USE_LLAMAFILE -pthread -Wno-multichar -Wno-write-strings -Wno-deprecated -Wno-deprecated-declarations -Wno-unused-variable -pthread -DGGML_USE_VULKAN -DSD_USE_VULKAN -c ggml/src/ggml-backend-reg.cpp -o ggml-backend-reg_vulkan.o
g++ -I. -Iggml/include -Iggml/src -Iggml/src/ggml-cpu -Iinclude -Isrc -I./common -I./vendor -I./vendor/stb -I./include -I./include/CL -I./otherarch -I./otherarch/tools -I./otherarch/sdcpp -I./otherarch/ttscpp/include -I./otherarch/ttscpp/src -I./otherarch/sdcpp/thirdparty -I./include/vulkan -O3 -fno-finite-math-only -std=c++17 -fPIC -DLOG_DISABLE_LOGS -D_GNU_SOURCE -DGGML_USE_CPU -DGGML_USE_CPU_REPACK -DNDEBUG -s -DGGML_USE_LLAMAFILE -pthread -Wno-multichar -Wno-write-strings -Wno-deprecated -Wno-deprecated-declarations -Wno-unused-variable -pthread ggml_v4_vulkan.o ggml-cpu.o ggml-ops.o ggml-vec.o ggml-binops.o ggml-unops.o ggml_v3.o ggml_v2.o ggml_v1.o expose.o gpttype_adapter_vulkan.o ggml-vulkan.o ggml-vulkan-shaders.o sdcpp_vulkan.o whispercpp_default.o tts_default.o embeddings_default.o llavaclip_vulkan.o llava.o ggml-backend_vulkan.o ggml-backend-reg_vulkan.o ggml-repack.o ggml-alloc.o ggml-cpu-traits.o ggml-quants.o ggml-cpu-quants.o kcpp-quantmapper.o kcpp-repackmapper.o unicode.o unicode-data.o ggml-threading.o ggml-cpu-cpp.o gguf.o sgemm.o common.o sampling.o kcpputils.o mtmdaudio.o -lvulkan -shared -o koboldcpp_vulkan.so -ldl
/usr/bin/ld: cannot find -lvulkan
collect2: error: ld returned 1 exit status
make: *** [Makefile:946: koboldcpp_vulkan] Error 1

I am not too familiar with Vulkan but I may have forgotten to install something ?

I can't seem to find aynthing related to how to install Vulkan/fix this issue specifically for RHEL9...

Best,

C.

[CorentinWicht]

My bad, I found out how to install Vulkan and could compile it properly:
sudo dnf install vulkan-loader vulkan-tools vulkan-validation-layers vulkan-headers vulkan-loader-devel

I have adapted your quick sanity test to our setting and it worked with vulkan:

I have thus adapted by the code line that we run inside a systemd service and it worked with Vulkan:

Are there any downside of using VULKAN instead of CUDA knowing that we have two RTX A600 (48Go VRAM) GPUs ?

Best,

C.

[CorentinWicht]

Hmm, what about this model? (it's not the edit model but the original qwen image) https://huggingface.co/QuantStack/Qwen-Image-GGUF/resolve/main/Qwen_Image-Q5_0.gguf
I remember we had precision issues that caused this before (see leejet/stable-diffusion.cpp#851 (comment)) but they should have been fixed.
Another thing you can try is the vulkan backend, if that works it most likely a precision issue.

Dear @LostRuins,

thanks a lot for your support, I have tried and I am getting another error:

Do you have any documentation about the "Vulkan backend" ? I am not sure to understand what it is and if I can use it knowing that I have Nvidia GPUs (not AMDs). Would that also impact text generation ?

Best wishes,

C.

I now tried to move from the Qwen_Image-Q5_0.gguf to the Qwen-Image-Edit-2509-Q4_K_S.gguf model and got the same error as reported before (but it works well when using the \sdui endpoint):

[LostRuins]

Are there any downside of using VULKAN instead of CUDA

It is slightly slower, but should still perform relatively well. I am guessing the CUDA calculations have some precision issues for this model that is causing your black image.

I now tried to move from the Qwen_Image-Q5_0.gguf to the Qwen-Image-Edit-2509-Q4_K_S.gguf model and got the same error as reported before (but it works well when using the \sdui endpoint):

It's strange that it works in /sdui but fails in Kobold Lite. When it fails, can you see what the error is on the terminal? You can also compare the input in the terminal, see whats the difference between both requests.

[CorentinWicht]

Are there any downside of using VULKAN instead of CUDA

It is slightly slower, but should still perform relatively well. I am guessing the CUDA calculations have some precision issues for this model that is causing your black image.

I now tried to move from the Qwen_Image-Q5_0.gguf to the Qwen-Image-Edit-2509-Q4_K_S.gguf model and got the same error as reported before (but it works well when using the \sdui endpoint):

It's strange that it works in /sdui but fails in Kobold Lite. When it fails, can you see what the error is on the terminal? You can also compare the input in the terminal, see whats the difference between both requests.

Dear @LostRuins, thanks a lot for your reply.

I guess my issues were related to GPU memory because when I tried to offload less layers (40/49) it worked with the Qwen-Image-Edit-2509-Q4_K_S.gguf model:

Though I must admin that the resulting image is weird and of poor quality.
Any idea why this might be ?

Now I am seeing the same when generated from /sdui:

I think I will stick to CUDA + Flux.1 Schnell as I can usually offload more layers without risking it to crash.

Thanks a lot and best wishes,

C.

[LostRuins]

alright. maybe in future updates you can try again

[CorentinWicht]

alright. maybe in future updates you can try again

Will surely do, thanks for your support !