i'll be away for about a week, will try to look at this after. once you've synced and tested you can unmark as draft first.

Hmm seems like your previous PR conflicts with this?

Yeah, I didn't touch the stb headers with this one, so we'll have at least a small conflict on main. I'll fix that one shortly.

Looks like leejet/stable-diffusion.cpp#484 is about to be included upstream, too! I should at least grab the current PR branch.

Edit: all four fixes got merged 🎉 I'll probably just reconstruct the branches.

Removed local changes due to merged upstream PRs:

• do not force CLIP to F32 (automatic now, since Koboldcpp doesn't enable textual inversions)
• VAE tiling fixes
• do not force VAE to F32 on SDXL
• NULL handling for sd_img_gen_params_t::input_id_images_path
• model loading progress display throttling (now using upstream version)

Also removed:

• memory buffer increments, which got increased upstream: leejet/stable-diffusion.cpp@dc46993

And included:

• forcing T5 to run on CPU (upstream now relies on the clip-on-cpu flag)
  The flag isn't enough for us, because we may not know we need T5 until after the model is loaded (it could be a single 'fat' model file)

master-0ebe6fe changed the Photomaker interface to receive a list of images instead of an image directory, which was exactly what our kcpp_img_gen_params_t did; so we don't need that diff anymore 🎉 I've removed the extra parameter and related processing code from the image generation functions. The only remaining Photomaker-related diff is the code that disables it for non-SDXL models.

i'm back from my trip, so I'll take a look at this over the next few days. kobold is overdue for a release though, so it'll probably have to wait for the next one.

Welcome back!

Yeah, the changes are extensive, and there is at least one still unsolved problematic upstream issue (844). But take a look at the three commits before the sd updates, they may be simple enough to include right now (I can put them in a separate PR if that helps).

I can put them in a separate PR if that helps

Yes please, that would be good

Image generation is kinda broken with the ggml version on the 1.99 release 😕 I'll open an upstream issue shortly.

Edit: leejet/stable-diffusion.cpp#847

Thanks, I will temporarily reverse ggml-org#16056 for a stable patch while this is investigated upstream.

Btw your other PR #1746 caused a merge conflict

@wbruna can you see if 1.99.1 has the prompt adherence work fine? I reverted the commit you listed.

@LostRuins , 1.99.1 seems to be working fine 👍

0cc4m's PR (16151) seems to work, too , but I need to run more tests - edit: test results are good, so I'll go back to the sdtype_adapter refactoring.

merged 16151 for 1.99.2

Will this be updated to support the new qwen model once that PR lands? Thats the main thing our users are waiting for

Will this be updated to support the new qwen model once that PR lands? Thats the main thing our users are waiting for

Since there are a lot of changes, I'd be more comfortable if we gave the update a bit of testing before including changes for specific support. Of course, if Qwen-Image doesn't need any other special parameters or treatment, it may be just a matter of merging in the new revisions.

@LostRuins , I moved the sdtype_adapter pre-update refactoring to #1753. I'll then focus this PR on the parts which need to be changed for the sd.cpp update.

Things that we'll need to consider at some point or another:

• we're still waiting for a fix to SDLX Black output afer PR 55c2e05 leejet/stable-diffusion.cpp#838 . It doesn't actually block this update, though, since we could apply a small diff to force one thread only for model loading
• the keep_clip_on_cpu parameter. It'd be really nice if we could run T5 on the GPU if possible, but that depends on the backend and the model quants. So the sensible approach would be yet another config parameter... Maybe a multi-state like the conv2d one: CLIP on CPU, VAE on CPU, both, neither?
• leejet used the flow-shift on the Qwen Image example, and that'd be another loading-time parameter
• possible new parameters for image generation: scheduler (the new smoothstep scheduler really help with Chroma generation, and I've always missed the ability to change it for specific gens), timestep-shift (the Nitro few-steps models would be really nice for in-chat image generation), guidance, img-cfg-scale...

Marking the PR as 'ready' since I've been testing it for several days.

Updated to master-306-2abe945 , which includes the fixes for leejet/stable-diffusion.cpp#838 and leejet/stable-diffusion.cpp#837 . Just a minor API change, and I don't think there is any major bug left, so it should be ready to go.

Oh, and I forgot about another nice load-time config parameter: offload_params_to_cpu . We need a bigger config screen 😅

cool, is this ready for review? still kinda busy but i'm hoping to get qwen image and wan into the next release.

It's ready for review. I still need to do a full round of testing, to catch anything I've missed, but all the code should be already in place.

As I mentioned before, the diff here is hard to follow, especially for the first updates. But the branches koboldcpp_sd_base and koboldcpp_sd_changes are up to date, so:

• wbruna/stable-diffusion.cpp@master...koboldcpp_sd_base will have all pending PRs (empty right now);
• wbruna/stable-diffusion.cpp@koboldcpp_sd_base...koboldcpp_sd_changes will have all Koboldcpp patches;
• this PR <-> koboldcpp_sd_changes will be empty (updated by script)

The koboldcpp_sd_changes branch builds and runs, so it can be used for testing too.

Tested: SD1.5, SDXL, LoRAs, SD3.5M, img2img, inpaint, Chroma, Flux Schnell, Photomaker, Kontext, TAESD (fixed a bug).

Remind me again - have we always been running the T5 on CPU previously? And what backend issues are there with using it on GPU

Alright did some basic testing and everything seems fine mostly. Will merge shortly

Merged. I'll download and check out WAN sometime, but for now please report any issues found

Remind me again - have we always been running the T5 on CPU previously? And what backend issues are there with using it on GPU

Yes; that behavior was changed as part of the Wan changes (hard to see on the web because of the size of the diff):

leejet/stable-diffusion.cpp@cb1d975#diff-a8e875e8f7684a2f9e35212d88ee9574001952eb2c049385e73d5e24fd6e2a83L323

But it's not always needed: it depends on the T5 quant and the backend. For me, on Vulkan it works for legacy quants and crashes with k-quants (that may have changed in the last versions); ROCm either crashes or makes all-black images. So we should probably add a loading time control to turn on keep_clip_on_cpu...