[New Extension] Dynamic Thresholding (High CFG Scale Fix)

[mcmonkey4eva]

Repo: https://github.com/mcmonkeyprojects/sd-dynamic-thresholding

Always-Visible extension with a checkbox that when checked enables a way to use higher CFG Scales without color issues.

Example of basic usage:

Example of pushing the settings hard to try to empower a 5 step render, top row is disabled, second row is with the extension on and maxed out on settings (50cfg/50mimic/cosine down/cosine down/100%), third/fourth are experiments to push it even harder

This extension solves the request in issue #3268

[mcmonkey4eva]

cc @ClashSAN

[fractal-fumbler]

why not to place this one in Discussions since extension is released? :)

[Michoko92]

Thank you! .👍👍

[ghost]

I was waiting for this some months since i've read that issue... Thanks man, works quite good <3

[ivangarciafilho]

Yeah, works pretty well, specially when using the high res fix :

8 steps with 8 of CFG Scale, no addon activated ( wrinkles and dramatic shading usually works better with higher CFG scales ):

8 steps 32 of CFG scale, no addon activated (ugly and evil):

8 steps with 32 of CFG to mimmic 8 of CFG through the addon activation:

I've asked for a "creepy and scary jester with rainbow colored hair" ...
Usually it's FUCKING HARD to bring "rainbow colored hair streaks" and "scary facial wrinkles" with such dramatic lighting without a higher amount of steps or a higher CFG scale

[BlueOnyxTeam]

I thought what I needed was new models, but THIS is actually what i've been needing!

There are so many prompts where the "vibe", and camera angle is perfect, but the Detailing is just too flat, or sorta washed out looking.
This makes otherwise 2D images pop out in glorious 3D.

Having the best luck with 99.5-99.9 Constant, and doing CFG values ~5-7 higher.

[FartyPants]

Fantastic. I thought I can learn something from the code - but that's an entirely new level.

[wmooring]

Yes! The inability to more heavily force attention to the entire prompt due to high CFG issues has been one of the things holding back Stable Diffusion since the beginning.

[wmooring]

I do think the default values for this extension should be adjusted to something more useful. A "Top percentile" of 100 seems to generally result in gray washed-out pictures, something like 99.8 or 99.9 would probably be a better starting point.

There's no explanation in the advanced settings of what the difference between "Minimum value of the CFG Scale Scheduler" and "Minimum value of the Mimic Scale Scheduler" are or how they should be used.

Just guessing, I assume than when not set to "Constant", the base and mimicked CFG values used will actually vary for different aspects of the picture generation, and those two values are supposed to define a floor they will not vary below. Which gets a little confusing if you set those values higher than the initial CFG and Mimicked CFG and it still seems to work, but I don't have a lot of info to go off of.

Having seen somewhat disappointing results with the Schedulers set to "Constant", I switched to "Linear Down" for both in testing.

Some settings I've gotten pretty good results with are:

Base CFG: 12
Mimic CFG Scale: 7
Top percentile: 99.975
Mimic Scale Scheduler: Linear Down
Minimum value of the Mimic Scale Scheduler: 5
CFG Scale Scheduler: Linear Down
Minimum value of the CFG Scale Scheduler: 10

But that's prelimary, there are a lot of potential variables. For instance, the above still worked pretty well when I accidentally set the "Mimic CFG Scale" to 12 also.

On a side note, I did confirm that the "Minimum value" sliders have no effect when the Scale Schedulers are set to "Constant"

My hopes for how the future of this extension goes:

1. Integration with "Send to txt2img", so that the parameters shown for pictures in the PNG Info tab or the Image Browser extention will be able to set the values for Dynamic Thresholding.

2. Integration with the X/Y/Z Plot script to allow for easier testing.

3. If enough testing can be done to determine a curve on a graph of the settings centered on where the best-quality results seem to be, an "Auto" setting for Dynamic Thresholding that will automatically pick a good set of parameters based on the requested CFG scale (and number of Steps, maybe?).

[mcmonkey4eva]

Still trying to work out what the best defaults are, or if some form of automated-setting-chooser based on steps/sampler/scale/etc might work better, or presets, or what (myself and JDMLeverton have been going back and forth on this for the past week, testing and trying to find how to do it right).
Several points in your post echo exactly what we've already been discussing.
If you'd like to help us with testing to find the best way to set up defaults (and/or find new tricks to take it even farther, eg the hidden experimental modes), poke me on Discord mcmonkey#6666 and I'll add you into a group with me and JDMLeverton so we can work together on it.

Most of the defaults right now are whatever's closest to "off" - percentile 100 disables percentile thresholding, mode constant disables scale-change-over-time, minimum 0 is equivalent to not setting a minimum at all.

Been mixed results in testing for ideal options thus far. Seems a bit situational.
Seems like both modes to Cosine down and set the minimums to 4 and 3 does some really good work (at least at low step counts), whereas everything 'off' has the fewest unexpected effects while still applying the core idea of compressing CFG scale based on a mimic target.

Minimum value indeed does nothing with 'constant' - if you have for example CFG mode = linear down, and CFG Scale = 30 and minimum = 3, the first step will run at CFG Scale = 30, the next at 29 (or whatever) and etc. until the last step runs at 3.

Top percentile 100% seems to (in my testing thus far) do the best at getting the idea of "just take the current image and enhance it" whereas lower values tend to produce more variation (sometimes helpful, sometimes not).

There's support coded into the infinite grid extension: https://github.com/mcmonkeyprojects/sd-infinity-grid-generator-script#supported-extensions that has been used for testing thus far.
You can view the first grid I generated for testing here: https://sd.mcmonkey.org/dynthresh/
For example if you graph cfg scale vs mimic scale and put percentile on the y super axis, you'll see that lower percentiles + high scales tends to result in the exact same burn the extension is trying to prevent, whereas high percentile prevents the burn but also reduces how much change the image goes through.

Integration with "Send to txt2img", so that the parameters shown for pictures in the PNG Info tab or the Image Browser extention will be able to set the values for Dynamic Thresholding. oh yeah, i'll have to figure out how to hack that in

[fractal-fumbler]

@mcmonkey4eva, hello :) thanks for your work

have you tried lora weights in prompt with dynamic threshold?
i am getting glitched results ^_^

[giteeeeee]

I've tried models trained with dreambooth. Some of them don't tolerant higher cfg values (generate colored noise) comparing to the base model. Just keep the cfg below 12 or something if it happens.
Just speaking from experience, Lora maybe a different story.

[fractal-fumbler]

ye, that can be done, but is this behaviour correct with loras.

i mean why dynthresh overcooking image when loras are used ^_^

[mcmonkey4eva]

I just testran a lora file (one I trained myself) with Scale=40 and mimic=7... came out beautifully. Turn off DT and it's just a purple splotch. Seems to work for me.
Is it possible there's something unstable in the particular Lora file you're using? I've seen some that if you raise them to :1.3 in the prompt or mix two together or etc. they start corrupting (especially lora files trained on the corrupted branch of xformers). DT can't really compensate for that if that's what's getting you.
If it's on publicly posted Lora files (on civitai or huggingface), you could link them and give an example prompt+seed, and I'll try it out myself.

[fractal-fumbler]

@mcmonkey4eva, interesting thing about corrupted branch of xformers. any insights on this issue?

regarding lora files: they aren't trained, but extracted via https://github.com/bmaltais/kohya_ss from custom models.
i'll try with different weights etc.

also here is link for one of extracted one https://drive.google.com/file/d/1JxBxwimIk7GChe5KdKqAOWu7M4G97-aS/view?usp=share_link

extracted from this checkpoint https://civitai.com/models/3452/ultraskin

p.s. discussion #7283
p.p.s. only tried extracted from SD-2.x loras

[Cyberbeing]

@mcmonkey4eva is it a known issue that the extension is working in txt2img, but not with img2img? When I try using it with img2img there is no change to the output image, no matter what settings are selected. Using it with txt2img, no problems at all.

[mcmonkey4eva]

Oops, indeed wasn't working - img2img loads the sampler in differently. I've fixed that now.

[wmooring]

Is it possible to add in the ability to send the Dynamic Thresholding parameters using "Send to txt2img" in the extension itself, or does that require changes to the main package?

[mcmonkey4eva]

Apparently it is possible! Added!

[wmooring]

Great, thanks! It's a little wonky currently because it doesn't turn on the extra UI elements until you uncheck and check "Enable Dynamic Thresholding" again, but the parameters seem to carry over fine.

EDIT: Also, it's able to check Dynamic Thresholding if it's unchecked on the UI, but pictures made without DT don't uncheck it when sending with "Send to txt2img", don't know if that's an issue.

I see there's a new wiki page with recommended settings, I've switched to using those and am getting much better results now (Half Cosine Up for both schedulers, 7 for Mimic CFG, both minimums set to 4). The only odd thing I noticed is that the "Top percentile of latents to clamp" value no longer seems to have any real effect. Is that just a side effect of how that scheduler works? It's a pretty nice change from it being finicky down to the sub-decimal level, I'm not complaining!

[mcmonkey4eva]

Took a bit of research but I found out how to fix both of those issues (UI visible not toggling, and it not going away when loading without enabled).

Clamp only makes a difference when CFG Scale significantly exceeds Mimic Scale. In the case of the Half modes that means it needs to be more than double before it applies.

[wmooring]

Oh also, is the new "Power Scheduler Value" slider needed for anything, or should it usually just be left at 4?

[Cyberbeing]

I believe Power Scheduler Value is only used when you have the new Power Up CFG mode selected.
I've not tested it myself yet, so I'm unsure about viable values.

[mcmonkey4eva]

Yes correct ^.
I added an initial wiki page to start explain the schedulers and options related to them https://github.com/mcmonkeyprojects/sd-dynamic-thresholding/wiki/Schedulers

[fractal-fumbler]

read wiki and not agree about down

getting pretty nice results with Half Cosine down set for CFG Scale Scheduler

Steps: 20, Sampler: DPM++ SDE Karras, CFG scale: 30, Seed: 2282081977, Size: 592x448, Model hash: dcd690123c, Model: SD-21_v2-1_768-ema-pruned, Denoising strength: 0.9, Hires resize: 1024x768, Hires upscaler: Latent, Dynamic thresholding enabled: True, Mimic scale: 7, Threshold percentile: 100, CFG mode: Half Cosine Down, CFG scale minimum: 0, Discard penultimate sigma: True

[mcmonkey4eva]

oooo that looks really good. I'll correct the note about Down schedulers in the wiki.

[fractal-fumbler]

tho need more testing and this is only for Half Cosine down ^_^

[fractal-fumbler]

hello :)
can Dynamic Thresholding extension be used in api scripts?

[mcmonkey4eva]

I'm not familiar entirely with how the webui's API works, but if it lets you select UI scripts and pass webparams, then do that, or if lets you set Processing params, then just set https://github.com/mcmonkeyprojects/sd-dynamic-thresholding/blob/master/scripts/dynamic_thresholding.py#L76-L83

If neither of those exist and it needs some form of special support to be integrated, I'm open to either a PR to add support, or for some links to technical docs I can use to implement it

[mcmonkey4eva]

Relevant issue thread: mcmonkeyprojects/sd-dynamic-thresholding#20

[fractal-fumbler]

they only techy document i could find is https://github.com/mix1009/sdwebuiapi#scripts-support