cfgpp samplers

[Enferlain]

I was trying to see if I could get them working. I think maybe you could probably just adopt the sampler codes from comfy

like

def sample_euler_ancestral_cfg_pp(model, x, sigmas, extra_args=None, callback=None, disable=None, eta=1., s_noise=1., noise_sampler=None):
    """Ancestral sampling with Euler method steps."""
    extra_args = {} if extra_args is None else extra_args
    noise_sampler = default_noise_sampler(x) if noise_sampler is None else noise_sampler

    temp = [0]
    def post_cfg_function(args):
        temp[0] = args["uncond_denoised"]
        return args["denoised"]

    model_options = extra_args.get("model_options", {}).copy()
    extra_args["model_options"] = comfy.model_patcher.set_model_options_post_cfg_function(model_options, post_cfg_function, disable_cfg1_optimization=True)

    s_in = x.new_ones([x.shape[0]])
    for i in trange(len(sigmas) - 1, disable=disable):
        denoised = model(x, sigmas[i] * s_in, **extra_args)
        sigma_down, sigma_up = get_ancestral_step(sigmas[i], sigmas[i + 1], eta=eta)
        if callback is not None:
            callback({'x': x, 'i': i, 'sigma': sigmas[i], 'sigma_hat': sigmas[i], 'denoised': denoised})
        d = to_d(x, sigmas[i], temp[0])
        # Euler method
        dt = sigma_down - sigmas[i]
        x = denoised + d * sigma_down
        if sigmas[i + 1] > 0:
            x = x + noise_sampler(sigmas[i], sigmas[i + 1]) * s_noise * sigma_up
    return x

Here I tried only changing comfy.model_patcher.set_model_options_post_cfg_function to backend.patcher.base.set_model_options_post_cfg_function but I couldn't find if it's used elsewhere and it complains like this

  File "D:\stable-diffusion-webui-forge\modules\sd_samplers_kdiffusion.py", line 238, in <lambda>
    samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
  File "D:\stable-diffusion-webui-forge\extensions\Euler-Smea-Dyn-Sampler\smea_sampling.py", line 295, in sample_lumi
    denoised = model(x, sigmas[i] * s_in, **extra_args)
  File "D:\stable-diffusion-webui-forge\venv\lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "D:\stable-diffusion-webui-forge\venv\lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
TypeError: CFGDenoiser.forward() got an unexpected keyword argument 'model_options'

And I also tried using the example from ddim_cfgpp but that gives burned-nan results

not really sure how to adopt it to all the different samplers

[Enferlain]

Uhh, I think this is all it takes maybe. Dunno if it's the right thing to do but it doesn't error anymore and temp[0] seems to be filled and the result looks different than any of the other samplers.

in modules.sd_samplers_cfg_denoiser.py

def forward(self, x, sigma, uncond, cond, cond_scale, s_min_uncond, image_cond, model_options=None):
    if state.interrupted or state.skipped:
        raise sd_samplers_common.InterruptedException

    if model_options is None:
        model_options = {}  # Initialize model_options if not provided

would be nice if someone else can check them as well

[Enferlain]

is cond and uncond supposed to have the same values across the board tho

[DenOfEquity]

last_noise_uncond can be used directly, no need to send through to_d(...). Also, the cond_scale_miltiplier isn't used anywhere; could be processed in the forward function in sd_samplers_cfg_denoiser.py.
These are the best I've managed; need lower CFG than standard samplers:

@torch.no_grad()
def sample_euler_cfgpp(model, x, sigmas, extra_args=None, callback=None, disable=None, s_churn=0., s_tmin=0., s_tmax=float('inf'), s_noise=1.):
    """Implements Algorithm 2 (Euler steps) from Karras et al. (2022)."""
    extra_args = {} if extra_args is None else extra_args
    model.need_last_noise_uncond = True
    s_in = x.new_ones([x.shape[0]])

    for i in trange(len(sigmas) - 1, disable=disable):
        gamma = min(s_churn / (len(sigmas) - 1), 2 ** 0.5 - 1) if s_tmin <= sigmas[i] <= s_tmax else 0.
        eps = torch.randn_like(x) * s_noise
        sigma_hat = sigmas[i] * (gamma + 1)
        if gamma > 0:
            x = x + eps * (sigma_hat ** 2 - sigmas[i] ** 2) ** 0.5
        denoised = model(x, sigma_hat * s_in, **extra_args)
        d = model.last_noise_uncond

        if callback is not None:
            callback({'x': x, 'i': i, 'sigma': sigmas[i], 'sigma_hat': sigma_hat, 'denoised': denoised})

        # Euler method
        x = denoised + d * sigmas[i+1]    # using sigma_hat here gives bad results
    return x


@torch.no_grad()
def sample_euler_ancestral_cfgpp(model, x, sigmas, extra_args=None, callback=None, disable=None, eta=1., s_noise=1., noise_sampler=None):
    """Ancestral sampling with Euler method steps."""
    extra_args = {} if extra_args is None else extra_args
    noise_sampler = default_noise_sampler(x) if noise_sampler is None else noise_sampler
    model.need_last_noise_uncond = True
    s_in = x.new_ones([x.shape[0]])

    for i in trange(len(sigmas) - 1, disable=disable):
        denoised = model(x, sigmas[i] * s_in, **extra_args)
        d = model.last_noise_uncond

        sigma_down, sigma_up = get_ancestral_step(sigmas[i], sigmas[i + 1], eta=eta)

        if callback is not None:
            callback({'x': x, 'i': i, 'sigma': sigmas[i], 'sigma_hat': sigmas[i], 'denoised': denoised})

        # Euler method
        x = denoised + d * sigma_down
        if sigmas[i + 1] > 0:
            x = x + noise_sampler(sigmas[i], sigmas[i + 1]) * s_noise * sigma_up
    return x

Anything based off the Comfy method just gave me burnt results, and in the case of Euler the results drifted to the negative prompt.

[Enferlain]

Yeah I'm trying to figure out why the comfy method gives burned results atm, since the code seems to be pretty much the same for that interaction with the backend. I did get it "working" for a bit yesterday like I wrote above, except I couldn't tell if it was actually working because in the prints/debug showed the same set of values for both cond and uncond, going as far as

stable-diffusion-webui-forge/backend/sampling/sampling_function.py

Line 154 in b20cb4b

def calc_cond_uncond_batch(model, cond, uncond, x_in, timestep, model_options):

and someone I asked said it's not supposed to be the same, but I'd need to try on comfy/swarm I guess to compare

[Enferlain]

O̶n̶e̶ ̶t̶h̶i̶n̶g̶ ̶I̶ ̶j̶u̶s̶t̶ ̶n̶o̶t̶i̶c̶e̶d̶ ̶i̶s̶ ̶t̶h̶a̶t̶ ̶s̶e̶t̶_̶m̶o̶d̶e̶l̶_̶o̶p̶t̶i̶o̶n̶s̶_̶p̶o̶s̶t̶_̶c̶f̶g̶_̶f̶u̶n̶c̶t̶i̶o̶n̶ ̶l̶e̶a̶d̶s̶ ̶t̶o̶ ̶a̶ ̶f̶u̶n̶c̶t̶i̶o̶n̶ ̶o̶u̶t̶s̶i̶d̶e̶ ̶o̶f̶ ̶M̶o̶d̶e̶l̶P̶a̶t̶c̶h̶e̶r̶,̶ ̶w̶h̶e̶r̶e̶a̶s̶ ̶i̶n̶ ̶c̶o̶m̶f̶y̶ ̶t̶h̶e̶ ̶s̶a̶m̶e̶ ̶o̶n̶e̶ ̶l̶e̶a̶d̶s̶ ̶t̶o̶ ̶a̶ ̶f̶u̶n̶c̶t̶i̶o̶n̶ ̶i̶n̶s̶i̶d̶e̶ ̶o̶f̶ ̶i̶t̶.̶ malarkey

[Enferlain]

Ah yeah I got around the burned results with directly adding this

        denoised = model(x, sigmas[i] * s_in, **extra_args)
        post_cfg_function({'denoised': denoised, 'uncond_denoised': denoised})

I just can't confirm if it actually does what it's supposed to, other than temp[0] and denoised both having the same values that cond and uncond does

[Enferlain]

nope, o7

[lllyasviel]

well in my opinion those cfgpp things can be better if implemented in a way like a checkbox or what so that all samplers can just use it directly, rather than adding independent samplers.

but I agree that Eular A cfgpp can be somewhat more emergency and maybe one should just have it directly

anyway I will take a look again later probably the next time when I am on forge again and you may post this again later in case I forget it.

also, Forge's sampling is about 80% a1111 + 15% my codes + 5% mixed from comfyui/diffusers/prs from people that I do not know. So usually if you want to add a thing, you can mainly take a look how a1111 handle it. Or you can directly PR a1111 webui and wait upstream to merge and we will also then pull ( but that will be very very slow

If you want to directly merge codes from comfyui to forge it is also okay but you need to make sure that the sampler is not already there, and make sure to use Forge's patching system. Forge's patching system is CFG for eps (unlike CFG for x0 in comfyui) to perfectly reproduce webui results (and a1111 features like "AND" blending, etc). And maybe thats why you get blurred results. If other things are in order then someone will merge it.

[yoinked-h]

the issue with implementing cfgpp on every sampler is that it needs a certain degree of messing around with (see dpmpp 2s ancestral), sure its possible to make some/most work with just a simple toggle, enabling certain code, but for more complex samplers its just not possible.
sidenote: ddim cfgpp is Not Doing CFG++, rather just shrinking cfg ranges from what i can tell
another issue that i can see, from a glanceover, uncond_denoised and denoised seem to return the same when it shouldnt, my best guess is that the fundamental sampling code is flawed

[Enferlain]

seems like there is still x0 code btw

stable-diffusion-webui-forge/modules/sd_samplers_cfg_denoiser.py

Line 74 in 95b54a2

def combine_denoised(self, x_out, conds_list, uncond, cond_scale, timestep, x_in, cond):

[Enferlain]

[DenOfEquity]

I think that function is just a relic, doesn't seem to be used anywhere.
Did you try the samplers I posted? In testing, sometimes I see benefit, other times it is not obvious (as is always the case with this sort of thing). I've added them to my ExtraSchedulers extension for testing.
example 1, left Euler a CFG 5.1; right Euler a CFG++ 2.3:
prompt: photograph, brunette on beach, waving, shipwreck background.
negative: soft focus, blurry, drawing, cartoon, painting, nsfw, tilted horizon

example 2, left Euler CFG 5.1; right Euler CFG++ 5.1 with APG:
prompt: photograph fantasy landscape, tall wizards tower, spire in clouds
negative: blurry, drawing, cartoon, painting

[gruevy]

Any chance you'll be willing to take another look at adding the ++ samplers to Forge?

[BoJangles002]

Is there any update on ++ samplers being added to forge?

[John6666cat]

Implementation of cfgpp in reForge
https://github.com/Panchovix/stable-diffusion-webui-reForge/blob/main/modules%2Fsd_samplers_extra.py#L77
https://github.com/Panchovix/stable-diffusion-webui-reForge/blob/main/ldm_patched%2Fk_diffusion%2Fsampling.py#L2167

[Panchovix]

The Den forge fork supports CFG++ https://github.com/DenOfEquity/ersatzForge