Running hires fix separately from the generation process

[kuschanow]

I want to be able to run hires fix separately, simply bypassing the necessary parameters and the image to work with, however, I did not find extensions that would allow this, and I also could not find the entry point to hires fix in the code. If I missed something please let me know. Otherwise, I'd like to ask the developers to consider adding such functionality.

[freecoderwaifu]

Hires fix is basically just a background img2img pass inside the txt2img tab, so img2img is already a separate hires fix.

[freecoderwaifu]

Playing with it a bit more with another seed (Seed: 12121212121212121000), now I'm not entirely sure if it's determinism since newer xformers fixed that and results seem consistent, maybe a slight offset in the denoising in highres fix but also not sure.

Still, the changes are still very minimal, color differences due to the Color Enhance extension aside.

Base image:

Highres fix

Manual img2img of the base image:

[kuschanow]

Thanks a lot. I usually use latent upscale, but this setting with the number of steps is most likely what I need. I'll check this a little later, but I think there was a fairly strong difference (This difference was much greater than can be attributed to ordinary non deterministic) in the results due to this setting.

[skfoo]

If you're using latent, you can't replicate the results of using img2img separately. When doing latent it doesn't use the VAE-decoded pixels in the file it writes as 'before hires fix.' Instead it upscales the original latent data in memory and sends that new latent data to img2img for processing. That's why latent looks special; the latent data is upscaled instead of VAE-decoded pixels.

When using an ESRGAN scaler, it converts the latent data into pixels using the VAE, ESRGAN upscales that by 4x, then it downscales those pixels to the desired scale (say 1.5x or 2x), converts the pixels into latent again using the VAE, and then runs that through img2img. So the 'before hires fix' file is actually what's used for img2img.

[Woisek]

Hires fix runs in latent space, while the img2img deals with already generated pixels. That's how I perceived it since the beginning.

[skfoo]

Well, to be clear, img2img and txt2img do the exact same thing; they take a latent image and denoise it using a prompt. For txt2img, it starts with a latent image of 100% noise and applies 100% denoising, where img2img takes a latent image made by the VAE encoding the pixel image you give it, using whatever denoising % you specify.

So when you use latent upscaling, the latent image output from txt2img is scaled up and then just passed to img2img for more steps with less denoising.

[skfoo]

Did you run the 4x-UltraMix_Restore upscaler before doing img2img? Send txt2img to the extras tab, run the upscaler with the desired output size, then take the result of that to img2img.

[freecoderwaifu]

Can't test right now but yeah, that would account for the smaller changes and another thing to consider to reproduce results.