Instruct Pix2Pix

[mykeehu]

I really like this innovation that you can replace almost anything with text without inpaint! It still handles colors too strongly, though, so you'll have to learn a different prompt for this model.

Who's experience? Have you used it?

Here is the project website, what the model can and cannot do:
https://www.timothybrooks.com/instruct-pix2pix/

and here is prompts:
https://instruct-pix2pix.eecs.berkeley.edu/gpt-generated-prompts.jsonl

Instructions from huggingface:

If you're not getting what you want, there may be a few reasons:

1. Is the image not changing enough? Your Image CFG weight may be too high. This value dictates how similar the output should be to the input. It's possible your edit requires larger changes from the original image, and your Image CFG weight isn't allowing that. Alternatively, your Text CFG weight may be too low. This value dictates how much to listen to the text instruction. The default Image CFG of 1.5 and Text CFG of 7.5 are a good starting point, but aren't necessarily optimal for each edit. Try:

• Decreasing the Image CFG weight, or
• Increasing the Text CFG weight, or

2. Conversely, is the image changing too much, such that the details in the original image aren't preserved? Try:

• Increasing the Image CFG weight, or
• Decreasing the Text CFG weight

3. Try generating results with different random seeds by setting "Randomize Seed" and running generation multiple times. You can also try setting "Randomize CFG" to sample new Text CFG and Image CFG values each time.
4. Rephrasing the instruction sometimes improves results (e.g., "turn him into a dog" vs. "make him a dog" vs. "as a dog").
5. Increasing the number of steps sometimes improves results.
6. Do faces look weird? The Stable Diffusion autoencoder has a hard time with faces that are small in the image. Try:

• Cropping the image so the face takes up a larger portion of the frame.

And here is how to create own model or training dataset:
https://github.com/timothybrooks/instruct-pix2pix#training-instructpix2pix

[edwios]

Doesn't work on the M1 Macs. Got an error showing cuda is not supported in model.eval().cuda()

[brkirch]

I have gotten it working and hope to get a PR together to add Mac support to the extension within the next few days or so.

[brkirch]

Support for CPU and MPS (what macOS uses) has been added to Klace's InstructPix2Pix extension.

[swankwc]

Not at all sure what is going on here. I tried to change the shirt color of a woman in a photo. It didn't reference that at all. I am running this on an M1 Mac. Yes, this is quite strange.

File "/Users/wesley/Documents/stable-diffusion-webui/extensions/stable-diffusion-webui-instruct-pix2pix/scripts/instruct-pix2pix.py", line 79, in generate
model.eval().cuda()
File "/Users/wesley/Documents/stable-diffusion-webui/venv/lib/python3.10/site-packages/pytorch_lightning/core/mixins/device_dtype_mixin.py", line 128, in cuda
device = torch.device("cuda", torch.cuda.current_device())
File "/Users/wesley/Documents/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/cuda/init.py", line 482, in current_device
_lazy_init()
File "/Users/wesley/Documents/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/cuda/init.py", line 211, in _lazy_init
raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled
83%|██████████████████████████████████ | 420M/506M [15:11<02:32, 591kB/s
Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
All done.
Total progress: 100%|███████████████████████████| 20/20 [18:51<00:00, 56.60s/it]

[mykeehu]

Soon you can merge any Instruct Pix2Pix model!

[MegaScience]

I attempted to merge a model, and the new model instantly runs out of VRAM. Since the base model worked on my 6GB of VRAM, it seems the base model barely fit and the merge made it too heavy. Though it did have me thinking: Is it trying to keep both the GPT-3 model and Stable Diffusion models active at the same time? Does it need them both active to function, or can the former be run and then the latter? I assume I could turn on VRAM optimizations which would slow me down significantly but allow it to work, but I'm curious if these models could be optimized at all. I suppose I'm inviting anyone with knowledge here on the subject of merges and efficiency.

[mykeehu]

I'll be curious, because I haven't tried it yet, I'm still stuck on Dreambooth, but I'm also interested in how to improve the basic model's structure by mixing

[SergeyPiskov]

Hi,

Getting the error when trying to add pix2pix extension

[w-e-w]

https://github.com/Klace/stable-diffusion-webui-instruct-pix2pix

DEPRECATED: This extension is no longer required. I have integrated the code into Automatic1111 img2img pipeline and the webUI now has Image CFG Scale for instruct-pix2pix models built into the img2img interface. This extension is obsolete.

[SergeyPiskov]

Thank you @w-e-w