Merge branch 'development' into main

This merge adds the following major features:

* Support for image variations.

* Security fix for webGUI (binds to localhost by default, use
--host=0.0.0.0 to allow access from external interface.

* Scalable configs/models.yaml configuration file for adding more
models as they become available.

* More tuning and exception handling for M1 hardware running MPS.

* Various documentation fixes.

Author: lstein

README-Mac-MPS.md

@@ -1,37 +1,81 @@
-# Apple Silicon Mac Users
+# macOS Instructions
 
-Several people have gotten Stable Diffusion to work on Apple Silicon
-Macs using Anaconda, miniforge, etc. I've gathered up most of their instructions and
-put them in this fork (and readme). Things have moved really fast and so these
-instructions change often. Hopefully things will settle down a little.
+Requirements
 
-There's several places where people are discussing Apple
-MPS functionality: [the original CompVis
-issue](https://github.com/CompVis/stable-diffusion/issues/25), and generally on
-[lstein's fork](https://github.com/lstein/stable-diffusion/).
+- macOS 12.3 Monterey or later
+- Python
+- Patience
+- Apple Silicon*
 
-You have to have macOS 12.3 Monterey or later. Anything earlier than that won't work.
+*I haven't tested any of this on Intel Macs but I have read that one person got
+it to work, so Apple Silicon might not be requried.
 
-Tested on a 2022 Macbook M2 Air with 10-core GPU and 24 GB unified memory.
+Things have moved really fast and so these instructions change often and are
+often out-of-date. One of the problems is that there are so many different ways to
+run this.
 
-How to:
+We are trying to build a testing setup so that when we make changes it doesn't
+always break.
+
+How to (this hasn't been 100% tested yet):
+
+First get the weights checkpoint download started - it's big:
+
+1. Sign up at https://huggingface.co
+2. Go to the [Stable diffusion diffusion model page](https://huggingface.co/CompVis/stable-diffusion-v-1-4-original)
+3. Accept the terms and click Access Repository: 
+4. Download [sd-v1-4.ckpt (4.27 GB)](https://huggingface.co/CompVis/stable-diffusion-v-1-4-original/blob/main/sd-v1-4.ckpt) and note where you have saved it (probably the Downloads folder)
+
+While that is downloading, open Terminal and run the following commands one at a time.
 
 ```
+# install brew (and Xcode command line tools):
+/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
+
+# install python 3, git, cmake, protobuf:
+brew install cmake protobuf rust
+
+# install miniconda (M1 arm64 version):
+curl https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh -o Miniconda3-latest-MacOSX-arm64.sh
+/bin/bash Miniconda3-latest-MacOSX-arm64.sh
+
+# clone the repo
 git clone https://github.com/lstein/stable-diffusion.git
 cd stable-diffusion
 
+#
+# wait until the checkpoint file has downloaded, then proceed
+#
+
+# create symlink to checkpoint
 mkdir -p models/ldm/stable-diffusion-v1/
-PATH_TO_CKPT="$HOME/Documents/stable-diffusion-v-1-4-original"  # or wherever yours is.
+
+PATH_TO_CKPT="$HOME/Downloads"  # or wherever you saved sd-v1-4.ckpt
+
 ln -s "$PATH_TO_CKPT/sd-v1-4.ckpt" models/ldm/stable-diffusion-v1/model.ckpt
 
-CONDA_SUBDIR=osx-arm64 conda env create -f environment-mac.yaml
+# install packages
+PIP_EXISTS_ACTION=w CONDA_SUBDIR=osx-arm64 conda env create -f environment-mac.yaml
 conda activate ldm
 
+# only need to do this once
 python scripts/preload_models.py
+
+# run SD!
 python scripts/dream.py --full_precision  # half-precision requires autocast and won't work
 ```
 
-After you follow all the instructions and run dream.py you might get several errors. Here's the errors I've seen and found solutions for.
+The original scripts should work as well.
+
+```
+python scripts/orig_scripts/txt2img.py --prompt "a photograph of an astronaut riding a horse" --plms
+```
+
+Note, `export PIP_EXISTS_ACTION=w` is a precaution to fix `conda env create -f environment-mac.yaml`
+never finishing in some situations. So it isn't required but wont hurt.
+
+After you follow all the instructions and run dream.py you might get several
+errors. Here's the errors I've seen and found solutions for.
 
 ### Is it slow?
 
@@ -52,27 +96,37 @@ One debugging step is to update to the latest version of PyTorch nightly.
 
 	conda install pytorch torchvision torchaudio -c pytorch-nightly
 
-Or you can clean everything up.
+If `conda env create -f environment-mac.yaml` takes forever run this.
+
+	git clean -f
+
+And run this.
 
 	conda clean --yes --all
 
-Or you can reset Anaconda.
+Or you could reset Anaconda.
 
 	conda update --force-reinstall -y -n base -c defaults conda
 
-### "No module named cv2" (or some other module)
+### "No module named cv2", torch, 'ldm', 'transformers', 'taming', etc.
+
+There are several causes of these errors.
 
-Did you remember to `conda activate ldm`? If your terminal prompt
+First, did you remember to `conda activate ldm`? If your terminal prompt
 begins with "(ldm)" then you activated it. If it begins with "(base)"
 or something else you haven't.
 
-If it says you're missing taming you need to rebuild your virtual
+Second, you might've run `./scripts/preload_models.py` or `./scripts/dream.py`
+instead of `python ./scripts/preload_models.py` or `python ./scripts/dream.py`.
+The cause of this error is long so it's below.
+
+Third, if it says you're missing taming you need to rebuild your virtual
 environment.
 
 	conda env remove -n ldm
 	conda env create -f environment-mac.yaml
 
-If you have activated the ldm virtual environment and tried rebuilding
+Fourth, If you have activated the ldm virtual environment and tried rebuilding
 it, maybe the problem could be that I have something installed that
 you don't and you'll just need to manually install it. Make sure you
 activate the virtual environment so it installs there instead of
@@ -83,6 +137,56 @@ globally.
 
 You might also need to install Rust (I mention this again below).
 
+### How many snakes are living in your computer?
+
+Here's the reason why you have to specify which python to use.
+There are several versions of python on macOS and the computer is
+picking the wrong one. More specifically, preload_models.py and dream.py says to
+find the first `python3` in the path environment variable. You can see which one
+it is picking with `which python3`. These are the mostly likely paths you'll see.
+
+	% which python3
+	/usr/bin/python3
+
+The above path is part of the OS. However, that path is a stub that asks you if
+you want to install Xcode. If you have Xcode installed already,
+/usr/bin/python3 will execute /Library/Developer/CommandLineTools/usr/bin/python3 or
+/Applications/Xcode.app/Contents/Developer/usr/bin/python3 (depending on which
+Xcode you've selected with `xcode-select`).
+
+	% which python3
+	/opt/homebrew/bin/python3
+
+If you installed python3 with Homebrew and you've modified your path to search
+for Homebrew binaries before system ones, you'll see the above path.
+
+	% which python
+	/opt/anaconda3/bin/python
+
+If you drop the "3" you get an entirely different python. Note: starting in
+macOS 12.3, /usr/bin/python no longer exists (it was python 2 anyway).
+
+If you have Anaconda installed, this is what you'll see. There is a
+/opt/anaconda3/bin/python3 also.
+
+	(ldm) % which python
+	/Users/name/miniforge3/envs/ldm/bin/python
+
+This is what you'll see if you have miniforge and you've correctly activated
+the ldm environment. This is the goal.
+
+It's all a mess and you should know [how to modify the path environment variable](https://support.apple.com/guide/terminal/use-environment-variables-apd382cc5fa-4f58-4449-b20a-41c53c006f8f/mac)
+if you want to fix it. Here's a brief hint of all the ways you can modify it
+(don't really have the time to explain it all here).
+
+- ~/.zshrc
+- ~/.bash_profile
+- ~/.bashrc
+- /etc/paths.d
+- /etc/path
+
+Which one you use will depend on what you have installed except putting a file
+in /etc/paths.d is what I prefer to do.
 
 ### Debugging?
 
@@ -139,7 +243,7 @@ the environment variable `CONDA_SUBDIR=osx-arm64`, like so:
 This error happens with Anaconda on Macs when the Intel-only `mkl` is pulled in by
 a dependency. [nomkl](https://stackoverflow.com/questions/66224879/what-is-the-nomkl-python-package-used-for)
 is a metapackage designed to prevent this, by making it impossible to install
-`mkl`, but if your environment is already broken it may not work. 
+`mkl`, but if your environment is already broken it may not work.
 
 Do *not* use `os.environ['KMP_DUPLICATE_LIB_OK']='True'` or equivalents as this
 masks the underlying issue of using Intel packages.
@@ -207,7 +311,7 @@ change instead. This is a 32-bit vs 16-bit problem.
 What? Intel? On an Apple Silicon?
 
 	Intel MKL FATAL ERROR: This system does not meet the minimum requirements for use of the Intel(R) Math Kernel Library.
-	The processor must support the Intel(R) Supplemental Streaming SIMD Extensions 3 (Intel(R) SSSE3) instructions.██████████████| 50/50 [02:25<00:00,  2.53s/it]
+	The processor must support the Intel(R) Supplemental Streaming SIMD Extensions 3 (Intel(R) SSSE3) instructions.
 	The processor must support the Intel(R) Streaming SIMD Extensions 4.2 (Intel(R) SSE4.2) instructions.
 	The processor must support the Intel(R) Advanced Vector Extensions (Intel(R) AVX) instructions.
 

README.md

@@ -21,17 +21,21 @@ text-to-image generator. This fork supports:
 2. A basic Web interface that allows you to run a local web server for
    generating images in your browser.
 
-3. A notebook for running the code on Google Colab.
-
-4. Support for img2img in which you provide a seed image to guide the
+3. Support for img2img in which you provide a seed image to guide the
       image creation. (inpainting & masking coming soon)
 
+4. A notebook for running the code on Google Colab.
+
 5. Upscaling and face fixing using the optional ESRGAN and GFPGAN
    packages.
 
 6. Weighted subprompts for prompt tuning.
 
-7. Textual inversion for customization of the prompt language and images.
+7. [Image variations](VARIATIONS.md) which allow you to systematically
+generate variations of an image you like and combine two or more
+images together to combine the best features of both.
+
+8. Textual inversion for customization of the prompt language and images.
 
 8. ...and more!
 
@@ -42,8 +46,11 @@ improvements and bug fixes.
 # Table of Contents
 
 1. [Major Features](#features)
-2. [Changelog](#latest)
+2. [Changelog](#latest-changes)
 3. [Installation](#installation)
+   1. [Linux](#linux)
+   1. [Windows](#windows)
+   1. [MacOS](README-Mac-MPS.md)
 4. [Troubleshooting](#troubleshooting)
 5. [Contributing](#contributing)
 6. [Support](#support)
@@ -331,8 +338,10 @@ and introducing a new vocabulary to the fixed model.
 
 To train, prepare a folder that contains images sized at 512x512 and execute the following:
 
+
+WINDOWS: As the default backend is not available on Windows, if you're using that platform, set the environment variable `PL_TORCH_DISTRIBUTED_BACKEND=gloo`
+    
 ```
-# As the default backend is not available on Windows, if you're using that platform, execute SET PL_TORCH_DISTRIBUTED_BACKEND=gloo
 (ldm) ~/stable-diffusion$ python3 ./main.py --base ./configs/stable-diffusion/v1-finetune.yaml \
                                             -t \
                                             --actual_resume ./models/ldm/stable-diffusion-v1/model.ckpt \
@@ -505,6 +514,30 @@ This will bring your local copy into sync with the remote one.
 
 ## Windows
 
+### Notebook install (semi-automated)
+
+We have a
+[Jupyter notebook](https://github.com/lstein/stable-diffusion/blob/main/Stable-Diffusion-local-Windows.ipynb)
+with cell-by-cell installation steps. It will download the code in this repo as
+one of the steps, so instead of cloning this repo, simply download the notebook
+from the link above and load it up in VSCode (with the
+appropriate extensions installed)/Jupyter/JupyterLab and start running the cells one-by-one.
+
+Note that you will need NVIDIA drivers, Python 3.10, and Git installed
+beforehand - simplified
+[step-by-step instructions](https://github.com/lstein/stable-diffusion/wiki/Easy-peasy-Windows-install)
+are available in the wiki (you'll only need steps 1, 2, & 3 ).
+
+### Manual installs
+
+#### pip
+
+See
+[Easy-peasy Windows install](https://github.com/lstein/stable-diffusion/wiki/Easy-peasy-Windows-install)
+in the wiki
+
+#### Conda
+
 1. Install Anaconda3 (miniconda3 version) from here: https://docs.anaconda.com/anaconda/install/windows/
 
 2. Install Git from here: https://git-scm.com/download/win