Hypernetwork training issue

[BlackAlbatros]

I keep getting this error when trying to train a hypernetwork. Any ideas?

loc("mps_add"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/810eba08-405a-11ed-86e9-6af958a02716/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":228:0)): error: input types 'tensor<1x4096x640xf16>' and 'tensor<*xf32>' are not broadcast compatible
LLVM ERROR: Failed to infer result type(s).
zsh: abort ./webui.sh
/opt/homebrew/Cellar/[email protected]/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
andydollinger@Andys-Mac-mini-3 stable-diffusion-webui %

[brkirch]

--no-half is required when training embeddings or hypernetworks on macOS (i.e. start web UI with ./webui.sh --no-half). Also, even simply using hypernetworks is currently broken for macOS without --no-half, although #7637 will fix that if it gets merged.

[BlackAlbatros]

Ok this worked.
Looks like i'm looking at 60-70 hours with a mac mini M1 16GB. Is this normal?

[brkirch]

I'd recommend using a nightly PyTorch build, see here:
#7453

Also I seem to remember training using ~32 GB or so on my 64 GB M1 Max, so I'm not sure 16 GB is really enough. If memory usage is too high it can result in swap being used, which will not only cause slowdowns but if overused could potentially shorten the lifespan of your computer's SSD. I'd recommend you check Activity Monitor and make sure the memory pressure graph under the "Memory" tab isn't spending too much time yellow or red.

[BlackAlbatros]

Yeah it's been the yellows all afternoon so i had to terminate it.
I'll try your above solution and see if it "kills" memory as well.
Otherwise i'll just have to put training aside till i upgrade the M1.
Do you know if anyone was able to train on Colab with Automatic1111?

Preparing dataset...
100% 64/64 [00:16<00:00, 3.78it/s]
0% 0/2000 [00:00<?, ?it/s]Traceback (most recent call last):
File "/content/stable-diffusion-webui/modules/hypernetworks/hypernetwork.py", line 535, in train_hypernetwork
loss = shared.sd_model(x, c)[0] / gradient_step
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/content/stable-diffusion-webui/repositories/stable-diffusion-stability-ai/ldm/models/diffusion/ddpm.py", line 846, in forward
return self.p_losses(x, c, t, *args, **kwargs)
File "/content/stable-diffusion-webui/repositories/stable-diffusion-stability-ai/ldm/models/diffusion/ddpm.py", line 903, in p_losses
logvar_t = self.logvar[t].to(self.device)
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)