Multiple-GPU Bug

[linesword]

Use ComfyUI Branch: Comfy-Org/ComfyUI#7063

When using multiple GPU Animatediff, using multiple Controlnets will result in an error. Initial investigation shows that this is because the Controlnet model instance is not copied to another GPU?

ErrorLog:

TypeError: 'NoneType' object is not callable
Exception in thread Thread-11 (_handle_context_batch):
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/opt/conda/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/root/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/sampling.py", line 796, in _handle_context_batch
    results.append(evaluate_context_windows(executor, model, x_in, conds, timestep, batch_windows, model_options_batch, CREF, ADGS,
  File "/root/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/sampling.py", line 890, in evaluate_context_windows
    sub_conds_out = executor(model, sub_conds, sub_x, sub_timestep, model_options)
  File "/root/ComfyUI/comfy/patcher_extension.py", line 105, in __call__
    return new_executor.execute(*args, **kwargs)
  File "/root/ComfyUI/comfy/patcher_extension.py", line 112, in execute
    return self.original(*args, **kwargs)
  File "/root/ComfyUI/comfy/samplers.py", line 321, in _calc_cond_batch
    c['control'] = control.get_control(input_x, timestep_, c, len(cond_or_uncond), transformer_options)
  File "/root/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/utils.py", line 697, in get_control_inject
    return self.get_control_advanced(x_noisy, t, cond, batched_number, transformer_options)
  File "/root/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/control.py", line 46, in get_control_advanced
    return self.sliding_get_control(x_noisy, t, cond, batched_number, transformer_options)
  File "/root/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/control.py", line 51, in sliding_get_control
    control_prev = self.previous_controlnet.get_control(x_noisy, t, cond, batched_number, transformer_options)
  File "/root/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/utils.py", line 697, in get_control_inject
    return self.get_control_advanced(x_noisy, t, cond, batched_number, transformer_options)
  File "/root/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/control.py", line 46, in get_control_advanced
    return self.sliding_get_control(x_noisy, t, cond, batched_number, transformer_options)
  File "/root/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/control.py", line 51, in sliding_get_control
    control_prev = self.previous_controlnet.get_control(x_noisy, t, cond, batched_number, transformer_options)
  File "/root/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/utils.py", line 697, in get_control_inject
    return self.get_control_advanced(x_noisy, t, cond, batched_number, transformer_options)
  File "/root/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/control.py", line 46, in get_control_advanced
    return self.sliding_get_control(x_noisy, t, cond, batched_number, transformer_options)
  File "/root/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/control.py", line 117, in sliding_get_control
    control = self.control_model(x=x_noisy.to(dtype), hint=self.cond_hint, timesteps=timestep.to(dtype), context=context.to(dtype), **extra)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/ComfyUI/comfy/cldm/cldm.py", line 426, in forward
    h = module(h, emb, context)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 69, in forward
    return forward_timestep_embed(self, *args, **kwargs)
  File "/root/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 38, in forward_timestep_embed
    x = layer(x, emb)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 239, in forward
    return checkpoint(
  File "/root/ComfyUI/comfy/ldm/modules/diffusionmodules/util.py", line 191, in checkpoint
    return func(*inputs)
  File "/root/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 252, in _forward
    h = self.in_layers(x)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/container.py", line 250, in forward
    input = module(input)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/ComfyUI/comfy/ops.py", line 129, in forward
    return super().forward(*args, **kwargs)
TypeError: 'NoneType' object is not callable

[Kosinkadink]

Can you post your workflow? I'm pretty sure I know how to replicate your issue, but seeing your exact controlnets will help me make sure I capture your particular case.

[linesword]

Sure! Here is a test workflow. @Kosinkadink

mutiple-gpu-Workflow.json

[Kosinkadink]

Looks like it is not an issue with anything not getting copied, but instead with v1 AnimateDiff models requiring a groupnorm hack that seems to break when it is applied out of sync between work units. I'll need to change some AnimateDiff code to get around it, and hopefully not cause single GPU performance to drop.

[linesword]

@Kosinkadink Thanks! Would be incredibly grateful if you could fix this issue!

[linesword]

In addition, I also found a serious bug. If you call it multiple times, you will find that the speed will drop sharply when you call it for the second time. I suspect it may be related to the cache of the sliding window? @Kosinkadink

36 frames for first time:

36 frames for second time:

[Kosinkadink]

@linesword can you describe in great detail how to replicate getting the slowdown you mentioned?