Is it possible to have a Mixture of Experts (MoE) in ComfyUI?

[MS00-GitIt]

I’m currently interested in exploring advanced techniques for model optimization and scalability, and I’m interested in utilizing a Mixture of Experts (MoE) approach within ComfyUI. Specifically, I want to know if ComfyUI supports this type of architecture either out-of-the-box or through custom extensions.

If this isn’t directly supported, are there any recommended practices or workarounds that would allow for implementing MoE in ComfyUI? Additionally, I would appreciate any guidance on how to efficiently route different inputs to specialized experts and manage the gating mechanisms within the UI.

Looking forward to any insights or examples.

[ltdrdata]

That is an interesting topic.
I think this is probably a topic that requires a long discussion. What kind of expert do you think of?

If we approach this more macroscopically rather than at the network level, it would be possible to use different checkpoints/LoRA etc. depending on the prompt interpreter. With a bit more application, it could be extended to include on-the-fly model merge functionality.
This approach could be very easily applied once we decide how to handle the prompt interpreter.

If we want to go to an even lower level than that, we'll need to consider resources and conduct more in-depth research on architecture and model patching.

[blepping]

they may be asking about this: https://huggingface.co/blog/segmoe

Diffusion MOE already exists, don't think any front ends actually support it though.

[ltdrdata]

If that's the case, then it would be this:
https://github.com/ZHO-ZHO-ZHO/ComfyUI-SegMoE

[blepping]

If that's the case, then it would be this: https://github.com/ZHO-ZHO-ZHO/ComfyUI-SegMoE

technically it's in ComfyUI but it's just a wrapper around Diffusers so you can't use any Comfy features with it. can't set a sampler, can't use LoRAs or model patches, no img2img since you can't set denoise, etc.

[MS00-GitIt]

I wasn't aware of SegMoE when I started this thought experiment. I was thinking of using smaller optimized models (to lower inference time) that were "experts" at a specific tasks (eg. rendering a [anime, realistic] sky, rendering [anime, realistic] character, rendering an [anime, realistic] object, rendering camera effects, segmentation, ), then using an llm as the gate keeper to decide on the which expert(s) to use (and in what order) depending on the prompt. I was hoping to be able to use comfyUI to visually orchestrate everything and make it easier to share with others to experiment with the idea.