Feature Request: Add ComfyUI as a backend

[afecelis]

Describe the feature you'd like to see:

Hi Ben,
It would be great if besides connecting to A1111 locally, blender could also connect to a running instance of ComfyUI. I've tried using its localhost address, but it cant connect tioit via API. Also, I don't know how much Comfy's node structure would impact what you have already developed around A1111.

Additional information

No response

[benrugg]

This would be a nice addition! I haven't personally used ComfyUI, but it looks great. I don't have the bandwidth to do this right now, but I'd love any collaboration!

(Each backend has its own API file and it should be pretty straightforward for a developer to add a new one for Comfy).

[Vidyut]

There's a ComfyUI addon that allows you to use nodes in blender - running it right through blender - very handy, works well. There could be useful code there.

[benrugg]

Thanks for that info, @Vidyut!

[RobeSantoro]

I added ComfyUI support for a project I made in the latest months. You can check my fork here:
https://github.com/RobeSantoro/AI-Render-ComfyUI-Support

I followed @benrugg's implementation for the most part, but my code is still messy and needs some cleanup.

ComfyUI is different from the other backends that the add-on supports, so I decided to create separate files and changed the approach during development. The result now mixes two kinds of methods to interact with the ComfyUI API.

I don't want to bring the nodes from ComfyUI to a node editor in Blender for now. I find ComfyUI pretty lovely to work with. My idea is to bring a set of selected nodes to Blender UI.

[RobeSantoro]

I'm planning a full rewrite of the ComfyUI support in a new fork. If someone is interested, let's get in touch.

[benrugg]

@RobeSantoro thanks for sharing this. A quick glance at your fork shows that you've done a lot of work! If you end up having time to clean it up, feel free to submit a PR!

[RobeSantoro]

@benrugg Of course, I want to clean it up! How do I proceed with the refactoring of my code?
Do you want me to incorporate the Comfy stuff into your files, or can I keep them separate like they are now?

E.g.
I created ./properties_comfy.py, operators_comfyui.py, ./ui/ui_panels_comfyui.py. Would you like me to move the code to the relative add-on files/folder?

[benrugg]

@RobeSantoro ah, interesting - I actually haven't had a chance to look at your code in more detail or run it. Is Comfy that different that there are unique needs for the UI panels? I'll leave it up to you how to best organize it to keep it modular and maintainable. I was mostly going off the fact that you mentioned it needed to be cleaned up 😄

[RobeSantoro]

Hey @benrugg !

Here's a video recorded in May:
https://www.patreon.com/posts/104721510

With ComfyUI, you can create stable diffusion pipelines and save it as a JSON for API usage.

My initial idea was to stick to your implementation of the other backends. While working on it, I decided to change the approach and create a ComfyUI Panel that auto-populates if a specific set of nodes is found in the JSON.

So now, in the code, there are some cases in which I'm using the initial approach, and I would like to refactor those.

I'll keep separate files for convenience; if you create a ComfyUI Support branch in your repo, I could make a pull request on that branch so that your main stays clean.

What do you think?

[benrugg]

@RobeSantoro wow, I didn't realize how different the support for ComfyUI was going to be. I've made a new branch comfyui-support. Feel free to submit a pull request there.

I will do my best to dive deeper into everything you've done as soon as I can. If possible, I will do that this weekend.

[RobeSantoro]

Hey @benrugg!
I’ve been busy this week.

Next week, I'll record a new video with a code walkthrough and a demo of the ComfyUI support achieved so far.

[RobeSantoro]

Hi @benrugg !

Sorry for the delay... I've been stuck on another project.

I made a PR for this repo's comfyui-support branch if anybody wants to check.

#147